Outage post-mortem

Please see our blog for an explanation of yesterday’s outage.

Phanfare is back up

We are still investigating why we were down.

Phanfare still down

We have confirmed its a networking problem in our NJ datacenter. From what we can tell we are completely off the network right now. We are working to figure out what the problem is and resolve it. Some of this involves outside vendors so we don’t have perfect visibility into the issue. We don’t currently have an ETA on when we will be back up.

Phanfare down – we are investigating

Phanfare.com is offline; looks like a network issue at the datacenter issue. We are investigating and will post status soon.

We are down for scheduled release, back in 30

here are some highlights of the coming release:

• h.264 video support
• visitor reports within iPhone view
• caption by filename is back
• Adobe Lightroom plugin
• RSS improvements
• Archive DVDs now for sale

Phanfare now operating normally

Amazon S3, our storage provider, has returned and Phanfare is now operating normally. We were never fully down due to our architecture. We buffer uploads so we can withstand S3 being down for about 2 days. We also cache hot content such that most content was still available. If you clicked on older content that was not cached during the outage, we would time out.

Phanfare partially down due to Amazon Web Services S3 outage

Amazon S3 is currently partially down. Phanfare uses Amazon S3 to store images. The way our architecture works, when they are down, we can’t serve any images that are not cached on our side. We also buffer uploads, so uploading should be operating normally. Amazon writes in their health status blog that they currently testing a potential solution for the outage. We don’t have any ETA on when this problem will clear.

You can see Amazon’s status here

Release out, Phanfare operating normally

Release scheduled for June 27th at 2am has been postponed

Phanfare is operating normally. We will reschedule the release and post further information soon.

Scheduled Downtime, June 27th, 2am-4am EDT

We will be releasing a new version of the service.