Summary:

Amazon S3 is a great option for (among other things) image hosting, YSlow is a great little tool for uncovering performance related issues with a site and using Bucket Explorer allows a person to easily upload files to the S3 service while setting custom headers. Using the tool to set far future expires headers facilitates caching which improves site performance.

Preface

I started using Amazon Simple Storage Service (Amazon S3) for image hosting this week to help cut down on my bandwidth costs. S3 is a great service offered by Amazon Web Services that allows developers to store and serve data using Amazon’s infrastructure for literally pennies per GB of storage and transfer.

Why have I begun the move? Well, as I hoped, I’ve had excellent growth in site traffic since I relaunched last year. Great stuff, except for the fact that my bandwidth usage has begun to strain the limits of my current hosting plan. I serve a lot of images* so I can eat through GB pretty quickly. I considered switching hosts, but after 8 years with FutureQuest, and literally nothing to complain about in all that time I’ve come to the conclusion that I really love my web host and I’m just not going to switch.**

So, with switching off the table, I had to figure out something else to do. If you want numbers- my October 2007 bandwidth usage was up 712% over my bandwidth usage in October of 2006.

That and I want to keep growing.

So… what’s a man to do?

Amazon S3 to the rescue

That’s where S3 comes in. I looked at S3 for a few weeks before signing up and I came up with the following conclusions- the infrastructure can’t be beat, it scales to the moon, I love the fact that I’ll pay for just what I use and for what I use the pricing is just fantastic.

I could move my images over to Amazon, see another 700% increase in bandwidth usage over the next year and still be making out with some savings. Check out these numbers:

Storage
$0.15 per GB-Month of storage used

Data Transfer
$0.10 per GB – all data transfer in
$0.18 per GB – first 10 TB / month data transfer out
$0.16 per GB – next 40 TB / month data transfer out
$0.13 per GB – data transfer out / month over 50 TB

Requests
$0.01 per 1,000 PUT or LIST requests
$0.01 per 10,000 GET and all other requests

Looking at those numbers and looking at the incredible tools out there already to make using S3 as simple as using a traditional FTP client to move files onto the web, it became pretty clear that S3 was the solution to my problem. So I signed up and went about the process of setting up one of my sub domains (media.drunkenfist.com) to point to Amazon so that the transition, for at least the files already served off of that domain, would be seamless.

As an aside- setting up a sub domain to point to Amazon is really pretty easy. All I had to do was create a bucket (an S3 organizational structure somewhat analogous to a typical folder) named media.drunkenfist.com and then set up (through my host) a CNAME record that pointed requests for media.drunkenfist.com over to that bucket on the Amazon servers.

Something like this:

media.drunkenfist.com CNAME media.drunkenfist.com.s3.amazonaws.com

As long as the bucket name matches the host name exactly, it’s really dead easy.

Anyway, back to the narrative.

Once the domain was pointed in the right direction and the files were being served from Amazon’s servers, I tested it, was happy with the results and gave myself a little pat on the back for a job well done.

I relaxed a little bit and my attention started to turn to my plan for moving the rest of my images (those not already served off of media) over to the new hosting server. I’ve got to take the rest of the migration slowly because I don’t want to jeopardize my image search traffic. I’ve got to make sure any changes I make to popular images won’t break Google’s snapshot of my site.

All in all, it’s a typical side-project for me. A little thinking, a little research, a little planning and then, some furious late-night execution across a few different disciplines.

Good times 🙂

And then came Christmas, where the one flaw in my original implementation poked its head out from the pages of High Performance Web Sites: Essential Knowledge for Front-End Engineers

Confessions of a YSlow junkie

Maybe “junkie” is too strong a phrase 🙂 I am interested in site performance, however, and the YSlow plugin for Firebug is a great tool for analyzing issues with the way pages are constructed and the manner in which site components are served.

Anyway, I got the High Performance Web Sites book for Christmas and while I was flipping through it I was struck by rule #3. In short #3 says to use a far future expires header for images in order to facilitate caching. Since all the images I was hosting over at Amazon were interface images (like my sprite***,) getting them to cache correctly was important business.

Thing is? Unlike the images served by my main web host****, the images over at Amazon weren’t getting served with any expires headers at all.

Bummer.

I had used the Amazon S3 Firefox Organizer(S3Fox) Firefox Add-on to move my files over to Amazon originally and while it integrates with my browser of choice and is dead easy to use, it doesn’t allow a user to set the expires header for files it creates on the Amazon server.

Oops.

The Solution

Some furious Googling led me to Bucket Explorer a handy little app that allows one to set custom headers for files being uploaded to the Amazon servers. Before finding that I had visions of writing my own uploader in PHP. Part of me thought that would be fun. The other part just wanted a nice, handy solution written by someone else since I’ve already got a million little side projects going on and adding a new one didn’t seem like the most efficient idea.

Anyway, to add custom headers using Bucket Explorer:

1. Fire up the program
2. Enter your access key and secret key
3. Navigate to the local folder in the left pane and the target bucket in the right pane
4. Select the image (or images) in the local folder
5. Right click and select “Upload with Custom Headers”
   
6. Click “Add” in the dialog that opens
   
7. And finally, enter your header information
   

   For more info on the expires header and the value to enter, check out the HTTP/1.1 Header Field Definitions provided by the w3c.

8. Hit OK a bunch of times, and that’s that. You’ve got far future expires set up and your return/multiple page visitors will be all the better for it.

As you can see, that’s really pretty straightforward.

The software costs $29.99, but there’s a 30 day trial if you want to check it out in advance. I’ll definitely be ponying up my hard earned dollars as Bucket Explorer does pretty much everything I need an interface to S3 to do and feels faster/more stable than some of the other options I’ve looked at.

As a note, if you’re interested in any of the topics covered in this post, I’ve got another performance related post to drop over the next couple of days and I’ll be updating the blog with the S3 stuff as often as I encounter anything interesting, so check in from time to time to see what’s new.

*Both here and in leeched form across the breadth of the “social web.” I decorate many myspace (and Hi5 and Friendster and YouTube, etc. etc. ) pages.

I love seeing 15-20 year old kids taking my stuff to decorate their pages, by the way. It’s the most honest validation of my work I could ever imagine.

**Especially since moving my main site, the four subdomains (comics., media., styles., and mobile.,), two alternate domains (roblarsen.org and robreact.com) and one redirected domain (itsalljustcomics.com) seems like an incredible pain in the ass. We’re talking DNS, code, databases, passwords, paths, blah, blah, blah. I just can’t see myself doing it without some sort of mythically powerful inducement. Maybe if I could host my own box somewhere or something. Anything short of that and it’s just a nightmare.

***It’s not quite as nice as Ask‘s, but it’s still pretty good, I think.

I love that Ask sprite, by the way. That’s some great attention to detail on their part and the execution of it is a dramatic lesson in how to construct a great Sprite image for serving a single interface image.

**** My main server is running Apache, so I have the following in my root .htaccess to set expires headers for images:

ExpiresActive On
ExpiresByType image/gif A2592000
ExpiresByType image/png A2592000
ExpiresByType image/jpeg A2592000