Moving Huge Files to the Cloud

Posted by Todd Bryant in General | Tagged | Leave a comment

After reading Mike Richwalsky’s post on Yottaa and cloud computing, I started experimenting with Amazon’s cloud service and possible integration with the Mixxer site running on Drupal.  This post is a bit more technical than what I usually write here, but even if you’re not heavy techie, it can hopefully give you an idea that’s simple and useful enough that your techie types wouldn’t mind implementing it.

With the growth of video, storage and bandwidth have become major issues for many instructional technology departments.  For most, this means either video projects are uploaded to YouTube with their accompanying restrictions and quality or an expensive server and accompanying bandwidth need to be paid for by the college.  This is the demand side of the equation.

Amazon on the other hand has a great deal of supply when it comes to servers, storage and bandwidth.  This is due not just to their huge size, but also because their business depends on the ability to handle large and sudden bursts of traffic around the holidays.  A fact that came in handy when Wikileaks supporters tried and failed to take down their servers.

What this means for us is that we can upload large files, or large numbers of files, to servers scattered around the world for extremely low prices.  Basically one first uploads all the files to a “bucket” on Amazon’s S3 servers.  S3 is simple storage.  If all you need is downloadable files from one server, you can stop there.  Just make each file public on upload then distribute the url to whomever may need the files.   Here is the pricing.

If you want to do streaming and video, you have to take it one step further by signing up for Cloudfront, creating a distribution of the bucket, and distributing another url.  In addition to streaming, this also means your content will be available on a number of servers around the world which can have a huge impact on performance.  In addition to the S3, you would have to look at this price chart as well.

Amazon has no minimum fee, which works out well for many class projects.  We often have a relative few number of very large files that will only be seen by a relative few people, but all at the same time.  To accommodate these peaks requires a rather expensive server and can strain bandwidth at peak times.  You’ll have to have an idea of how many GBs you would be uploading and distributing per month to compare prices, but if like us you’re dealing mostly with short Quicktime videos it’s quite cheap.

The integration with Drupal is on hold for now.  For those interested, the CDN and CloudFront modules are the places to start.  I am going to wait until I upgrade to Drupal 7 when it looks the CDN module in particular will be able to serve all of my static files from Cloudfront without tinkering with the core Drupal php code, which I try to avoid.

Update:

Google has updated their storage options.  Also incredibly cheap, but pay attention to single file size limits.  One GB of video isn’t a lot.

http://rww.to/ieQO5m

Leave a Reply