Thingiverse, SQS, and Open Source

Here at Thingiverse, we use a lot of open source software. Sometimes it works great, and sometimes it needs a bit of tweaking. Sometimes it needs a lot of tweaking. Here’s a story about how open source works for us. If we take a peek under the hood, Thingiverse has a distributed rendering system for producing web-friendly images of the various funky file formats that digital designs come in. To do this, we have 2 different ‘tiers’ of software: the front-end servers which serve up all the images, html, etc. and the rendering servers which take all the images (dxf, stl, png, etc) and render them to the lowest common denominator: jpg.

When I build things, and Thingiverse in particular, I want to build something of quality and power. Instead of rolling my own queue system, I decided to use Amazon’s SQS aka Simple Queue Service aka super-scalable queueing service. This is a simple little service allows me to put messages on the queue from one place, and get messages off the queue from another place. Thats about it. I use this (as well as Amazon S3) to let the front-end tell the renderers that there are jobs to be done. I also use it to let the renderers let the front-end know that jobs are finished. It works great and it was a really enjoyable system to write.

So, where is this all leading? In order to talk to Amazon SQS, you need some sort of API. I found a PHP class developed by Intellispire that implements it. It was a nice class, but unfortunately it was from 2006 and Amazon had changed the SQS protocol since then. I had to give the class a complete work-over in order to get it working properly with the new SQS and in the process slipped a few improvements in there as well. As a member of the open source community, I’d like to offer my new and improved version back to the community so that others may use and enjoy it.

You can download it here.