litany against fear

Messaging with Ruby on Rails Notes

Part of my series of notes from the Professional Ruby Conference. See them all here.

This is a talk from Blaine Cook, formerly of Twitter. Now he’s in Ireland!

Building messaging systems is simple. Talking. Mail. Telephones. Messaging helps out a lot with AJAX. Parallel processing too, especially with technologies like MapReduce. Also important to decrease latency. Definitely important because latency determines context.

HTTP really isnt messaging. There’s a requester, and a response. Not ideal…ends up with a system that constantly asks “are we there yet?” Also need a specific address for it, and low latency. You want online messaging to be like fedex: deliver straight to your door, not to a mailbox where you have to go pick it up. Definitely needs to be asynchronous!

There are many options to do this online: RabbitMQ is Erlang, but a bit odd. ActiveMQ is more enterprisey and Java-based. Starling/beanstalkd are very lightweight, simple, but not a lot of features. Gearman/TheSchwartz is more of a distributing processing framework than a queue server. cron could be used, but it’s not designed to handle large numbers of messages. TONS of protocols too that each use.

The queue server/broker needs to be durable: There needs to be a guarantee if the broker dies that the messages are going to get delivered. Beanstalkd doesn’t have any durability, Spread not so much in some situations. Obviously durability is very important, especially for some domains (financial stuff). Reliability is important too! Starling works a lot like Mongrel: can scale very easily, works well with multiple brokers acting together.

Another important idea: Safety! If the queue server can’t finish a job, the job will get picked up by another broker. Starling doesn’t have this…results in dropped messages because workers crashed. Very application specific, but basically you need to be able to ensure that the message isn’t sent twice while recovering.

Blaine recently built and released ActiveQueue. There’s many different ways to tackle this problem: Workling, Nanite. Probably 15 different queue servers released in the past 2 years…which is fantastic since it shows that people are using it and it’s growing. Doing it yourself is possible too. Clients talk to a queue, and workers pull jobs off the queue. Workers talk to the queue, and don’t forget to throw in some monitoring of the workers & clients.

The future: XMPP and cross-site messaging. Moving towards a low-latency, addressable internet! Right now, it’s like checking to see if everyone you know has sent YOU mail. We can do better than that! Servers need message-oriented backends to deliver message-oriented frontends.