How to scale twitter

Scaling Twitter redux--the ESB should be your best friend | Negative Approach - CNET News.com

Scaling Twitter redux--the ESB should be your best friend
Posted by Dave Rosenberg 1 comment

As we Twitt-iots sit around bemoaning the fact that we can't send each other useless junk on a flaky service, I thought I would take this chance to address the notion that this message-scaling problem is new.

It's not. It's very common, and it can be solved.

Scaling a messaging platform is why IBM sells a boatload of MQ series, why the AMQP protocol was developed, and why JMS is nearly ubiquitous. Pretty much every large enterprise has similar scale issues related to messaging, especially in financial services. But they don't have downtime, and if they did as frequently as Twitter, the people behind them would all get fired.

This is a topic I actually know something about. (Disclosure: my company develops an open-source enterprise service bus, or ESB, called Mule.) All of our use cases involve some kind of complicated messaging architecture, whether it be Web-service based, publish and subscribe, one to many, direct connection, etc. And most deal with data transformations and a vast array of protocols.

In my view, what Twitter needs is to adopt a bus-type of architecture that separates the transport from the application and uses a middleman to process the transactions. This is a very common enterprise scenario that needs to be applied. This is what an ESB does.

publish and subscribe!