Rory Belvins is an alumnus of Spektrix.
Here in the Spektrix Engineering team, it’s not all about developing new features. We also spend a lot of time thinking about how we can keep improving the reliability and performance of our system. For the past 4 months, we’ve been working on the way we use message queues – a way of processing complex data without compromising how the system performs other tasks, like selling tickets. Queuing might not seem like the most interesting topic to write about, but it’s actually an important way of leveraging the power of the cloud and passing on the benefits to our clients – and that makes it pretty exciting.
What is message queuing?
Perhaps the easiest way to answer this question is by explaining what it does. When you use Spektrix to run a task that takes a little time, such as a Report or a Customer Segmentation, we don’t want to keep you waiting while Spektrix processes all the data and returns a result to you. Instead, the system compresses your task into a digital message and places it in a virtual queue, known as a message queue.
This queue behaves much like the queue at the post office: in the post office, customers arrive and join a queue for a number of cashier windows. As a window becomes free (“cashier number two please!”), the customer at the front of the queue goes to that window, and all the other customers in the queue move up one place. Similarly, in a message queue system, messages arrive and are placed in a single queue for our job-processing servers, which are separate from the main Spektrix server cluster. If a free server is available, the first message on the queue is removed from the queue and processed. In the case of our job system, this means that the message is uncompressed back into a job and run.
While this is happening, the system stays responsive to you, so that you can carry on with other work in Spektrix without disruption. When the job is completed, you receive a notification from Spektrix: usually either an email or that little yellow box in the top announcing that your report is ready.
Why message queues?
Using message queuing like this has several important advantages:
- Scalability. If we discover that messages are arriving faster than they are being processed and backing up in the queue, we can simply add more servers to our job-processing cluster to process them faster.
- Load balancing. If the queue distributes messages across many servers, and holds messages in the queue until there is a server available. This evens out the computational load on the job processing cluster and prevents too many jobs at once causing problems on the cluster.
- Reduced coupling. Because data-intensive jobs are processed on a separate server cluster, the core Spektrix system has more resources for critical tasks. So even if you’re running masses of reports, it doesn’t affect your ability to sell tickets.
- Reliability. Message queues back up the messages that they receive and only get rid of them when they receive confirmation from our job cluster that the message has been processed successfully. This means that if something does go wrong with a server that’s processing a message, the message will be returned to the queue and reallocated to another server. So even if a server fails, your jobs will still run.
We’ve been using message queues for some time in Spektrix, but as we’ve grown, we recently decided to move from MSMQ, Microsoft’s message queue, to the open-source message queue RabbitMQ. We like many of Microsoft’s products, but in this case we wanted the flexibility and performance offered by an open-source solution. In particular, the lack of restrictive licensing and wider OS support, means that we can set up and run RabbitMQ queues much more easily. This increased flexibility gives us the ability to improve the performance and reliability of Spektrix in the future by adding messaging to other areas of the system, such as the Spektrix App and order processing. RabbitMQ has improved routing options, allowing us to better sort and direct messages to the appropriate processing cluster. It’s also got a convenient web-based management interface which allows our DevOps team to better manage our message queues.
We’re really impressed with the way that message queuing has ramped up Spektrix’s performance and believe it has hugely benefited our clients. We’ll be incorporating messaging queues into other elements of the system in the future.