Scaling web applications with a Message Queue (MQ) service

Message Queue (MQ) services make it possible for applications to communicate asynchronously, by sending messages to each other via a queue, allowing applications to scale as usage and traffic increases.

This is particularly evident in more complex web applications.

This is apparent in more complex web applications, where there are many processes running, high levels of automation and loads of functionality.

So what is message queuing exactly?

Let’s first take a look at message queues.

A queue is a sequential list of items that are waiting to be handled (hence the word queue), in order starting from beginning until the end of the queue. Once an item is executed it can send a confirmation response and then is deleted.

A message is the data that is sent between the sender application and the receiver application. Essentially it is a piece of data. Therefore it isn’t a traditional message in the sense of an email or text message but more of an instruction to place a certain task in the message queue, for processing.

 

Where is it applicable (which types of projects)?

Message queues can be used in a variety of web and software related. In this article, we will be looking at where it is applicable in web apps.

You would typically want to use a message queuing solution where there are processes and functions that need to be executed but can run “in the background”. Suppose you have a system that requires a time consuming process that is initiated by a user. You wouldn’t want the user to have to wait for this process to finish before they can continue with the next step. You would want to queue that process and move the user to the next step straight away, therefore the message queue would receive the instruction and then process it independently of what the user is doing.

A simple example would be that during a certain step in the user’s journey (for example registering on a website) you want to send out an email via SMTP to the user and three administrators. That’s four emails to need to be sent out but because SMTP is the secure method of sending mail, encryption and handshakes needs to take place before the mail is actually sent off, meaning that this can take some time but we don’t want to let the user wait while this happens. The message queue will execute this task independently.

A use case for message queuing

Imagine that you have a web service that receives loads of requests every second, not a single request can be lost and all these requests need to be processed very quickly. In other words, this web service has to be constantly available and be able to receive new requests and not be locked by the processing of previously received requests.

In this example, placing a message queue between the web service and the processing service is a great solution. The web service can put the “start processing” message in a queue and the processing service can take and execute the messages in sequence that they were added to the queue. The two processes are therefore decoupled (separated) from one another and do not need to wait for the other to finish. If you have a lot of requests coming in a short amount of time, the processing system will be able to process them all. The queue will persist with the requests even if their number grows.

Then imagine that the web application’s users grow tremendously, more features are added and the workload increases significantly, resulting in the system needing to be scaled up. You can create multiple workers to accept jobs if the queue gets too long. You can run as many workers as the server resources permits, to process the queue faster.

What are the benefits?

There are numerous benefits to using message queues, which include:

  1. Better performance: as you are separating the processing of tasks from what the user is currently busy with. Users are not stalled having to wait for a process to finish running.
  2. Increased reliability: by using a message queue, we are ensuring that these tasks get executed as planned.
  3. Scalability: as your web application is being used more and more the message queue takes care of many onerous tasks and ensures they are processed, ensuring no downtime or lag.
  4. Reporting: we are able to draw reports on the status of the message queue and the processes that have run or need to run.
  5. Reduce user frustration: as we are separating the processing from the user’s interaction, there is less likelihood of users being stuck or slowed down due to heavy task processing.
  6. Transaction guarantee: message queues are designed to be transactional. When you process a message off the queue, you have to tell it where you completed processing the message before it is completely removed from the queue. This helps to ensure that a transaction only occurs once.
  7. Break larger tasks into many smaller ones: another good use for queues is breaking up a larger task into lots of little pieces and then queuing them all up for processing.
  8. Monitoring: Message queuing services enable you to monitor how many items are in a queue, how fast messages are processed and other important stats. This can be very helpful from an application monitoring standpoint to keep an eye on how data is flowing through your system and if it is getting backed up.

How does it improve on web apps that don’t have message queuing?

By introducing messaging queuing to an existing project, you are reducing the risk for potential errors (not to say that the current application has errors). This also makes the web app more robust and scalable. Performance should also increase due to the separation of processing of tasks using the queue.

What are the potential issues or shortfalls?

The only real shortfall is that more development is required to setup the message queue service and also integrate the web application with this message queue service.

More web hosting power will be required as the message queue service needs certain software to be installed on the server and to be customised.

Does every website / web app need message queues?

No. This is only needed where there are tasks that need to be run independently of the main web application, however, it can apply to almost all web apps but it’s not a necessity (can be seen as overkill in simpler web applications and websites). We would, however, suggest starting to think about improving the performance of your web applications by using background tasks to offload some long running transactions.

What message queue service do we use?

We are avid fans of RabbitMQ (https://www.rabbitmq.com), which is the most widely deployed open source message queuing system. RabbitMQ is lightweight and easy to deploy on premises and in the cloud

Message queue conclusion

Message queues can be a very critical component to your web application, allowing it to scale, reduce errors and have improved performance, all while reducing potential user frustration.

While it can be more work to setup the likes of RabbitMQ, etc. and doing additional development to integrate with these types of message queue services, if you are serious about the performance of your web application, you should strongly consider implementing this solution.

Don't forget to share this post!
About
Having been established in 2005, in Cape Town, South Africa, Elemental is primarily focused on providing web development services and digital marketing campaigns.