Which messaging queue is best

-

Messaging is one of the most important aspects of modern programming techniques. Majority of today’s systems consist of several modules and external dependencies. If they weren’t able to communicate with each other in an efficient manner, they wouldn’t be very effective in carrying out their intended functions. There are quite a few messaging queues available on the cloud today, some of the popular ones being Apache Kafka, RabbitMQ, Amazon SQS, Google Cloud Pub/Sub etc. There is no doubt that we need a messaging queue for several modules but which one? In this blog post, I would like to give a try on answering that question.

RabbitMQ

RabbitMQ is an open source message broker software managed by Pivotal. It implements the Advanced Message Queuing Protocol (AMQP). RabbitMQ comes with a built-in management dashboard for easy configuration and monitoring of message queues. It also supports requests for message acknowledgments and is easily deployed in HA mode with minimal effort. With a message rate of about 20k+ msgs/sec which is much less than Kafka, it’s sufficient enough for most use cases. Due to its platform-independent framework and ease of use, RabbitMQ is a very mature message broker offering with a lot of extended support from the community.

Amazon SQS

Simple Queue Service (SQS) is Amazon’s cloud-based message queuing system made available as part of Amazon Web Services (AWS). Unlike the other brokers mentioned here, there is barely any setup/deployment required for an application to use SQS. All you need is AWS credentials to be able to use it. Since it is a SaaS (Software-as-a-service), this makes it a much lower-cost option since there is no infrastructure cost. You only pay for what you use. SQS also has the notion of in-flight messages. This means that the message is pulled off the queue but never deleted until the ‘delete‘ command is sent for that message ID. So if you lose your worker mid-processing of the event, that event isn’t lost for good.

One drawback of the SQS implementation is the need for polling of a message queue to determine if new messages have appeared. This is a bit of an issue being as you must now model your application to perform a polling cycle in order to determine if new messages are available. While the SQS costs are reasonable, Amazon places a limit on the message rate which can be increased on request and also charges extra based on the overall message rate.

Apache Kafka

Apache Kafka is a message broker that was originally developed by LinkedIn before going open source under the Apache umbrella. Kafka’s main focus was speed and efficiency. It can maintain a message throughput rate of 100k+ msgs/sec which is significantly higher than most other alternatives. However, other basic features found in other brokers are eschewed in favor of efficiency. Features like message acknowledgments, routing rules, and an admin UI are not available as standard. Even though Kafka is defined as ‘stable’ and ‘production-ready’, deploying Kafka in HA mode is tougher than it is with other providers. It is highly dependent on the Zookeeper module in order to maintain synchronization between multiple Kafka nodes and client apps.

Google Cloud Pub/Sub

Google’s Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between applications or server. It is developed specifically to facilitate messaging among applications deployed on the Google Compute Engine (GCE) cloud. Its basic features are very similar to that of Amazon’s SQS in that there are no infrastructure costs and you pay for what you use. The major difference from SQS is that Google charges based only on volume and not transaction speed i.e. you will not be charged extra for a sudden increase in message rate.

 

Category Rabbit MQ Amazon SQS Apache Kafka Google Cloud Pub/Sub
Message Rate (msg/sec) 20k+ Scalable 100k+ 10k+
Message Acknowledgements Yes Yes No Yes
Built-in UI Yes Yes No Yes
Routing Rules Yes Yes No Yes
Protocol AMQP HTTP REST Kafka HTTP REST
Cost Medium Pay-per-use High Pay-per-use
Supported Frameworks C/C++, Python, Go,Erlang, .NET, Java,Ruby, Node.js, Perl, PHP C/C++, Python, Go,Erlang, .NET, Java,Ruby, Node.js, Perl, PHP C/C++, Python, Go,Erlang, .NET, Java,Ruby, Node.js, Perl, PHP C/C++, Python, Go,Erlang, .NET, Java,Ruby, Node.js, Perl, PHP
Use Cases Robust systems where features like acknowledgements, deeper management are important Useful when deploying applications on AWS EC2 for low latency communication High ingestion platforms where speed,scale and efficiency is the prime concern Works well with applications deployed on GCE

 

With this data, I hope you have a clear understanding that there is no single queue which can cater to all types of needs, different queues are good at different things, you just have to pick the right one for the problem.

Thanks for dropping by !!! Feel free to comment to this post or you can drop an email to naik899@gmail.com.

 

 

 

 

The post Which messaging queue is best appeared first on Ravindra Naik.

Comments

Post a Comment

Popular posts from this blog

Serve GZip CSS and JS from AWS S3

-
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. If you are looking to gzip  your static JS and CSS when hosting on S3, however, it’s not obvious how to do so, and results often bring up other AWS hosting services as well. Here’s the quick and simple way to serve gzipped JS and CSS from AWS S3. S3 needs those files compressed before they are uploaded, it can’t do the actual gzipping for us. This is done with a simple command, which I recommend be added as a script to your build step, or your continuous integration process. gzip -9 /filepath/bootstrap.js or gzip -9 /filepath/bootstrap.css The -9 denotes that we want the
Read more

Create a Central Logging System

-
Image
Logging is the most important part of any system. They give you insights about the application, what kind of errors are occurring and what components are causing the errors and also about how the application is reacting and working when something wrong happens. The usual practice is almost every application write to logs either in the file system or in Database. If your application is running on multiple hosts then designing a central logging system becomes crucial since you can collect, aggregate and maintain logs at one centralized location and do operations on top of them. There are many tools available to which can solve some part of the problem but we need to build a robust application using all these tools.                                                                           Create a Centralised Logging Application Collection Imagine you have a web application running on the server and if something goes down, your developers or operations team need to access lo
Read more

Create a calendar invite (ics file) in ASP NET

-
Creating a calendar invite (ics file) and attaching a email via service or sending an email with downloadable calendar invite is what we are going to learn in following posts. In this post, I will show how you can use ICal library from Nuget and create a calendar invite. The calendar begins with the keyword  BEGIN:VCALENDAR  and ends with  END:VCALENDAR . In between the calendar you can add as many events as you want beginning each event with  BEGIN:VEVENT  and ending with  END:VEVENT . There is a catch when adding multiple events to a calendar, and that is the type of METHOD you use. With METHOD:PUBLISH, you can add multiple events and upon opening the iCal with MS Outlook it will create a new calendar, separate from your main Exchange calendar. METHOD:REQUEST creates a calendar invite, so you can only include one event and upon opening with MS Outlook it will ask you to accept the event and add it to your main Exchange calendar. With this method you have to include the organizer
Read more