Introduction to RabbitMQ and Kafka Technologies
I, Arslan Mirbzergi, would like to discuss in this article complete information about RabbitMQ and Kafka technologies, their applications and advantages. For some reason, many Developer believe that RabbitMQ and Kafka technologies can replace each other. Although this is true in some cases, there are various fundamental differences between these platforms.
As a result, different scenarios require a different solution, and making the wrong choice may severely affect your ability to design, develop and maintain a software solution.
In the following, the introduction of Asynchronous messaging patterns and then the introduction of RabbitMQ, Kafka,their internal structures, the fundamental differences between these platforms, their different advantages and disadvantages, and how to choose between the two are discussed.
Asynchronous Messaging Patterns
Asynchronous Messaging is a messaging scheme in which the production of messages by Producers is separated from its processing by Consumer. In messaging systems, we usually have two main messaging patterns: 1) queue waiting queue 2) publishing/subscribing.
In the pattern associated with the Message-queuing method, queues separate Producer from Consumer in terms of time. Several Producers can send messages to the same Queue. However, when a Consumer processes a message, that message is locked or removed from Queue and is no longer available. In fact, a Consumer can only process a specific message.
If Consumer can’t process a specific message, the messaging platform usually returns the message to Queue made available to other Consumers. In addition to time separation, Queues allow us to categorize Producers and Consumers independently. We can also provide a degree of Fault-tolerance against processing errors.
In the publish/share communication pattern (or pub/sub), a message can be received and processed by multiple Consumers at the same time.
This template allows a Publisher, for example, to let all Subscriptions know what happened on the system.
In general, there are two types of Subscription:
- A fleeting Subscription where subscription is only active as long as Consumer is active and running. As soon as Consumer shuts down, your subscription and unprocessed messages will be lost.
- Durable Subscription maintains the message platform until the subscription is completely deleted, even if Consumer is turned off, and message processing can continue later.
RabbitMQ is the implementation of a message exchange interface, often known as Service bus. RabbitMQ supports both messaging patterns described in the previous section. Other popular messaging exchange interfaces include ActiveMQ, ZeroMQ, Azure Service Bus and Amazon Simple Queue Service (SQS). All of these interfaces have a lot in common, and many of the concepts described in this section can be used for most of them.
RabbitMQ supports Classic messages that are out of Queue. A Developer Defines Named Queues to which Publishers can send messages. Consumers also use the same Queues to pick up messages for processing purposes.
RabbitMQ implements the pub/sub method using message exchange. A Publisher publishes messages in Message exchange without knowing who subscribers of these messages are. Any Consumer who wishes to subscribe to Subscribe in Exchange will create a Queue. Message exchange then places the generated messages for Consumer use in Queue. Messages are also filtered for some Subscribers based on the rules we set.
Note that RabbitMQ supports both fleeting and durable Subscription. Consumer can determine the type of Subscription you want to use through RabbitMQ’s API.
You can also develop a hybrid approach according to rabbitMQ architecture. That is, some subscribers that work together. Together, Consumer groups form competitive Consumers to process a specific Queue. In this way, the pub/sub pattern is implemented. At the same time, some subscribers are allowed to meet their criteria for managing incoming messages.
Apache Kafka is not a messaging intermediary, but a distributed Streaming platform. Unlike RabbitMQ, which is based on Queue and Exchange, the Kafka storage layer runs using a split Transaction log. Kafka also provides an API Streams for processing Streams in Real Time and an API Connectors for easy integration with different data sources.
Cloud vendors offer alternative solutions to kafka storage layer. These solutions include Azure Event Hubs and AWS Kinesis Data Streams. There are also Cloud-specific and Open-source options for Stream processing capabilities in Kafka.
Kafka does not run the Queue concept. Instead, it stores a set of record in categories called Topic. For each Topic, Kafka retains a partitioned report of messages. Each partition is a special and immutable order of record that messages are constantly added to.
Kafka adds messages to their partitions when they arrive, and by default, uses Round-robin to play messages uniformly among partitions.
Producers can change this to create Logical stream message. For example, in a Multitenant app, you may want to create logical stream messages according to tenant ID of each message. In an IoT scenario, identity map of any Producer may be needed to exist continuously on a specific partition. It should ensure that for Consumer, the delivery of all messages from Logical stream to the same partition is guaranteed.
Consumers read and use messages successively by registering and keeping Offset Indx on partitions. A Consumer can use multiple Topics and rate the number of available partitions. As a result, when creating a topic, you should carefully consider the amount of operational throughput from Messaging in that topic. A group of Consumers who work together to use a Topic is called the Consumer Group. Kafka’s API typically controls the partition processing balance between consumers in a Consumer group and the current storage of Consumers’ Offsets.
Implementing messaging patterns with Kafka
Kafka implementation map is completely in accordance with pub/sub pattern. Producer can send messages to a specific topic, and several Consumer groups can use the same message. Each Consumer group can be categorized individually to manage loads. Since Consumers, Offset records their partition-related, they can have a durable Subscription that retains their Offset when restarting subscription or fleeting. After each start, remove offset and restart from the last record on each partition.
However, it is not very suitable for the Message-Queuing template to use Kafka. Of course, to compete with the Classic message queuing template, you can only have one Topic with a Consumer group, which still has a number of problems.
Regardless of whether consumers have used messages or not, Kafka keeps messages on partitions for a predefined period of time. This means consumers can read old messages again. In addition, Developer can use the Kafka storage layer to implement mechanisms such as Event sourcing and audit Logs.
And in the end,
While RabbitMQ and Kafka can sometimes be replaced, their implementations and implementations are very different. As a result, we can’t put them together in a group of tools. One of them is the message intermediary and the other is a distributed Streaming platform. As a result, we need to recognize these differences and then examine which of these solutions we can use for a particular problem.