Data Spine RabbitMQ

Overview

RabbitMQ was selected to be the Message Bus/Broker of the Data Spine. RabbitMQ is a message broker or message-oriented middleware that implements AMQP (Advanced Message Queuing Protocol). It is one of the most popular and most widely deployed open source message brokers.

Some of the features of RabbitMQ and their significance in EFPF are explored below:

  • Support for protocols
    • In EFPF, we primarily make use of MQTT, MQTTS and AMQP (0-9-1)
    • RabbitMQ supports AMQP inherently and MQTT/MQTTS via a plugin
    • It also supports STOMP, AMQP 1.0, HTTP and WebSockets
  • Deployment
    • Docker container has been selected as a deployment unit in EFPF
    • RabbitMQ’s Docker images are made available with each release
  • Management and Monitoring
    • RabbitMQ provides a management GUI (see Figure 1 below) and an HTTP-based API for administration, management and monitoring of channels/topics, users, dataflow stats, etc. via a plugin.
    • In addition, RabbitMQ also provides a management command line tool ‘rabbitmqadmin’ that can be used in performance critical environments such as production as opposed to the GUI.
  • Identity and Access Management
    • RabbitMQ supports multiple SASL (Simple Authentication and Security Layer) authentication mechanisms out of which, three are built into the server - PLAIN, AMQPLAIN and RABBIT-CR-DEMO and one ‘EXTERNAL’ is supported via a plugin. More such authentication mechanisms are supported via plugins. In essence, RabbitMQ supports widely used password-based, token-based and client certificates based authentication.
    • RabbitMQ also supports multi/tenant authorization with the help of ‘virtual hosts’ which enable logical grouping and separation of resources such as connections, exchanges, queues, bindings, user permissions, policies, etc.
  • Performance and Scalability
    • RabbitMQ supports clustered deployment for high availability and throughput.
  • Extensibility, Tools & Plugins
    • RabbitMQ’s flexible plug-in-approach supports extension of functionality through the use of plugins.
    • It provides official client libraries for many programming languages and also various developer tools for supporting frameworks such as the Spring Framework.
  • Documentation
    • The documentation provided by the RabbitMQ developers and community is comprehensive and it covers tutorials and guides from installation, setup and usage of RabbitMQ to developments of new plugins.

High-level Architecture

High-level Architecture Figure 1. High-level Architecture of the Data Spine Message Bus (RabbitMQ)

Figure 1 shows the conceptual components of the Message Bus. At the core of the Message Bus is the ‘PubSub Service’. The PubSub Service provides a Pub/Sub API. The published messages are accepted by the PubSub Service and these messages are stored in queues/buffers provided by the Message Bus until they are forwarded to the designated subscribers. The Message Bus is capable of having multiple topics/channels and also sub-topics over which multiple publishers can publish messages and each topic/sub-topic can have multiple subscribers. In addition, the Message Bus also has an Identity and Access Management component so that the identities of the publishers and subscribers can be verified and their publications and subscriptions can be access controlled. The Message Bus supports use of username-password based authentication. Finally, the Message Bus provides interfaces for user and topic administration, management and monitoring which include an HTTP API, a Management GUI, and also CLI tools such as ‘rabbitmqctl’.

RabbitMQ Operation Figure 2. RabbitMQ Operation

Image source: https://www.cloudamqp.com/blog/part1-rabbitmq-for-beginners-what-is-rabbitmq.html

For more information, read this tutorial.

RabbitMQ Management GUI Figure 3. RabbitMQ Management GUI

Data Spine RabbitMQ Documentation

References

  1. RabbitMQ Documentation: https://www.rabbitmq.com/documentation.html
  2. RabbitMQ for beginners: https://www.cloudamqp.com/blog/part1-rabbitmq-for-beginners-what-is-rabbitmq.html