Everything You Need To Know About Publish/Subscribe

The Publish/Subscribe pattern, also known as pub/sub, is an architectural design pattern that provides a framework for exchanging messages between publishers and subscribers. This pattern involves the publisher and the subscriber relying on a message broker that relays messages from the publisher to the subscribers. The host (publisher) publishes messages (events) to a channel that subscribers can then sign up to.

Although Pub/Sub is based on earlier design patterns like message queuing and event brokers, it is more flexible and scalable. The key to this is the fact Pub/Sub enables the movement of messages between different components of the system without the components being aware of each other’s identity.

This in-depth tutorial explains the fundamental concepts behind the Pub/Sub pattern. It presents scenarios where Pub/Sub is a good fit, with information about its general advantages and its potential pitfalls. Finally, the tutorial explores a few open-source and commercial implementations of Pub/Sub, and ways you can optimize these.

Realtime you can depend on. Ably allows you to easily build complete, simplified realtime applications that scale. Try our APIs for free.

Copy link to clipboardContext – Pub/Sub for Dynamic Scaling

The Pub/Sub pattern evolved out of the necessity to expand the scale of information systems. In the pre-Internet era, and even during the early days of the Internet, the systems were mostly scaled statically. However, with the expansion of the Internet and web-based applications, fueled by the massive adoption of mobile and IoT devices, systems needed to scale dynamically.

To understand the difference between dynamic scaling and static scaling, think of a telephone exchange. If there are 100 telephone subscribers, the exchange needs 100 input lines. When the request for new connections arrive, the exchange needs to add more input lines manually.

Even though all subscribers won’t use their telephones at once, the telephone exchange has to maintain one input line for every subscriber. This is static scaling.

However, a cloud telephone system can allocate servers based on traffic. It can increase or decrease server resources to handle the change in traffic. This is dynamic scaling, and it uses resources more efficiently than static scaling.

The design of information systems has also undergone a similar evolution in terms of scaling.

Earlier, the components of a system were:

  • Subroutines within a program.

  • Multiple processes running on a computer.

  • Daemon services running on a few servers.

The components were connected statically (mostly through hard-coding), either through IPC (Inter-Process Communication) mechanism or local/dedicated network connections.

Today, typical systems tend to work at Internet-scale around geographically distributed data centers. Their servers host ephemeral application modules for handling user traffic. Seasonal traffic spikes, latency, and network data corruption add more layers of complexity to information systems dealing with today’s volume of data.

The decoupled nature of the Pub/Sub pattern makes it a good candidate for governing the architecture of dynamically scalable systems. Pub/Sub makes it possible to manage scale without overloading the program logic of system components. Let’s look at exactly how Pub/Sub achieves this.

Copy link to clipboardHow Does The Publish-Subscribe Pattern Work?

Software design patterns are based on building reusable arrangements of modules and their interconnections. These modules are typically classes or objects represented in a UML design diagram. However, when you look at modern architectural patterns, the modules are larger, self-executing processes spread across distributed systems.

To appreciate the advantages of the Pub/Sub pattern, you must start from the basic pattern upon which an information system is built and follow its evolution towards a distributed system.

Typically, an information system is an assembly of a generalized set of software modules that follow this simple sequential pattern.

Sequential pattern of a set of software modules.

Think of the illustration above as simple software consisting of three modules. The input module takes the user input and sends the data to the processing module in the form of a message. The processing module processes the data and sends it to the output module as yet another message. The output module displays the data on the user’s screen.

However, the real world is never that simple. At a reasonable scale, the system will need multiple input & output modules for handling concurrent requests.

A system needs multiple input & output modules for handling concurrent requests.

At this scale, the system faces the problem of routing messages from input modules to their respective output modules. To solve this problem, the input and output modules will need an addressing mechanism. The processing module will process the messages and route them to the correct recipient based on an address. All three modules collaborate in solving the problem of routing.

All three modules collaborate in solving the problem of routing.

At Internet scale, the system will handle thousands of concurrent connections. The system will receive messages from and send messages to users all over the world. It needs to also be capable of handling high volume and global geographical spread of users.

System receiving messages from and sending messages to users all over the world.

However, at such a massive scale, the system modules will not work as expected.

  • The processing module can’t handle the load. Because of the high volume and geographical spread, the load needs to be distributed between multiple processing modules.

  • At this scale, the dynamics of input and output change. Pre-defined addressing between the modules becomes a huge overhead.

You can solve the first problem by introducing multiple processing modules. This has the effect of splitting the system horizontally. However, this increases routing complexity. The input modules must now route the messages to the correct processing module.

Attaching module specific routing metadata to messages becomes a bottleneck at internet-scale. Under such circumstances, the design of message passing from one module to the next begs a radical rethink.

Copy link to clipboardEnter Pub/Sub

Having to program the modules to maintain a shared knowledge of addresses for other modules is burdensome for developers. The complexity of this dependency will increase with scale and it will eventually break the system.

The best way to ease this burden is to minimize the shared knowledge of addresses. To achieve this, you can tweak the design of the modules such that they perform their tasks and then interact through a common forum.

You can think of this common forum as a channel or a data pipe. The modules can post their messages to it or retrieve the messages posted by other modules from it.

Pub Sub architecture

With this architecture, a developer can program the modules to have isolated and well-defined responsibilities, so the modules no longer need to maintain shared knowledge on the whereabouts of other modules. The input modules only accept user input, processing modules only process the data, and the output modules only display the output.

The only additional thing the modules need to know is the channel for posting and retrieving messages. The input module will gather the user input and post the message in the pre-processing channel. The processing module will pick the messages from this channel, process it and post it to the post-processing channel. Finally, the output module will collect the message from the post-processing channel and display it on the users’ screen.

The same pattern can be followed at any scale.

Pub Sub at scale

But what about the original routing problem? How do the input modules know about their corresponding output modules or vice versa?

The short answer is, they don’t need to know. The problem of a one-to-one input-output address mapping is now transformed to posting to a channel and retrieving from a channel. Therefore, you are now witnessing a change in the paradigm of communication from point-to-point information exchange to information sharing on a common forum.

Information sharing on a common forum.

This approach of sharing information forms the fundamental repeat unit of the Pub/Sub pattern.

To recap on the initial overview, in the Pub/Sub parlance, you call the module that posts messages a publisher. You call the module that retrieves messages from a subscriber. The publisher publishes messages on a topic. The subscriber subscribes to the topic to receive messages. You can now assemble the generalized information system as shown below.

Publisher Subscriber in action

The input module publishes the messages on the pre-processing topic. The processing module also subscribes to this topic to receive all messages published by the input module. The processing module further acts as the publisher of processed messages on the post-processing topic. The output module subscribes to this topic.

The topics are like virtual pathways. You can create and destroy them on the fly. This makes the management and administration of topics a separate responsibility, abstracted from the modules. As a result, a developer is not faced with additional complexity in programming the modules, even at scale.

The responsibility of managing the topics is now entirely shifted to the message broker. The message broker is an independent component and has its own implementation for the administration of published messages and their delivery to the subscribers. As Pub/sub is a design pattern, it does not address these implementation details.

Copy link to clipboardWhen Should You Use the Publish-Subscribe Pattern?

Chat applications are a classic use case of the Pub/Sub pattern. In a chat application, participants can subscribe to chat rooms which have a designated Pub/Sub topic.

When a user sends a message to a chat room, her chat app instance publishes the message on that chat room’s topic. Subscribers of the topic receive the message.

Chat applications are a classic use case of the Pub/Sub pattern

An example of how you might add Pub/Sub functionality to a chat app using Ably’s Realtime SDK:

When the app launches, the SDK initializes and subscribes to the topic that represents a public chat room.

var realtime = new Ably.Realtime('api-key');
// Obtaining a channel instance
var chatRoomTopic = realtime.channels.get('channel-name');

chatRoomTopic.subscribe(function(message) {
  console.log('message received for event ' + message.name);
  console.log('message data:' + message.data);

Subsequently, when the user wants to send a chat message, the chat app publishes the message on the same topic.

chatRoomTopic.publish('event', 'This is my payload', function(err) {
  if(err) {
    console.log('Unable to publish message; err = ' + err.message);
  } else {
    console.log('Message successfully sent');

The app unsubscribes from the channel when the user logs out or leaves the chat room.

Pub/Sub in a chat app using Ably’s Realtime SDK.

Let’s explore the case where two users want to have a private chat.

Since chat app instances do not interact directly, it would seem at first glance that a complicated workaround is required, but this isn’t the case. Just as in public chats, private chats also take place over a topic. In a private chat, the two chat applications open up a temporary topic. The topic is named by a predefined logic, for example, by combining the user ids.

Topic for private chat and topic for public chat

The coordination for opening up a temporary topic can be achieved by yet another common topic which is reserved only for publishing the private chat requests.

Copy link to clipboardAdvantages of Using Publish-Subscribe Pattern

Building an information system at scale using the Pub/Sub pattern benefits all stakeholders. The software development process benefits from the simplicity of the Pub/Sub pattern. Let’s look at these advantages from the software architects’, developers’, and testers’ points of view.

Copy link to clipboardSoftware Architects

  • Loose Coupling Between System Components: The Pub/Sub pattern decouples communication logic and business logic, allowing isolation of components. Isolated components make the system robust and keep the code maintainable.

  • Better View of the System-wide Workflow: The premise of the Pub/Sub pattern is simple, so Pub/Sub systems are easier for software architects to refactor and expand.

  • Enables Better & Faster Integration: Pub/Sub is programming language and communication protocol agnostic, which allows disparate components of a system to be integrated faster compared to legacy alternatives.

  • Ensures Smoother Scalability: Systems using Pub/Subs are scalable without the fear of breaking functionality because communication logic and business logic separate entities. Software architects can redesign the message broker’s topic architecture without the worry of breaking the business logic.

  • Guaranteed Consistent Reliability: The Pub/Sub pattern also helps in keeping the system reliable when features change. Pub/Sub supports asynchronous message delivery, which ensures reliable message delivery even when the message broker’s topic architecture changes.

  • Builds Elasticity: Pub/Sub based systems are elastic because the business logic does not depend on the number of active publishers and subscribers at an instance. In other words, Pub/Sub based systems can accommodate a surge in usage.

Copy link to clipboardDevelopers

  • Software Modularization: Systems built on Pub/Sub allows developers to split the system into modules based on the system’s business logic. Modularization and separation of concerns leads to better code.

  • Language Agnostic Software Development: The Pub/Sub pattern is used to make software development language agnostic. The application is broken down into smaller, logical parts. Parts are programmed using a suitable programming language. The inter-communication between the parts takes place via Pub/Sub. In effect, Pub/Sub stitches the components into a single working software.

  • Clarity in Business Logic: The message broker takes the responsibility of reliably delivering the message, thus freeing the developer from writing additional code. Codebase becomes simpler and reader-friendly.

  • Improves Responsiveness: In the Pub/Sub software pattern, communication is asynchronous. It allows the modules to be also programmed asynchronously. Message delivery does not block the publisher. The publisher gets back to its task after publishing the message. Similarly, the subscriber gets interrupted only when there is a message on a subscribed topic.

Copy link to clipboardSoftware Testers

  • Streamlined Testing Practices: Pub/Sub makes testing modular because business logic isn’t coupled to message broker.

  • Improved Debugging: In a Pub/Sub based system, topics channel messages from publishers to subscribers. It is easier to monitor and inspect topics for debugging integration issues between modules.

  • Better Unit Testing Framework: In a Pub/Sub system, a component under unit test cares only about the topic and messages on that topic. It does not care about external components at all. So, you can create a generic stub. The generic stub acts as an external component sending messages on a topic. Further, it can be replicated for all components which is a better approach to building unit testing frameworks than having separate stubs for each component.

Copy link to clipboardSituations Where Publish-Subscribe Pattern is a Bad Choice

Every design pattern has limitations and trade-offs. There are scenarios where Pub/Sub is the wrong choice:

  • Overkill for simpler systems: Pub/Sub is an overkill for simple system which are unlikely to scale up. If you’re operating systems where elastic scaling is not required and where static scaling will suffice, think twice before using Pub/Sub.

  • Not Suitable for Media Streaming: Audio and video streaming have nuanced requirements for smooth rendering on the user’s end. Synchronous point-to-point communication between the two endpoints is the best solution for media streaming. Pub/Sub is not suitable for carrying VoIP or video telephony traffic over the Internet. However, you can still use Pub/Sub to ring the phones.

  • Inappropriate for Periodic/Background Tasks: Pub/Sub is an asynchronous method of sending information. Therefore, it is not suitable for systems that run as periodic background tasks, like cron jobs which are triggered over a particular time period

Copy link to clipboardSome Well-known Messaging Frameworks Based on Publish-Subscribe Pattern

Since Pub/Sub pattern is a generic guideline, it stops short of stating the granular implementation details of the message broker.

If you want to explore a few messaging frameworks which follow Pub/Sub’s topic-based publishing and subscribing philosophy, then you have a few options.

  • Apache Kafka: Apache Kafka is a popular and robust messaging which has the usual Pub/Sub features. Apache Kafka is also designed to include a message log. Subscribers can step back and forth in time to ‘replay’ messages.

  • RabbitMQ: RabbitMQ is a message queue. It doesn’t strictly follow the Pub/Sub pattern. However, we can configure it for a direct or a fan-out message exchange between two or more components of the system.

  • PushPin: PushPin is an open-source, real-time messaging framework that is suitable for sending real-time push notifications. It is meant to be deployed at the edges of a distributed system and uses the standard protocols like WebSocket to push messages. It is suitable for low throughput applications.

Some other popular protocols like the Message Queue Telemetry Transport ( MQTT ) also follows the Pub/Sub pattern. MQTT is an ISO protocol for messaging between IoT devices. Mosquitto is one of the well known open-source MQTT broker.

Copy link to clipboardPublish-Subscribe at Scale

In a real world system, all modules are stitched together via a sequence of publishers and subscribers which are linked to topics. There can be single or multiple levels of topics based on the load and scale. This forms the overall topic fabric of the system which is managed by a cluster of message brokers.

Publish-Subscribe at Scale

Despite the many virtues of Pub/Sub, it is crucial to choose the right infrastructure for message broker implementation. Otherwise it can hinder scaling and create reliability issues. Apart from scale and reliability, there are several design considerations for choosing a suitable message broke infrastructure:

  • Latency: The latency of message transfer is the most important consideration for building real-time applications. The addition of a message broker to the existing messaging infrastructure should not adversely impact the latency.

  • Bandwidth: You have to consider the maximum message payload size that can be handled by the message broker. Message broker splits larger payloads into smaller ones, introducing latency and computing overhead.

  • Message Handling: Based on application requirements, you may need to evaluate whether the message broker honors the sequence of published messages and preserves its integrity.

  • Service Availability: To handle the load based on criteria like geographical proximity, time of day, and usage patterns, etc., message brokers should be deployed as a cluster for achieving high availability. An additional software architecture is required to manage these message broker clusters at Internet scale to ensure high availability.

  • Service Reliability: Message brokers should honour the published messages and ensure their idempotency across the topics even in the case of a failure of single broker. They should be able to synchronize and swap the messages and topics in the case of failure. The infrastructure should have fault handling policies to ensure message delivery.

  • Security: The inherent mechanism of passing messages through shared topics creates security challenges. The message broker should support a granular access control mechanism such that it permits only authorized components to publish on or subscribe to a topic.

A Pub/Sub messaging infrastructure can be self-deployed or deployed on a pre-configured “as-a-service” cloud infrastructure.

Messaging frameworks described earlier are offered as stand-alone software packages. If you want to self deploy, then you need to setup server infrastructure, install a framework, and configure it as per your design considerations addressed above. However, you can opt for a managed “as-a-service” Pub/Sub infrastructure as well.

Advantages of managed “as-a-service” deployment over self-deployment are:

  • Development Time: Pub/Sub isolates application development from the messaging infrastructure. Therefore using an “as-a-service” infrastructure reduces development time.

  • System Tuning: Self-deployment might seem easy, but fine-tuning, security, and design considerations require lots of work. Managed infrastructure comes pre-configured, and you can tune it via a service interface.

  • Programming Options: Managed services support popular programming languages and frameworks. Whereas, message broker frameworks support only a few languages. Building and maintaining SDKs for your own message broker is a deviation of the development effort and time.

  • Cost: Most “as-a-service” models offer pay as you go option. Self-managed infrastructure requires upfront capital expenditure for setting up data centers and servers.

Copy link to clipboardAbly and pub/sub

Ably is an enterprise-ready pub/sub messaging platform. We make it easy to efficiently design, quickly ship, and seamlessly scale critical realtime functionality delivered directly to end-users. Everyday we deliver billions of realtime messages to millions of users for thousands of companies.

We power the apps people, organizations, and enterprises depend on everyday like Lightspeed System’s realtime device management platform for over seven million school-owned devices, Vitac’s live captioning for 100s of millions of multilingual viewers for events like the Olympic Games, and Split’s realtime feature flagging for one trillion feature flags per month.

We’re the only pub/sub platform with a suite of baked-in services to build complete realtime functionality such as showing a driver’s live GPS location on a home-delivery app, instantly loading the most recent score when opening a sports app, and automatically handling reconnection when swapping networks.

Our platform is mathematically modeled around Four Pillars of Dependability so we’re able to ensure messages don’t get lost while still being delivered at low latency over a secure, reliable, and highly available global edge network.

Try our APIS for free and see why developers from startups to industrial giants choose to build on Ably because they simplify engineering, minimize DevOps overhead, and increase development velocity.

Copy link to clipboardFurther Reading

Matthew O’Riordan

Matthew is the CEO and co-founder of Ably. He is a realtime advocate and open data and open source supporter. He is a serial entrepreneur and seasoned developer with over 20 years of hands-on development experience. He is also a governing board member of the OpenAPI initiative. Matthew co-founded and was CTO of Econsultancy, a global digital marketing subscriptions business, which was sold in a trade sale in 2012.