Skip to main content

Pub/Sub (Publish-Subscribe)

The publish-subscribe model is an asynchronous communication pattern where senders (publishers) do not send messages directly to receivers (subscribers). Instead, messages are sent to an intermediary (topic or message broker), and subscribers receive messages based on their expressed interest in the topics.

This model is commonly used in event-driven systems and microservices architectures to enable decoupled communication between components.

Overview of the Publish-Subscribe Model​

  1. Publisher: The entity that sends messages to a topic.
  2. Topic: A channel or stream where messages are published. Subscribers listen to these topics.
  3. Subscriber: The entity that listens to topics and processes messages sent to them.
  4. Message Broker: The intermediary that facilitates message delivery between publishers and subscribers.

Components of Pub/Sub Communication Design​

Publishers​

  • Producers of events or data.
  • Publish messages to a specific topic.
  • Publishers do not need to know about the subscribers.

Subscribers​

  • Consumers of events or data.
  • Subscribe to specific topics to receive relevant messages.
  • Subscribers can exist in many forms, such as:
    • Microservices
    • Event processors
    • Data pipelines

Message Broker​

  • Facilitates the decoupling of publishers and subscribers.
  • Examples:
    • Kafka, RabbitMQ, Google Cloud Pub/Sub, Amazon SNS, Redis Streams.
  • Handles:
    • Message routing.
    • Persistence (optional, e.g., durable topics in Kafka).
    • Delivery guarantees (e.g., at-least-once, at-most-once, exactly-once).

Message​

  • The data being transmitted between publisher and subscriber.
  • Contains:
    • Payload: The actual data (e.g., JSON, Protobuf).
    • Metadata: Attributes such as message ID, timestamp, priority, etc.

Characteristics of Pub/Sub​

  1. Asynchronous Communication:

    • Publishers send messages without waiting for subscribers to process them.

  2. Decoupling:

    • Publishers and subscribers are unaware of each other’s existence, making the system highly modular and scalable.

  3. Scalability:

    • Multiple publishers and subscribers can interact with the same topic.
    • Horizontal scaling is easier with message brokers.

  4. Event-Driven:

    • Ideal for systems where events trigger downstream actions.

  5. Fan-out Pattern:

    • One message can be delivered to multiple subscribers.

Design Considerations for Pub/Sub​

Topic Design​

  • Use descriptive and hierarchical naming for topics:
    • Example: ecommerce.orders.created, ecommerce.orders.canceled.
  • Avoid creating too many granular topics, which can lead to unnecessary complexity.
  • Use wildcard subscriptions (if supported by the broker) for flexibility (e.g., subscribing to all topics under ecommerce.orders.*).

Message Format​

  • Use a standardized format:
    • JSON: Easy to use but not as compact.
    • Protobuf/Avro: Compact and suitable for high-performance systems.
  • Include metadata such as:
    • eventType: To describe the event.
    • timestamp: To log when the event occurred.
    • correlationId: For tracing across distributed systems.

Delivery Guarantees​

  • Choose a delivery mechanism based on use cases:
    • At-most-once: No retries; suitable for non-critical events.
    • At-least-once: Ensures delivery but may result in duplicates.
    • Exactly-once: Ensures each message is delivered exactly once (e.g., Kafka with idempotent producers/consumers).

Durability​

  • Decide whether messages should persist in the broker:
    • Ephemeral topics: Messages are discarded after delivery.
    • Durable topics: Messages persist until processed or expired.
  • Use retention policies for durable topics (e.g., retain messages for 7 days or until disk space is full).

Scaling​

  • Use partitions to scale topics (e.g., Kafka partitions).
  • Implement consumer groups to scale subscribers horizontally.
  • Design for high throughput by batching messages.

Error Handling​

  • Ensure messages that fail processing are not lost:
    • Use dead-letter queues (DLQs) for unprocessable messages.
    • Retry policies: Implement exponential backoff for retries.

Security​

  • Use encryption in transit (e.g., TLS).
  • Restrict access to topics via role-based access control (RBAC).
  • Authenticate publishers and subscribers using API keys, OAuth, or other mechanisms.

Monitoring​

  • Track message metrics such as:
    • Latency (time between publishing and delivery).
    • Throughput (number of messages per second).
    • Error rates.
  • Use tools like Prometheus, Grafana, or broker-specific monitoring tools.

Advanced Features of Pub/Sub​

Message Filtering​

  • Enable subscribers to filter messages based on attributes, reducing unnecessary processing.
  • Example: Subscribing to a topic with a filter like orderStatus='pending'.

Event Replay​

  • Allow subscribers to replay past events from a topic.
  • Useful for debugging, analytics, or onboarding new services.
  • Example: Kafka allows consumers to rewind to a specific offset.

Priority Messaging​

  • Assign priority levels to messages so that high-priority events are processed first.

Message Ordering​

  • Ensure messages are delivered in the order they were published (e.g., Kafka’s partition-based ordering).

Fan-in Patterns​

  • Multiple publishers send events to the same topic, enabling aggregation of related events.

Schema Registry​

  • Use a schema registry to ensure consistent message formats.
  • Prevent schema evolution issues by validating schemas at runtime.

Common Use Cases​

  1. Event-Driven Microservices:

    • Example: An order.created event triggers notifications, payment processing, and inventory updates.

  2. Real-Time Data Pipelines:

    • Example: Streaming data from IoT devices to analytics systems.

  3. Decoupled Communication:

    • Example: A web application publishing events to a topic that mobile apps subscribe to.

  4. Notification Systems:

    • Example: Sending push notifications to subscribed users.

  5. Data Replication:

    • Example: Replicating database changes across regions using change data capture (CDC) and pub/sub.

Challenges and Solutions​

Challenge: Message Duplication​

  • Solution: Use idempotent message processing and deduplication strategies.

Challenge: Deadlocks or Message Blocking​

  • Solution: Use parallelism in message consumption and design non-blocking systems.

Challenge: High Latency​

  • Solution: Optimize broker configurations, use faster storage, and minimize consumer processing time.

Challenge: Scaling Topics and Subscribers​

  • Solution: Use partitioned topics and load-balance consumers.

Challenge: Debugging Failures​

  • Solution: Use tracing tools (e.g., OpenTelemetry) and implement correlation IDs.

Tools and Frameworks for Pub/Sub​

Message Brokers​

  • Apache Kafka: High throughput, durable message storage.
  • RabbitMQ: Flexible routing and support for different protocols.
  • Google Cloud Pub/Sub: Fully managed and scalable.
  • Amazon SNS/SQS: Serverless message queuing.
  • Redis Streams: Lightweight and fast pub/sub.

Monitoring and Debugging​

  • Prometheus/Grafana: For monitoring broker and consumer metrics.
  • Elastic Stack (ELK): For logging and analytics.
  • Jaeger/Zipkin: For distributed tracing.