Engineering Fluency OS

Why This Matters

Imagine your e-commerce app needs to process an order: charge payment, update inventory, send a confirmation email, and notify the warehouse. If you do all of these synchronously in one request, any failure brings down the whole flow. A message queue decouples these operations. The order service publishes a message, and independent consumers handle each task at their own pace.

Message queues are the backbone of microservice architectures. They enable asynchronous processing, absorb traffic spikes, and let services fail and recover independently. Understanding them is essential for building scalable systems.

Define Terms

Visual Model

Producer 1Order Service

Producer 2Payment Service

Message QueueFIFO, durable

Consumer 1Email

Consumer 2Inventory

Consumer 3Analytics

Dead Letter QueueFailed messages

Publish

Consume

Failed after retries

The full process at a glance. Click Start tour to walk through each step.

Message queues decouple producers from consumers. Failed messages go to a dead letter queue.

Code Example

Code

// Simple message queue implementation

class MessageQueue {
  constructor(name) {
    this.name = name;
    this.messages = [];
    this.deadLetterQueue = [];
    this.subscribers = [];
  }

  // Producer publishes a message
  publish(message) {
    const envelope = {
      id: Math.random().toString(36).substr(2, 9),
      body: message,
      timestamp: Date.now(),
      attempts: 0
    };
    this.messages.push(envelope);
    console.log(`Published: ${JSON.stringify(message)}`);

    // Notify subscribers (pub/sub pattern)
    for (const subscriber of this.subscribers) {
      subscriber(envelope);
    }
  }

  // Consumer pulls next message (point-to-point)
  consume() {
    const message = this.messages.shift();
    if (!message) return null;
    message.attempts++;
    return message;
  }

  // Subscribe to all messages (pub/sub)
  subscribe(handler) {
    this.subscribers.push(handler);
    console.log(`Subscriber added to ${this.name}`);
  }

  // Move to DLQ after max retries
  moveToDeadLetter(message, maxRetries = 3) {
    if (message.attempts >= maxRetries) {
      this.deadLetterQueue.push(message);
      console.log(`Message ${message.id} moved to DLQ`);
      return true;
    }
    // Put back in queue for retry
    this.messages.push(message);
    return false;
  }
}

const orderQueue = new MessageQueue("orders");

// Pub/sub: multiple consumers
orderQueue.subscribe((msg) =>
  console.log(`  Email service: sending confirmation for order ${msg.body.orderId}`)
);
orderQueue.subscribe((msg) =>
  console.log(`  Inventory service: updating stock for order ${msg.body.orderId}`)
);

orderQueue.publish({ orderId: "ORD-001", item: "Laptop", qty: 1 });
orderQueue.publish({ orderId: "ORD-002", item: "Mouse", qty: 2 });

Interactive Experiment

Try these exercises to explore messaging patterns:

Implement a priority queue where high-priority messages are consumed first regardless of arrival order.
Add a visibility timeout: once a consumer takes a message, it becomes invisible to other consumers for 30 seconds. If not acknowledged, it reappears.
Simulate a slow consumer that takes 5 seconds per message while the producer sends 10 messages per second. What happens to the queue?
Implement a message filter where consumers only receive messages matching certain criteria (e.g., only orders over $100).

Quick Quiz

Coding Challenge

Simple Task Queue

Write a class called `TaskQueue` with methods `enqueue(task)` to add a task, `dequeue()` to get the next task (FIFO), and `size()` to return the number of pending tasks. If the queue is empty, dequeue should return null.

Loading editor...

Real-World Usage

Message queues are everywhere in production architectures:

RabbitMQ: A traditional message broker supporting multiple protocols. Commonly used for task distribution and RPC-style communication between services.
Apache Kafka: A distributed event streaming platform. Used for real-time data pipelines, event sourcing, and log aggregation at massive scale.
Amazon SQS: A fully managed message queue service. Provides at-least-once delivery with automatic scaling.
Redis Streams: A lightweight queue built into Redis, useful when you already use Redis for caching.
Celery (Python): A distributed task queue that uses RabbitMQ or Redis as its message broker to process background jobs.

Message Queues