Message Queues: Kafka and RabbitMQ

Message Queues in Modern Architecture #

Microservices communicate asynchronously through message queues and event streaming platforms. Instead of Service A calling Service B directly (synchronous HTTP), Service A publishes a message to a queue, and Service B consumes it when ready. This decouples services and improves resilience.

Two dominant technologies serve this purpose:

RabbitMQ — A traditional message broker. Messages are routed through exchanges to queues. Once a consumer acknowledges a message, it is removed from the queue. Best for task distribution and request/reply patterns.

Apache Kafka — A distributed event streaming platform. Messages (events) are appended to topic partitions and retained for a configurable period. Consumers track their position (offset) and can replay events. Best for event sourcing, data pipelines, and high-throughput streaming.

RabbitMQ Concepts for Testers #

Exchanges, Queues, and Bindings #

Producer → Exchange → Binding → Queue → Consumer

Key Test Scenarios for RabbitMQ #

1. Message delivery:

• Producer sends a message — does it arrive in the correct queue?
• Message with routing key “order.created” — does it reach the “orders” queue but not the “payments” queue?

2. Consumer acknowledgment:

• Consumer processes message and sends ACK — is the message removed?
• Consumer crashes without ACK — is the message redelivered?
• Consumer sends NACK (negative acknowledgment) — is the message requeued or sent to DLQ?

3. Dead letter queue:

• Message fails processing 3 times — does it move to the DLQ?
• DLQ contains the original message with failure metadata?

4. Message TTL:

• Message with a 60-second TTL expires — is it removed or dead-lettered?

# Python example: Testing RabbitMQ with pika
import pika
import json
import time

connection = pika.BlockingConnection(
    pika.ConnectionParameters('localhost')
)
channel = connection.channel()

# Declare queue with DLQ
channel.queue_declare(
    queue='orders',
    arguments={
        'x-dead-letter-exchange': '',
        'x-dead-letter-routing-key': 'orders.dlq',
        'x-message-ttl': 60000,
    }
)
channel.queue_declare(queue='orders.dlq')

# Publish a message
channel.basic_publish(
    exchange='',
    routing_key='orders',
    body=json.dumps({'order_id': 123, 'amount': 99.99}),
    properties=pika.BasicProperties(
        delivery_mode=2,  # persistent
        content_type='application/json',
    )
)

# Consume and verify
method, properties, body = channel.basic_get('orders', auto_ack=False)
assert method is not None, "No message in queue"
order = json.loads(body)
assert order['order_id'] == 123
channel.basic_ack(method.delivery_tag)

Kafka Concepts for Testers #

Topics, Partitions, and Consumer Groups #

Producer → Topic (Partition 0, 1, 2, ...) → Consumer Group → Consumers

Key Test Scenarios for Kafka #

1. Message production:

• Producer sends an event — does it appear in the correct topic and partition?
• Key-based partitioning — do events with the same key always go to the same partition?

2. Message ordering:

• Events within a single partition maintain order — verify by producing events A, B, C and consuming in the same order.
• Events across partitions have no ordering guarantee — verify this behavior.

3. Consumer groups:

• Two consumers in the same group — each partition is consumed by exactly one consumer.
• Two consumers in different groups — both receive all messages (pub/sub pattern).

4. Consumer lag:

• Producer sends 1,000 messages, consumer processes 500 — consumer lag should be 500.
• This is a critical metric for monitoring production systems.

5. Exactly-once semantics:

• Producer with idempotent writes — duplicate produces result in only one message in the topic.
• Transactional producer — either all messages in a batch are committed or none.

# Python example: Testing Kafka with kafka-python
from kafka import KafkaProducer, KafkaConsumer
import json

# Producer
producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    key_serializer=lambda k: k.encode('utf-8') if k else None,
)

# Send event with key (ensures partition ordering)
producer.send(
    'orders',
    key='user-123',
    value={'order_id': 456, 'status': 'created'}
)
producer.flush()

# Consumer
consumer = KafkaConsumer(
    'orders',
    bootstrap_servers='localhost:9092',
    group_id='order-processor',
    auto_offset_reset='earliest',
    value_deserializer=lambda m: json.loads(m.decode('utf-8')),
    consumer_timeout_ms=5000,
)

messages = []
for message in consumer:
    messages.append(message.value)
    if len(messages) >= 1:
        break

assert messages[0]['order_id'] == 456
assert messages[0]['status'] == 'created'

Testing Message Serialization #

Messages must be serialized (to bytes) and deserialized correctly. Common formats:

Schema evolution test: If you change the message schema (add/remove fields), verify that existing consumers can still process old messages and new consumers can handle both formats.

Testing with Docker Compose #

Set up local message infrastructure for testing:

services:
  rabbitmq:
    image: rabbitmq:3-management
    ports:
      - "5672:5672"
      - "15672:15672"

  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.5.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    depends_on:
      - zookeeper

Exercise: Message Queue Testing Lab #

Part 1: RabbitMQ Testing #

Start RabbitMQ with Docker and test the following scenarios:

docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management

Task 1.1: Exchange routing

Create a topic exchange called “events” with two queues:

• “orders_queue” bound with routing key “order.*”
• “payments_queue” bound with routing key “payment.*”

Publish messages and verify routing:

1. Message with key “order.created” → should appear in orders_queue only.
2. Message with key “payment.completed” → should appear in payments_queue only.
3. Message with key “user.registered” → should appear in neither queue.

Task 1.2: Dead letter queue

Configure the orders queue with a DLQ:

1. Set max delivery attempts to 3.
2. Publish a message that your consumer intentionally rejects (NACK).
3. Verify the message is retried 3 times, then moved to the DLQ.
4. Inspect the DLQ message — does it contain the original payload and rejection metadata?

Task 1.3: Message persistence

1. Publish 10 messages to a durable queue with persistent delivery mode.
2. Restart the RabbitMQ container.
3. After restart, verify all 10 messages are still in the queue.

Part 2: Kafka Testing #

Start Kafka with Docker Compose and test:

Task 2.1: Partition ordering

1. Create a topic “test-orders” with 3 partitions.
2. Produce 10 events with key “user-A” and 10 events with key “user-B”.
3. Consume all events and verify:
   • All “user-A” events are in order relative to each other.
   • All “user-B” events are in order relative to each other.
   • Events may interleave between users (different partitions).

Task 2.2: Consumer groups

1. Start two consumers in the same group (“order-processors”).
2. Produce 100 messages to a topic with 3 partitions.
3. Verify that each consumer processes a subset of messages (partition assignment).
4. Stop one consumer and verify the other takes over all partitions (rebalancing).

Task 2.3: Consumer lag monitoring

1. Produce 1,000 messages to a topic.
2. Start a slow consumer (with 100ms delay per message).
3. Use kafka-consumer-groups.sh --describe to monitor consumer lag.
4. Record how lag changes over time as the consumer catches up.

# Check consumer lag
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-processors

Part 3: Comparison Report #

After completing both parts, write a comparison:

Include your observations about testing complexity, setup effort, and debugging tools for each platform.