Messaging Systems: Kafka vs RabbitMQ vs SQS vs Pulsar
Fast recall
Quick revision
Core answer
Deep understanding of messaging system trade-offs, not just surface features
Defaulting to Kafka for everything (overkill for simple task queues); defaulting to RabbitMQ (doesn't scale to event streaming); ignoring managed options
Discuss pull vs push, commit log vs queue, consumer groups vs competing consumers, message ordering guarantees, backpressure, durability, operational complexity, and cost models
Interview-ready guide
Interview answer
Four dominant messaging systems, each with a different architectural philosophy. Picking wrong causes scaling walls, unexpected ordering bugs, operational burden, or cost overruns.
Kafka — distributed commit log
- Append-only log partitioned across brokers
- Pull-based: consumers track their own offset
- Retention for days or weeks
- Multiple consumer groups read the same data independently
- Very high throughput: millions per second per broker
- Best for event streaming, analytics pipelines, change data capture
- Operational complexity: Zookeeper or KRaft, partition management, consumer lag monitoring
RabbitMQ — traditional broker
- Exchanges, queues, bindings; rich routing (direct, topic, fanout, headers)
- Push-based with prefetch limit
- Excellent for task queues and RPC patterns
- Tens of thousands per second per node; not millions like Kafka
- Messages deleted after consumer ack
- Best for flexible routing and lower-volume workloads
SQS — AWS managed queue
- Zero ops; fully managed
- Standard queue: at-least-once, no ordering, massive throughput
- FIFO queue: exactly-once, per-group ordering, 3000 per second per group
- Simple API: SendMessage, ReceiveMessage, DeleteMessage
- Cost per message: $0.40 per million Standard, $0.50 FIFO
- Best for AWS-native task queues; cheap at low and medium volume
Pulsar — unified streaming + queuing
- Stateless brokers plus BookKeeper for storage; independent scaling
- Supports both streaming (Kafka-like) and queuing (RabbitMQ-like) via subscription types
- Tiered storage: cold data to S3 cheaply
- Multi-tenancy first-class
- Best for platforms needing both models and long retention
Key architectural differences
Pull vs push
- Kafka, SQS, Pulsar are pull: natural backpressure, consumer controls rate
- RabbitMQ is push with prefetch: lower latency but slow consumers can be overwhelmed if prefetch too high
Consumer model
- Kafka consumer group: each partition assigned to one consumer; parallelism limited by partitions; ordering per partition
- RabbitMQ competing consumers: all pull from queue; ordering lost across consumers
- SQS: similar to competing consumers
- Pulsar: exclusive, shared, key-shared, failover subscriptions
Ordering
- Kafka: per-partition total order; cross-partition no guarantee
- RabbitMQ: per-queue FIFO with single consumer; breaks with competing
- SQS Standard: no ordering; FIFO: per message-group-id
- Pulsar: per-partition
Retention and replay
- Kafka: time or size based retention; replay by seeking to offset
- RabbitMQ: deleted after ack; stream plugin adds limited replay
- SQS: max 14 days; no replay
- Pulsar: tiered storage for months cheaply
Decision framework
Pick Kafka when
- Event-driven architecture, 100K+ messages per second
- Need replay for reprocessing
- Multiple consumer groups on same stream
- Change data capture with Debezium
- Ecosystem: Kafka Connect, Schema Registry, Kafka Streams
Pick RabbitMQ when
- Task queues with flexible routing
- RPC patterns with reply-to
- Polyglot: AMQP clients everywhere
- Lower throughput but flexibility matters more
Pick SQS when
- On AWS, want zero ops
- Task queue model fits
- Lambda triggers natively
- Cost per message acceptable
Pick Pulsar when
- Both streaming and queuing in one system
- Multi-tenant SaaS
- Extreme retention with tiered storage
- Geo-replication requirements
Exactly-once reality
- Kafka's exactly-once is within Kafka only; doesn't extend to external DB writes
- SQS FIFO has 5-minute dedupe window via deduplication ID
- Pulsar similar to Kafka
- Always combine at-least-once delivery with idempotent consumers for true effectively-once
Cost models
- Kafka: $0 license, high ops cost; Confluent Cloud expensive at scale
- RabbitMQ: $0 license, lower ops; CloudAMQP or AWS MQ managed
- SQS: pay per request; cheap low/medium, expensive at billions
- Pulsar: complex to self-host; StreamNative managed SaaS
The hybrid reality
- Most orgs run multiple systems
- Kafka for event streaming and CDC
- SQS for AWS-native background tasks
- RabbitMQ for legacy RPC
- NATS for ultra-low-latency internal
The strategic answer
- No one-size-fits-all; pick based on throughput, retention, routing, ops budget
- Start simple; evolve
- On AWS, small team: SQS
- Event streaming strategic: Kafka
- Flexible routing: RabbitMQ
- Multi-tenant or very long retention: Pulsar
Messaging Systems: Kafka vs RabbitMQ vs SQS vs Pulsar
Direct Answer
Four dominant messaging systems, each optimized differently: Kafka is a distributed commit log — pull-based, high throughput, durable event streaming, replay; best for events, analytics pipelines, CDC. RabbitMQ is a traditional broker — push-based (with prefetch), flexible routing (exchanges), great for task queues and RPC; weaker for high throughput. SQS is fully managed AWS — zero ops, at-least-once, no ordering (standard) or FIFO queues, great for task queues on AWS. Pulsar combines Kafka-style streaming with RabbitMQ-style queuing, has tiered storage and multi-tenancy. Rule of thumb: Kafka for events at scale, RabbitMQ for flexible routing + RPC, SQS for AWS-native task queues, Pulsar if you need both streaming and queuing in one system.
Common Mistakes & Pitfalls
Defaulting to Kafka for everything (overkill for simple task queues); defaulting to RabbitMQ (doesn't scale to event streaming); ignoring managed options
Check your understanding
Answer these without looking back at the article.
- How does Kafka's exactly-once semantics work, and why doesn't it extend to external DB writes?
- What is Kafka's consumer group protocol and how does partition rebalancing work?
- How does RabbitMQ's prefetch count affect throughput and backpressure?
- When would you choose Amazon Kinesis over Kafka on AWS?
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.