AWS Messaging Compared: SQS, SNS, EventBridge, Kinesis, MSK

AWS has multiple messaging services because there's no one-size-fits-all messaging primitive. Each fits a specific pattern.

SQS — point-to-point queue

• Producer sends messages; one consumer pulls and processes each; once acknowledged with DeleteMessage, it's gone
• Two flavors: Standard (unlimited throughput, at-least-once, no order) and FIFO (ordered, exactly-once-ish, capped at 30K messages per second)
• Use for background job processing, buffering bursts, decoupling services
• The visibility timeout trap: when a message is pulled it's invisible for the timeout (default 30 seconds); if processing exceeds that, the message reappears and another worker picks it up — duplicates; use heartbeat or longer timeout

SNS — fan-out pub/sub

• Publisher sends one message; SNS delivers to N subscribers (SQS queues, Lambda, HTTP endpoints, email, SMS)
• Subscribers don't know about publishers; classic for notifying multiple downstream services
• The legacy pattern: SNS → SQS for fan-out to multiple queues; EventBridge often better today because of filtering

EventBridge — event bus with content-based routing

• Publishers send events; rules match events by JSON patterns and route to targets (Lambda, Step Functions, SQS, SNS, third-party SaaS)
• Schema registry tracks event shapes — good practice for cross-team event contracts
• The modern default for event-driven architecture; replaces SNS → SQS fan-out with smarter content-based routing
• Limitation: rules match individual events; no aggregation (use Kinesis or Step Functions if you need "trigger after N events")

Kinesis Data Streams — ordered stream with replay

• Records partitioned by shard; multiple consumers can each maintain their own offset/checkpoint
• Records persist 24 hours up to 1 year, enabling replay from a specific point
• Use for clickstream, log aggregation, real-time analytics, IoT telemetry
• Shard limits: 1MB/s write, 2MB/s read, 1000 records/s per shard; hot partition is bottleneck; design partition keys carefully

MSK — managed Kafka

• Same Kafka API, AWS handles brokers, ZooKeeper or KRaft, patches
• Use when you need Kafka specifically: ecosystem (Connect, Streams, ksqlDB), exactly-once with transactions, multi-cloud portability, existing Kafka skills
• Higher operational complexity than Kinesis but better economics at huge scale and richer feature set

The decision tree

• Background jobs, queue pattern, one consumer per message: SQS
• Notify multiple destinations of one event with no filtering: SNS
• Modern event-driven architecture with content-based routing: EventBridge
• Ordered stream with multiple consumers and replay: Kinesis
• Need Kafka API specifically: MSK
• Migrating from on-prem ActiveMQ/RabbitMQ: MQ (otherwise don't pick)

Hidden trade-offs

• SNS pricing is per delivery, not per publish — fan-out to 1000 subscribers is 1000 charged deliveries
• EventBridge has 300 rules per bus default; complex routing across hundreds of event types may hit limits
• Kinesis charges per shard-hour even when idle; over-provisioning gets expensive
• Kinesis Firehose vs Streams — Firehose lands data in S3/Redshift/OpenSearch with no consumer code; Streams needs consumers but allows custom real-time processing
• DLQs are mandatory across all of these — configure from day one to handle unprocessable messages, otherwise you lose them or block forever

Default recommendation

• New event-driven architecture: EventBridge as the bus, SQS for work queues, Lambda or ECS as targets
• Stream processing: Kinesis Data Streams for AWS-native, MSK if you specifically need Kafka
• Avoid: SNS → SQS fan-out for new projects (use EventBridge), MQ (legacy migration only)