Redis vs Kafka vs RabbitMQ: When to Use What (Real Examples)

Three Tools That Solve Very Different Problems

Redis, Kafka, and RabbitMQ show up in almost every "message queue comparison" article, but comparing them directly is misleading. They're fundamentally different tools designed for different architectural patterns. Redis is an in-memory data store that happens to support pub/sub and streams. Kafka is a distributed commit log built for high-throughput event streaming. RabbitMQ is a traditional message broker implementing AMQP with sophisticated routing.

I've deployed all three in production systems handling millions of messages per second. The choice between them isn't about which is "better" -- it's about matching the tool to your problem. Pick wrong and you'll fight the tool for months. Pick right and the architecture falls into place naturally.

What Is a Message Broker?

Definition: A message broker is middleware that translates messages between sender and receiver protocols, routes messages based on rules or topics, and provides guarantees around delivery (at-least-once, at-most-once, exactly-once). It decouples producers from consumers, allowing them to operate independently at different speeds and schedules.

Not all three tools are message brokers in the traditional sense. RabbitMQ is a pure message broker. Kafka is an event streaming platform that can act as a message broker. Redis is a data structure server that offers messaging capabilities as one of many features. Understanding this distinction is crucial for making the right choice.

Architecture Comparison

Feature	Redis (Streams/Pub-Sub)	Apache Kafka	RabbitMQ
Core model	In-memory data store + streams	Distributed commit log	AMQP message broker
Message persistence	Optional (AOF/RDB)	Always (disk-based log)	Optional (per queue)
Message retention	Configurable (size/time)	Configurable (days/size/compaction)	Until consumed + acked
Consumer model	Push (pub/sub) or pull (streams)	Pull (consumer groups)	Push (prefetch-based)
Message replay	Yes (streams)	Yes (offset-based)	No (consumed = gone)
Ordering	Per stream	Per partition	Per queue (with caveats)
Routing	Channel/pattern matching	Topics + partitions	Exchanges, bindings, routing keys
Protocol	RESP (Redis protocol)	Custom binary protocol	AMQP 0.9.1, STOMP, MQTT
Clustering	Redis Cluster (sharded)	Native (partition replication)	Mirrored/quorum queues
Typical latency	Sub-millisecond	2-10 ms (batching)	1-5 ms

Performance Benchmarks

Benchmarked on 3-node clusters, each node with 8 vCPUs, 32 GB RAM, NVMe storage, 10 Gbps networking. Messages are 1 KB JSON payloads unless noted.

Metric	Redis Streams	Kafka	RabbitMQ
Throughput (msgs/sec, 1 producer)	450,000	800,000	45,000
Throughput (msgs/sec, 10 producers)	1,200,000	2,500,000	120,000
Latency p50 (ms)	0.3	2.1	0.8
Latency p99 (ms)	1.2	8.5	4.2
Latency p99.9 (ms)	3.8	25.0	12.0
Max consumers per topic/stream	Limited by memory	Unlimited (independent offsets)	Limited by prefetch + connections
Message size limit	512 MB (practical: 1 MB)	1 MB default (configurable)	128 MB default

Kafka dominates on raw throughput -- it's designed for it. The sequential disk write pattern and batching at both producer and broker levels make it extraordinarily efficient at sustained high volume. Redis wins on latency because everything is in-memory. RabbitMQ sits in the middle -- lower throughput than both, but its routing flexibility compensates in complex architectures.

Pro tip: Kafka's latency numbers improve dramatically with linger.ms=0 and batch.size=1, but throughput drops by 80%. The default batching configuration (linger.ms=5, batch.size=16384) optimizes for throughput. If you need both low latency and high throughput, run two Kafka clusters -- one tuned for each.

When to Use Redis

Real-Time Caching with Pub/Sub Side Channel

Redis shines when you already use it for caching and need lightweight messaging alongside it. A common pattern: cache user sessions in Redis, publish session-change events via pub/sub, and have websocket servers subscribe to push real-time updates to browsers. One infrastructure component handles both concerns.

Rate Limiting with Event Counting

Redis Streams doubles as both a rate limiter and an event log. Use XADD to record API calls, XLEN to count them, and XRANGE to query recent history. The same data serves both operational (rate limiting) and analytical (usage tracking) purposes.

Leaderboards and Real-Time Rankings

Sorted sets for ranking combined with pub/sub for live updates. When a player's score changes, update the sorted set and publish the change. Clients subscribed to the leaderboard channel get instant updates without polling.

Watch out: Redis pub/sub has no persistence -- if a subscriber is disconnected when a message is published, that message is lost forever. Redis Streams fix this with consumer groups and acknowledgments, but many teams still use basic pub/sub without realizing the data loss risk. For any message that matters, use Streams, not pub/sub.

When to Use Kafka

Event Sourcing and Audit Logs

Kafka's append-only log is a natural fit for event sourcing. Every state change becomes an immutable event in a Kafka topic. You can rebuild any service's state by replaying its input topics from the beginning. This pattern powers systems at LinkedIn, Netflix, and Uber where audit trails are non-negotiable.

Stream Processing Pipelines

Kafka Connect ingests data from databases (CDC with Debezium), APIs, and files into Kafka topics. Kafka Streams or Flink processes events in real-time -- aggregating, filtering, joining, and enriching. The results land in downstream topics, databases, or search indexes. The entire pipeline is fault-tolerant, replayable, and horizontally scalable.

Cross-Service Data Integration

When you have 20+ microservices that all need to react to the same business events (order placed, user signed up, payment processed), Kafka's consumer group model lets each service consume at its own pace without affecting others. Adding a new consumer doesn't require changing the producer. This decoupling is Kafka's strongest selling point.

High-Volume Metrics and Logging

Kafka handles 10M+ messages per second on a modest cluster. This makes it ideal for aggregating metrics from thousands of servers, collecting application logs, and feeding monitoring systems. The retention period ensures you can reprocess historical data when you add new analytics.

When to Use RabbitMQ

Task Queues with Complex Routing

RabbitMQ's exchange-binding-queue model is unmatched for complex routing. A single message can be routed to different queues based on routing keys, headers, or topic patterns. Email notifications go to the email queue. SMS alerts go to the SMS queue. The producer doesn't know or care about the consumers -- the exchange handles routing.

Request-Reply Patterns

RabbitMQ natively supports the request-reply pattern with reply-to queues and correlation IDs. A service publishes a request, waits for a response on a temporary queue, and correlates it by ID. This is clunky to implement in Kafka (which doesn't have temporary queues) but built into RabbitMQ's AMQP model.

Priority Queues

RabbitMQ supports message priorities (0-255). Premium customers' requests can jump ahead of free-tier users'. Kafka doesn't have message-level priorities -- you'd need separate topics and consumer logic to achieve the same effect, which is messier.

Delayed and Scheduled Messages

With the delayed message exchange plugin, RabbitMQ delivers messages after a specified delay. Schedule a reminder email for 24 hours later, retry a failed webhook in 5 minutes, or implement exponential backoff -- all with native broker support. Kafka requires external schedulers or timestamp-based consumers for delayed delivery.

Cost Comparison: Running in Production

Factor	Redis	Kafka	RabbitMQ
Minimum viable cluster	3 nodes (Cluster mode)	3 brokers + 3 ZK (or KRaft)	3 nodes (quorum queues)
RAM requirements per node	8-64 GB (data lives in RAM)	4-8 GB (OS page cache helps)	4-16 GB
Disk requirements per node	Minimal (AOF backup only)	High (all data on disk)	Moderate (persistent queues)
AWS managed service	ElastiCache ($0.068/hr, r7g.large)	MSK ($0.21/hr, kafka.m5.large)	Amazon MQ ($0.13/hr, mq.m5.large)
Monthly cost (3-node managed)	~$150	~$460	~$285
Monthly cost (self-managed, 3x m6i.large)	~$210	~$210 + $100 storage	~$210
Operational complexity	Low	High (partitions, rebalancing, configs)	Medium (queue management, flow control)

Redis is cheapest to run but stores everything in RAM, so costs scale linearly with data volume. Kafka is the most expensive managed option but cheapest per message at high volume -- the cost-per-message at 1M msgs/sec is fractions of a cent. RabbitMQ sits in the middle and is the easiest to operate for teams without dedicated platform engineers.

Decision Framework: 5 Steps to Choose

1. Check if you already run one of these -- if Redis is already in your stack for caching, adding Streams for lightweight messaging avoids new infrastructure. Don't introduce Kafka for 10,000 messages/day.
2. Estimate message volume -- under 50,000 msgs/sec, all three work. Over 100,000 msgs/sec sustained, Kafka is the clear choice. Redis Streams can handle it but memory costs become prohibitive.
3. Determine if you need replay -- if consumers must reprocess historical messages (event sourcing, recomputing analytics, debugging), Kafka's log retention is essential. RabbitMQ deletes messages after consumption.
4. Evaluate routing complexity -- if messages need to reach different consumers based on content, headers, or priority, RabbitMQ's exchange model saves you from building routing logic in application code.
5. Assess operational capacity -- Kafka requires experienced operators (partition management, consumer group rebalancing, retention policies). RabbitMQ is simpler. Redis is simplest if your team already knows it.

Pro tip: Many production architectures use two of these three. A common pattern is Kafka for the event backbone (high-volume, cross-service events) and RabbitMQ for task queues (email sending, PDF generation, webhook delivery). Each tool handles what it's best at. Don't force one tool to do everything.

Frequently Asked Questions

Can Redis replace Kafka for event streaming?

For small-to-medium workloads (under 100,000 msgs/sec with limited retention), Redis Streams is a viable alternative. It supports consumer groups, message acknowledgment, and time-based retention. However, Redis stores everything in RAM, making long-term retention expensive. Kafka stores data on disk with configurable retention for days or weeks at a fraction of the cost.

Is RabbitMQ faster than Kafka?

RabbitMQ has lower latency for individual messages (sub-millisecond p50 vs Kafka's 2+ ms). But Kafka has 10-20x higher throughput because it batches writes and uses sequential disk I/O. If you need to process 1 million messages per second, Kafka wins. If you need each message delivered in under 1 ms, RabbitMQ wins.

What about Amazon SQS as an alternative?

SQS is a managed message queue that eliminates all operational overhead. It handles 3,000 msgs/sec per queue (higher with batching) at $0.40 per million messages. For teams that don't want to manage infrastructure, SQS is a solid choice for task queues. It lacks Kafka's event replay and RabbitMQ's routing sophistication, but the zero-ops model is hard to beat.

Does Kafka guarantee exactly-once delivery?

Kafka supports exactly-once semantics (EOS) within a Kafka-to-Kafka pipeline using idempotent producers and transactional consumers. End-to-end exactly-once (Kafka to an external system) requires the consumer to implement idempotent processing. In practice, most teams design for at-least-once delivery with idempotent consumers rather than relying on Kafka's EOS.

Should I use Redis pub/sub or Redis Streams?

Use Streams for almost everything. Pub/sub is fire-and-forget -- messages are lost if no subscriber is connected. Streams persist messages, support consumer groups for parallel processing, and allow message acknowledgment. The only use case for pub/sub is ephemeral real-time notifications where message loss is acceptable, like live dashboard updates.

How many Kafka partitions should I create?

Start with the number of consumers you expect in the largest consumer group. Kafka assigns one partition per consumer, so 12 partitions supports up to 12 parallel consumers. Over-partitioning (hundreds of partitions per topic) increases metadata overhead and rebalancing time. Under-partitioning limits parallelism. A good default is 6-12 partitions for most topics.

Making the Call

Use Redis when you need sub-millisecond messaging alongside caching, and data volume fits in memory. Use Kafka when you need high-throughput event streaming with replay capability and long-term retention. Use RabbitMQ when you need sophisticated routing, priority queues, or request-reply patterns with moderate volume.

The worst choice is picking a tool because it's trendy. Kafka for a queue processing 100 messages per minute is over-engineering. Redis pub/sub for critical financial events is under-engineering. Match the tool to the problem, and the system will be simpler, cheaper, and more reliable.