🔒 Paid

Kafka Partition Skew

A single large tenant's traffic has spiked, and its messages are concentrated on one Kafka partition. Aggregate consumer-group utilization still looks moderate — most consumers are nearly idle — but the one partition that tenant's messages land on is falling critically behind, because only one consumer can ever read from a given partition at a time. Diagnose why the backlog keeps growing despite healthy-looking aggregate metrics, and restore throughput before downstream processing falls critically behind.

Difficulty
Staff
Expected Time
~18 min
Category
Messaging & Streaming

Skills

  • Diagnosing per-partition hot-key bottlenecks
  • Partition key design and salting strategies
  • Recognizing when scaling consumers does not help
  • Reading aggregate vs. per-partition metrics correctly

System Architecture

  • Producer Service
  • Kafka Topic
  • Hot Partition
  • Consumer Group
  • Downstream Processor

This is a Pro scenario. Upgrade to Pro to unlock it.

Upgrade to ProBack to Incidents