kafka-development

SKILL.md

Kafka Development

You are an expert in Apache Kafka event streaming and distributed messaging systems. Follow these best practices when building Kafka-based applications.

Core Principles

Kafka is a distributed event streaming platform for high-throughput, fault-tolerant messaging

Unlike traditional pub/sub, Kafka uses a pull model - consumers pull messages from partitions

Design for scalability, durability, and exactly-once semantics where needed

Leave NO todos, placeholders, or missing pieces in the implementation

Architecture Overview

Core Components

Topics: Categories/feeds for organizing messages

Partitions: Ordered, immutable sequences within topics enabling parallelism

Producers: Clients that publish messages to topics

Consumers: Clients that read messages from topics

Consumer Groups: Coordinate consumption across multiple consumers

Brokers: Kafka servers that store data and serve clients

Key Concepts

Messages are appended to partitions in order

Each message has an offset - a unique sequential ID within the partition

Consumers maintain their own cursor (offset) and can read streams repeatedly

Partitions are distributed across brokers for scalability

Topic Design

Partitioning Strategy

Use partition keys to place related events in the same partition

Messages with the same key always go to the same partition

This ensures ordering for related events

Choose keys carefully - uneven distribution causes hot partitions

Partition Count

More partitions = more parallelism but more overhead

Consider: expected throughput, consumer count, broker resources

Start with number of consumers you expect to run concurrently

Partitions can be increased but not decreased

Topic Configuration

retention.ms: How long to keep messages (default 7 days)

retention.bytes: Maximum size per partition

cleanup.policy: delete (remove old) or compact (keep latest per key)

min.insync.replicas: Minimum replicas that must acknowledge

Producer Best Practices

Reliability Settings

acks=all               # Wait for all replicas to acknowledge
retries=MAX_INT        # Retry on transient failures
enable.idempotence=true # Prevent duplicate messages on retry

Performance Tuning

batch.size: Accumulate messages before sending (default 16KB)

linger.ms: Wait time for batching (0 = send immediately)

buffer.memory: Total memory for buffering unsent messages

compression.type: gzip, snappy, lz4, or zstd for bandwidth savings

Error Handling

Implement retry logic with exponential backoff

Handle retriable vs non-retriable exceptions differently

Log and alert on send failures

Consider dead letter topics for messages that fail repeatedly

Partitioner

Default: hash of key determines partition (null key = round-robin)

Custom partitioners for specific routing needs

Ensure even distribution to avoid hot partitions

Consumer Best Practices

Offset Management

Consumers track which messages they've processed via offsets

auto.offset.reset: earliest (start from beginning) or latest (only new messages)

Commit offsets after successful processing, not before

Use enable.auto.commit=false for exactly-once semantics

Consumer Groups

Consumers in a group share partitions (each partition to one consumer)

More consumers than partitions = some consumers idle

Group rebalancing occurs when consumers join/leave

Use group.instance.id for static membership to reduce rebalances

Processing Patterns

Process messages in order within a partition

Handle out-of-order messages across partitions if needed

Implement idempotent processing for at-least-once delivery

Consider transactional processing for exactly-once

Timeouts and Failures

Implement processing timeout to isolate slow events

When timeout occurs, set event aside and continue to next message

Maintain overall system performance over processing every single event

Use dead letter queues for messages failing all retries

Error Handling and Retry

Retry Strategy

Allow multiple runtime retries per processing attempt

Example: 3 runtime retries per redrive, maximum 5 redrives = 15 total retries

Runtime retries typically cover 99% of failures

After exhausting retries, route to dead letter queue

Dead Letter Topics

Create dedicated DLT for messages that can't be processed

Include original topic, partition, offset, and error details

Monitor DLT for patterns indicating systemic issues

Implement manual or automated retry from DLT

Schema Management

Schema Registry

Use Confluent Schema Registry for schema management

Producers validate data against registered schemas during serialization

Schema mismatches throw exceptions, preventing malformed data

Provides common reference for producers and consumers

Schema Evolution

Design schemas for forward and backward compatibility

Add optional fields with defaults for backward compatibility

Avoid removing or renaming fields

Use schema versioning and migration strategies

Kafka Streams

State Management

Implement log compaction to maintain latest version of each key

Periodically purge old data from state stores

Monitor state store size and access patterns

Use appropriate storage backends for your scale

Windowing Operations

Handle out-of-order events and skewed timestamps

Use appropriate time extraction and watermarking techniques

Configure grace periods for late-arriving data

Choose window types based on use case (tumbling, hopping, sliding, session)

Security

Authentication

Use SASL/SSL for client authentication

Support SASL mechanisms: PLAIN, SCRAM, OAUTHBEARER, GSSAPI

Enable SSL for encryption in transit

Rotate credentials regularly

Authorization

Use Kafka ACLs for fine-grained access control

Grant minimum necessary permissions per principal

Separate read/write permissions by topic

Audit access patterns regularly

Monitoring and Observability

Key Metrics

Producer: record-send-rate, record-error-rate, batch-size-avg

Consumer: records-consumed-rate, records-lag, commit-latency

Broker: under-replicated-partitions, request-latency, disk-usage

Lag Monitoring

Consumer lag = last produced offset - last committed offset

High lag indicates consumers can't keep up

Alert on increasing lag trends

Scale consumers or optimize processing

Distributed Tracing

Propagate trace context in message headers

Use OpenTelemetry for end-to-end tracing

Correlate producer and consumer spans

Track message journey through the pipeline

Testing

Unit Testing

Mock Kafka clients for isolated testing

Test serialization/deserialization logic

Verify partitioning logic

Test error handling paths

Integration Testing

Use embedded Kafka or Testcontainers

Test full producer-consumer flows

Verify exactly-once semantics if used

Test rebalancing scenarios

Performance Testing

Load test with production-like message rates

Test consumer throughput and lag behavior

Verify broker resource usage under load

Test failure and recovery scenarios

Common Patterns

Event Sourcing

Store all state changes as immutable events

Rebuild state by replaying events

Use log compaction for snapshots

Enable time-travel debugging

CQRS (Command Query Responsibility Segregation)

Separate write (command) and read (query) models

Use Kafka as the event store

Build read-optimized projections from events

Handle eventual consistency appropriately

Saga Pattern

Coordinate distributed transactions across services

Each service publishes events for next step

Implement compensating transactions for rollback

Use correlation IDs to track saga instances

Change Data Capture (CDC)

Capture database changes as Kafka events

Use Debezium or similar CDC tools

Enable real-time data synchronization

Build event-driven integrations