Introduction

Kafka Producer is a key component for sending messages to a Kafka cluster. Proper producer configuration is critical for achieving high performance and system reliability.

Key tuning parameters

1. Batching and Compression

# Increase batch size for better throughput
batch.size=32768
linger.ms=5

# Enable compression to save bandwidth
compression.type=lz4

2. Memory and Buffer

# Buffer configuration
buffer.memory=67108864
max.block.ms=60000

3. Acknowledgments and Durability

# For high reliability
acks=all
retries=2147483647
enable.idempotence=true

Real-world examples

High Throughput Scenario

For high volume data scenarios:

  • Increase batch.size to 64KB
  • Set linger.ms=10-20
  • Use compression

Low Latency Scenario

For low latency scenarios:

  • Decrease batch.size to 1KB
  • Set linger.ms=0
  • Disable compression

Monitoring

Key metrics to track:

  • record-send-rate
  • batch-size-avg
  • compression-rate-avg

Performance Optimization Tips

Producer Configuration Best Practices

  1. Batch Size Optimization

    # For throughput
batch.size=65536
linger.ms=20

# For latency
batch.size=1024
linger.ms=0

  2. Memory Management

    # Prevent OutOfMemory
buffer.memory=134217728
max.block.ms=10000

  3. Error Handling

    # Robust error handling
retries=Integer.MAX_VALUE
retry.backoff.ms=1000
delivery.timeout.ms=300000

Common Issues and Solutions

Issue: High Latency

Symptoms: Messages take long to send Solution: Reduce batch.size and linger.ms

Issue: Low Throughput

Symptoms: Poor message/sec rate Solution: Increase batch.size and enable compression

Issue: Memory Issues

Symptoms: OutOfMemoryError Solution: Tune buffer.memory and max.block.ms