Skip to main content
skill CX Codex

kafka-expert — Claude Code skill: what it does, how to install

kafka-expert — a skill for Apache Kafka work: producer and consumer patterns, Kafka Streams topologies, partitioning and consumer groups, ready whenever a project touches event streaming.

Apache-2.0 220 installs ★ 45 updated 2026-10-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add personamanagmentlayer/pcl --skill kafka-expert -a claude-code
Open on GitHub

What kafka-expert does

It activates when the conversation turns to Kafka, Kafka Streams or an event-driven design. The vocabulary comes first: topics, partitions, offsets, consumer groups, Kafka Connect, exactly-once processing.

Working samples follow. The Python producer waits for every replica to acknowledge and retries on failure; the consumer turns auto-commit off and commits offsets by hand. The Streams sample is Java — filter a stream, map the values, write to an output topic. Both point at localhost:9092, so plan on replacing that with your own broker addresses.

Then a practices list covering partition keys, consumer lag, idempotent producers and rebalance handling, with a set of anti-patterns beside it: one-partition topics, synchronous processing, no monitoring. Permission is narrow. Only Bash(kafka:*) is granted, so shell calls outside the kafka command won't run.

What it helps with

  • Write a producer that waits for acknowledgement from every replica
  • Switch a consumer to manual offset commits
  • Build a Kafka Streams topology that filters and maps values
  • Choose partition keys that preserve ordering per entity
  • Work out why consumer lag keeps climbing

Example prompts for the agent

  • Write a Python producer that publishes to user-events with acks='all' and retries
  • Why is consumer lag climbing, and how do I move to manual commits?
  • Build a Kafka Streams topology that drops short records and uppercases the rest

How to install kafka-expert

1Skills CLI (Vercel)

npx skills add personamanagmentlayer/pcl --skill kafka-expert -a claude-code

2Manually: put the files in place

~/.claude/skills/kafka-expert/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/kafka-expert/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about kafka-expert

How do I install kafka-expert in Claude Code?

Run npx skills add personamanagmentlayer/pcl --skill kafka-expert -a claude-code; the skill then sits at ~/.claude/skills/kafka-expert/SKILL.md.

What does kafka-expert do?

It gives the agent Kafka know-how: core concepts, runnable producer and consumer code, a Kafka Streams topology, plus practices and anti-patterns to check against.

Which tools is the skill allowed to use?

Read, Write, Edit and Bash(kafka:*). From bash only kafka commands go through; anything else is blocked.

Source text

SKILL.md · Apache-2.0
---
name: kafka-expert
version: 1.1.0
description: >-
  Expert-level Apache Kafka, event streaming, Kafka Streams, and distributed messaging. Use
  when the user mentions streaming, messaging, event driven, or Kafka Streams.
category: data
tags: [kafka, streaming, messaging, event-driven, kafka-streams]
allowed-tools:
  - Read
  - Write
  - Edit
  - Bash(kafka:*)
---

# Apache Kafka Expert

Expert guidance for Apache Kafka, event streaming, Kafka Streams, and building event-driven architectures.

## Core Concepts

- Topics, partitions, and offsets
- Producers and consumers
- Consumer groups
- Kafka Streams
- Kafka Connect
- Exactly-once semantics

## Producer

```python
from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    acks='all',  # Wait for all replicas
    retries=3
)

# Send message
future = producer.send('user-events', {
    'user_id': '123',
    'event': 'login',
    'timestamp': '2024-01-01T00:00:00Z'
})

# Wait for acknowledgment
record_metadata = future.get(timeout=10)
print(f"Topic: {record_metadata.topic}, Partition: {record_metadata.partition}")

producer.flush()
producer.close()
```

## Consumer

```python
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'user-events',
    bootstrap_servers=['localhost:9092'],
    group_id='my-group',
    auto_offset_reset='earliest',
    enable_auto_commit=False,
    value_deserializer=lambda m: json.loads(m.decode('utf-8'))
)

for message in consumer:
    print(f"Received: {message.value}")

    # Process message
    process_event(message.value)

    # Manual commit
    consumer.commit()
```

## Kafka Streams

```java
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

StreamsBuilder builder = new StreamsBuilder();

KStream<String, String> source = builder.stream("input-topic");

// Transform and filter
KStream<String, String> transformed = source
    .filter((key, value) -> value.length() > 10)
    .mapValues(value -> value.toUpperCase());

transformed.to("output-topic");

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();
```

## Best Practices

- Use appropriate partition keys
- Monitor consumer lag
- Implement idempotent producers
- Use consumer groups for scaling
- Set proper retention policies
- Handle rebalancing gracefully
- Monitor cluster metrics

## Anti-Patterns

❌ Single partition topics
❌ No error handling
❌ Ignoring consumer lag
❌ Producing to wrong partitions
❌ Not using consumer groups
❌ Synchronous processing
❌ No monitoring

## Resources

- Apache Kafka: https://kafka.apache.org/
- Confluent Platform: https://www.confluent.io/

© personamanagmentlayer · Apache-2.0 · personamanagmentlayer/pcl

Similar skills

All skills for Data and databases →

From the same set104

The whole set →
standards-expert standards-expert gives the agent a working reference on ISO management systems — processes, risk scoring, audits — for teams preparing a certification. CX Codex +1 226installs flask-expert flask-expert guides Python work with Flask apps, REST APIs, extensions, and deployment. Use it when a task involves Jinja2, Flask structure, authentication, or API design. CX Codex +1 225installs csharp-expert The csharp-expert skill guides agents through modern C# work with .NET, ASP.NET Core, LINQ, and async code. Use it for backend services and enterprise application patterns. CX Codex +1 221installs maritime-expert The maritime-expert skill gives an agent maritime domain knowledge for vessel tracking, port operations, cargo systems, and shipping software. CX Codex +1 221installs fastapi-expert fastapi-expert — a skill for building FastAPI APIs: Pydantic models, JWT auth, SQLAlchemy integration and background tasks, all with working code patterns. CX Codex +1 218installs nodejs-expert nodejs-expert guides the AI assistant in backend development with Node.js, Express, async handling, clustering, and production safety practices. CX Codex +1 216installs databricks-expert databricks-expert hands the agent Databricks practice for clusters, Delta Lake, PySpark and MLflow. Useful when a pipeline crawls or a join falls apart. CX Codex +1 215installs kubernetes-expert kubernetes-expert guides cluster operations, deployment manifests, network debugging, and Helm chart setups. Useful for DevOps engineers and container deployments. CX Codex +1 215installs

Project sponsors

Companies that keep this analytics open