kafka-expert — Claude Code skill: what it does, how to install
kafka-expert — a skill for Apache Kafka work: producer and consumer patterns, Kafka Streams topologies, partitioning and consumer groups, ready whenever a project touches event streaming.
Apache-2.0 220 installs ★ 45 updated 2026-10-12
npx skills add personamanagmentlayer/pcl --skill kafka-expert -a claude-codeWhat kafka-expert does
It activates when the conversation turns to Kafka, Kafka Streams or an event-driven design. The vocabulary comes first: topics, partitions, offsets, consumer groups, Kafka Connect, exactly-once processing.
Working samples follow. The Python producer waits for every replica to acknowledge and retries on failure; the consumer turns auto-commit off and commits offsets by hand. The Streams sample is Java — filter a stream, map the values, write to an output topic. Both point at localhost:9092, so plan on replacing that with your own broker addresses.
Then a practices list covering partition keys, consumer lag, idempotent producers and rebalance handling, with a set of anti-patterns beside it: one-partition topics, synchronous processing, no monitoring. Permission is narrow. Only Bash(kafka:*) is granted, so shell calls outside the kafka command won't run.
What it helps with
- Write a producer that waits for acknowledgement from every replica
- Switch a consumer to manual offset commits
- Build a Kafka Streams topology that filters and maps values
- Choose partition keys that preserve ordering per entity
- Work out why consumer lag keeps climbing
Example prompts for the agent
- Write a Python producer that publishes to
user-eventswithacks='all'and retries - Why is consumer lag climbing, and how do I move to manual commits?
- Build a Kafka Streams topology that drops short records and uppercases the rest
How to install kafka-expert
1Skills CLI (Vercel)
npx skills add personamanagmentlayer/pcl --skill kafka-expert -a claude-code2Manually: put the files in place
~/.claude/skills/kafka-expert/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/kafka-expert/
1Skills CLI (Vercel)
npx skills add personamanagmentlayer/pcl --skill kafka-expert -a codex2Manually: put the files in place
.agents/skills/kafka-expert/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add personamanagmentlayer/pcl --skill kafka-expert -a cursor2Manually: put the files in place
.cursor/skills/kafka-expert/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install personamanagmentlayer/pcl stdlib/data/kafka-expert2Manually: put the files in place
.github/skills/kafka-expert/SKILL.md1Skills CLI (Vercel)
npx skills add personamanagmentlayer/pcl --skill kafka-expert -a gemini-cli2Platform CLI
gemini skills install https://github.com/personamanagmentlayer/pcl --path stdlib/data/kafka-expertManually: put the files in place
.devin/skills/kafka-expert/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add personamanagmentlayer/pcl --skill kafka-expert -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about kafka-expert
How do I install kafka-expert in Claude Code?
Run npx skills add personamanagmentlayer/pcl --skill kafka-expert -a claude-code; the skill then sits at ~/.claude/skills/kafka-expert/SKILL.md.
What does kafka-expert do?
It gives the agent Kafka know-how: core concepts, runnable producer and consumer code, a Kafka Streams topology, plus practices and anti-patterns to check against.
Which tools is the skill allowed to use?
Read, Write, Edit and Bash(kafka:*). From bash only kafka commands go through; anything else is blocked.
Source text
SKILL.md · Apache-2.0
---
name: kafka-expert
version: 1.1.0
description: >-
Expert-level Apache Kafka, event streaming, Kafka Streams, and distributed messaging. Use
when the user mentions streaming, messaging, event driven, or Kafka Streams.
category: data
tags: [kafka, streaming, messaging, event-driven, kafka-streams]
allowed-tools:
- Read
- Write
- Edit
- Bash(kafka:*)
---
# Apache Kafka Expert
Expert guidance for Apache Kafka, event streaming, Kafka Streams, and building event-driven architectures.
## Core Concepts
- Topics, partitions, and offsets
- Producers and consumers
- Consumer groups
- Kafka Streams
- Kafka Connect
- Exactly-once semantics
## Producer
```python
from kafka import KafkaProducer
import json
producer = KafkaProducer(
bootstrap_servers=['localhost:9092'],
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
acks='all', # Wait for all replicas
retries=3
)
# Send message
future = producer.send('user-events', {
'user_id': '123',
'event': 'login',
'timestamp': '2024-01-01T00:00:00Z'
})
# Wait for acknowledgment
record_metadata = future.get(timeout=10)
print(f"Topic: {record_metadata.topic}, Partition: {record_metadata.partition}")
producer.flush()
producer.close()
```
## Consumer
```python
from kafka import KafkaConsumer
consumer = KafkaConsumer(
'user-events',
bootstrap_servers=['localhost:9092'],
group_id='my-group',
auto_offset_reset='earliest',
enable_auto_commit=False,
value_deserializer=lambda m: json.loads(m.decode('utf-8'))
)
for message in consumer:
print(f"Received: {message.value}")
# Process message
process_event(message.value)
# Manual commit
consumer.commit()
```
## Kafka Streams
```java
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> source = builder.stream("input-topic");
// Transform and filter
KStream<String, String> transformed = source
.filter((key, value) -> value.length() > 10)
.mapValues(value -> value.toUpperCase());
transformed.to("output-topic");
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();
```
## Best Practices
- Use appropriate partition keys
- Monitor consumer lag
- Implement idempotent producers
- Use consumer groups for scaling
- Set proper retention policies
- Handle rebalancing gracefully
- Monitor cluster metrics
## Anti-Patterns
❌ Single partition topics
❌ No error handling
❌ Ignoring consumer lag
❌ Producing to wrong partitions
❌ Not using consumer groups
❌ Synchronous processing
❌ No monitoring
## Resources
- Apache Kafka: https://kafka.apache.org/
- Confluent Platform: https://www.confluent.io/
© personamanagmentlayer · Apache-2.0 · personamanagmentlayer/pcl