Skip to main content
subagent CX Codex

kafka-expert · 0xfurai — Claude Code subagent: what it does, how to install

kafka-expert — a subagent for Kafka work: partition planning, broker tuning, Kafka Streams pipelines, security and monitoring. Useful when a topic layout or a lagging consumer group needs a decision.

MIT ★ 1,029 updated 2025-10-15

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/agents/kafka-expert.md
Open on GitHub

What kafka-expert does

One subject only: Apache Kafka. The agent carries sections on cluster and broker configuration, partition layout, producer and consumer tuning, Streams processing, offsets and group coordination, retention with log compaction, SASL/Kerberos and TLS security, JMX monitoring.

Expect deliverables, not conversation. It writes broker config files, a partition and replication plan, producer and consumer code with retries and idempotence, alert rules, an upgrade procedure, stress-test output with bottleneck notes.

Stream processing here is at-least-once. For exactly-once delivery, state transactions as a separate requirement — the agent's checklist does not build them in by itself.

What it helps with

  • split a topic into partitions that match real traffic
  • tune brokers for throughput instead of leaving defaults
  • make a producer idempotent with retries that work
  • find out why a consumer group keeps lagging on offsets
  • set up JMX dashboards and alert rules

Example prompts for the agent

  • Design the partition layout for the orders topic against our current event volume.
  • Consumer group billing is falling behind on offsets — dig in and propose settings.
  • Use the kafka-expert subagent to review our broker config before we scale up.

How to install kafka-expert

Manually: put the files in place

.claude/agents/kafka-expert.md

For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about kafka-expert

How do I install kafka-expert in Claude Code?

Create the file .claude/agents/kafka-expert.md in your project and paste the agent contents into it.

What does kafka-expert do?

It handles Kafka architecture: partitions, brokers, producers and consumers, Streams, security, monitoring. You get configs, plans and code rather than general advice.

Which delivery semantics does it assume?

At-least-once. If you need exactly-once, describe transactions and their verification as a separate requirement.

Source text

kafka-expert.md · MIT
---
name: kafka-expert
description: Write highly efficient, scalable, and fault-tolerant Kafka architectures. Handles Kafka stream processing, cluster setup, and performance optimization. Use PROACTIVELY for Kafka architecture design, troubleshooting, or improving Kafka performance.
model: claude-sonnet-4-20250514
---

## Focus Areas

- Kafka cluster setup and configuration
- Partitioning strategy for scalability
- Producer and consumer optimization
- Kafka Streams and real-time processing
- Handling offsets and consumer group coordination
- Fault-tolerance and high availability
- Data retention and compaction strategies
- Security (encryption, authentication, authorization)
- Monitoring and alerting Kafka clusters
- Upgrading and maintaining Kafka clusters

## Approach

- Configure brokers with optimal settings for throughput
- Design topic partitioning based on load and access patterns
- Implement idempotent and transactional producers
- Use consumer poll loop and backpressure handling
- Use Kafka Streams DSL for processing pipelines
- Implement replication and failover for data resilience
- Optimize message sizes and batch configuration
- Use SASL/Kerberos and TLS for secure communication
- Monitor using JMX and Kafka-specific metrics
- Plan cluster resources for future growth and scaling

## Quality Checklist

- Brokers configured with sufficient heap memory
- Topics have adequate partitions and replication factor
- Producers handle retries and idempotence properly
- Consumers balance load across partitions
- Stream processing follows at-least-once semantics
- Secure connections and policies are enforced
- Retention and log compaction are configured per requirements
- Regular auditing of ACLs and access patterns
- Effective handling and alerting of cluster anomalies
- Perform routine maintenance with minimal downtime

## Output

- Optimized Kafka cluster configuration files
- Partition and replication plans for scalability
- Producer and consumer code with best practices
- Stream processing code with error handling
- Security configurations and policy documents
- Monitoring dashboard setups and alert rules
- Documentation of upgrade and scaling procedures
- Stress test results with bottleneck analysis
- Incident response and troubleshooting playbooks
- Capacity planning and resource allocation reports

© 0xfurai · MIT · 0xfurai/claude-code-subagents

Similar subagents

All subagents for DevOps and infrastructure →

kafka-expert by other authors comes first.

kafka-expert carlrannaberg kafka-expert diagnoses Apache Kafka streaming issues in projects, resolves consumer rebalancing errors, tunes producer reliability, and checks under-replicated partitions. CX Codex ★ 763repo stars kafka-development Mindrally The kafka-development skill for Claude Code brings Apache Kafka best practices to producer/consumer pipelines, topic partitioning, Kafka Streams, and Schema Registry. CX Codex +1 1,200installs Kafka Streaming taste f Kafka Streaming taste is a prompt for ChatGPT, Claude.ai, and Claude Code that defines Kafka streaming architecture and design rules. GP ChatGPT ★ 178,201repo stars kafka-stream-processing manutej kafka-stream-processing guides your coding assistant through Apache Kafka stream pipelines, event streaming patterns, and Kafka Streams task topologies. Fits data engineering workflows. CX Codex +1 482installs strimzi-kafka-cli SystemCraftsman strimzi-kafka-cli gives your AI agent 42 tools to manage Kafka clusters, topics, users and connectors through the Strimzi operator. CX Codex +1 ★ 91repo stars mirrord-kafka MetalBear mirrord-kafka is a Claude Code skill that prepares Kubernetes resources and message filters for Kafka queue splitting in mirrord. It supports setup and session troubleshooting. CX Codex +1 ★ 31repo stars neo4j-kafka-skill neo4j-contrib neo4j-kafka-skill — a Claude Code skill for configuring the Neo4j Connector for Kafka and native CDC streams. It connects Kafka event topics with Neo4j graph nodes and relationships. CX Codex +1 814installs

From the same set137

The whole set →
css-expert css-expert — a CSS-focused subagent for responsive layouts, animation, accessibility, and style cleanup. Use it when a stylesheet needs review or refactoring. CX Codex ★ 1,029repo stars sequelize-expert sequelize-expert — a Claude subagent for Sequelize work: model design, associations, migrations, seeders and transactions. CX Codex ★ 1,029repo stars angularjs-expert The angularjs-expert subagent refactors legacy AngularJS codebases, tunes digest cycles, and structures MVC components. It helps maintain older frontends without degrading runtime speed. CX Codex ★ 1,029repo stars pulumi-expert The pulumi-expert subagent helps build and maintain Pulumi infrastructure, from cloud resources and state to secrets, policies, and CI/CD workflows. Use it when working on infrastructure code or deployments. CX Codex ★ 1,029repo stars javascript-expert javascript-expert refactors modern JavaScript using ES6+ standards, async patterns, and strict equality. Helps optimize DOM queries and find race conditions. CX Codex ★ 1,029repo stars keycloak-expert keycloak-expert — a Keycloak subagent for realms, clients, roles, user federation, and OIDC or SAML security work. Use it when an IAM task needs a structured setup. CX Codex ★ 1,029repo stars deno-expert deno-expert — a subagent for Claude Code that specializes in Deno—a modern runtime for JavaScript and TypeScript with built-in security and tooling. Covers permissions model, ES modules, type safety, and performance optimization. CX Codex ★ 1,029repo stars expo-expert expo-expert guides development and optimization of React Native mobile apps built with the Expo framework. CX Codex ★ 1,029repo stars

Project sponsors

Companies that keep this analytics open