# kafka-stream-processing

Guides your coding assistant through Apache Kafka stream pipelines, event streaming patterns, and Kafka Streams task topologies. Fits data engineering workflows.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 480 installs · ★ 62
- Author: manutej
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/manutej/luxor-claude-marketplace/tree/HEAD/plugins/luxor-data-engineering/skills/kafka-stream-processing/SKILL.md
- Updated: 2026-06-13
- HTML: https://affpapa.org/en/nevibecode/skills/kafka-stream-processing

## What it does

This skill teaches coding agents how to build event-driven backends on Apache Kafka. It outlines partition architecture, producer behaviors, and consumer coordination. The material also details delivery guarantees, contrasting standard at-least-once retries with transactional exactly-once processing.

Execution limits mirror Kafka's internal structure. In Kafka Streams, the number of input partitions dictates your maximum task parallelism. If a topic has 5 partitions, launching 6 application instances leaves one node completely idle. That idle instance serves only as a backup.

Topic maintenance requires caution. While `kafka-topics.sh` lets you raise partition counts via `--alter`, Kafka cannot lower them later. Adding partitions also leaves historical records right where they were, which can disrupt custom partitioning logic.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill kafka-stream-processing -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/kafka-stream-processing/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/kafka-stream-processing/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill kafka-stream-processing -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/kafka-stream-processing/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill kafka-stream-processing -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/kafka-stream-processing/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install manutej/luxor-claude-marketplace kafka-stream-processing
```

2. **Manually: put the files in place**
- File: `.github/skills/kafka-stream-processing/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill kafka-stream-processing -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/manutej/luxor-claude-marketplace --path plugins/luxor-data-engineering/skills/kafka-stream-processing
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/kafka-stream-processing/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill kafka-stream-processing -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/manutej/luxor-claude-marketplace/tree/HEAD/plugins/luxor-data-engineering/skills/kafka-stream-processing/SKILL.md

## When to use it
- configure partitions and replication settings using `kafka-topics.sh`
- manage parallel stream processing across Kafka Streams tasks
- apply transactional exactly-once semantics to data pipelines
- structure producer and consumer groups for microservice events

## Example prompts
- Create a new Kafka topic named orders with 20 partitions and replication factor 3 using `kafka-topics.sh`.
- Explain how Kafka Streams distributes 5 topic partitions among worker application instances.
- Configure producer retries and consumer offsets for an at-least-once messaging pipeline.

## Similar
- [mongodb-atlas-stream-processing](https://affpapa.org/en/nevibecode/skills/mongodb-atlas-stream-processing.md) — skill · 3600 installs
- [kafka-development](https://affpapa.org/en/nevibecode/skills/kafka-development.md) — skill · 1200 installs
- [neo4j-kafka-skill](https://affpapa.org/en/nevibecode/skills/neo4j-kafka-skill.md) — skill · 772 installs
- [strimzi-kafka-cli](https://affpapa.org/en/nevibecode/mcp/strimzi-kafka-cli.md) — MCP server · ★ 91
- [spring-boot-event-driven-patterns](https://affpapa.org/en/nevibecode/skills/spring-boot-event-driven-patterns.md) — skill · 3800 installs
- [managing-amazon-msk](https://affpapa.org/en/nevibecode/skills/managing-amazon-msk.md) — skill · 5100 installs

From the same set: [manutej/luxor-claude-marketplace](https://affpapa.org/en/nevibecode/by/manutej/luxor-claude-marketplace.md) — 65 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md