Skip to main content
skill CX Codex

multi-agent-performance-profiling — Claude Code skill

multi-agent-performance-profiling spawns five parallel profiling agents to investigate system layers: instrumentation, database config, client library, batch sizing, and synthesis.

MIT 126 installs ★ 3 updated 2026-10-25

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add terrylica/cc-skills --skill multi-agent-performance-profiling -a claude-code
Open on GitHub

What multi-agent-performance-profiling does

When production underperforms SLO and the bottleneck is unclear, this skill structures the search across multiple layers. Five agents run in parallel: one instruments every pipeline stage using time.perf_counter() and tracemalloc, another examines QuestDB config and database settings, a third inspects client API patterns (batch sends, buffer sizes), the fourth models batch size tradeoffs between memory and throughput. The fifth agent reads all reports, identifies consensus, and ranks fixes by priority level.

Inside is a framework for prioritization: P0 (>5x gains), P1 (2–5x), P2 (1.2–2x quick wins). Each agent leaves a profiling script, Markdown report, and evidence. In practice, this skill traced a CloudFront → ZIP → CSV → QuestDB pipeline and found the true bottleneck was CloudFront download (90% of time), not the database ingesting at 1.1M rows/sec.

Requires Python for profiling scripts and access to pipeline code and system configs.

What it helps with

  • pinpoint the actual bottleneck instead of optimizing by assumption
  • measure each stage's contribution to total time (download, extract, parse, ingest)
  • prioritize optimizations by measured impact and implementation effort
  • profile both memory and time across system layers simultaneously
  • avoid wasting effort on non-bottleneck optimizations

Example prompts for the agent

  • Profile the CloudFront → unzip → CSV → QuestDB pipeline hitting 47K/sec instead of 100K target
  • Run parallel agents to analyze database config, batch sizing, and client throughput
  • Get a synthesis report with priorities: which stage yields 5x improvement, which gives 2x, which are quick wins

How to install multi-agent-performance-profiling

1Skills CLI (Vercel)

npx skills add terrylica/cc-skills --skill multi-agent-performance-profiling -a claude-code

2Manually: put the files in place

~/.claude/skills/multi-agent-performance-profiling/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/multi-agent-performance-profiling/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about multi-agent-performance-profiling

How do I install multi-agent-performance-profiling in Claude Code?

Run npx skills add terrylica/cc-skills --skill multi-agent-performance-profiling -a claude-code, then open ~/.claude/skills/multi-agent-performance-profiling/SKILL.md.

What does multi-agent-performance-profiling do?

Deploys five agents in parallel: one profiles each pipeline stage with timing, one audits database config, one analyzes client library usage, one models batch-size tradeoffs, and one synthesizes findings with consensus priorities.

What do I need to use this skill?

Python for profiling scripts, access to pipeline code and system configs, and the tools Bash, Read, Write, Glob, Grep for file inspection.

Source text

SKILL.md · MIT
---
name: multi-agent-performance-profiling
description: Multi-agent performance profiling for pipeline bottlenecks. TRIGGERS - performance profiling, bottleneck analysis, pipeline optimization.
allowed-tools: Bash, Read, Write, Glob, Grep
---

# Multi-Agent Performance Profiling

> **Self-Evolving Skill**: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

## Overview

Prescriptive workflow for spawning parallel profiling agents to comprehensively identify performance bottlenecks across multiple system layers. Successfully discovered that QuestDB ingests at 1.1M rows/sec (11x faster than target), proving database was NOT the bottleneck - CloudFront download was 90% of pipeline time.

## When to Use This Skill

Use this skill when:

- Performance below SLO (e.g., 47K vs 100K rows/sec target)
- Multi-stage pipeline optimization (download → extract → parse → ingest)
- Database performance investigation
- Bottleneck identification in complex workflows
- Pre-optimization analysis (before making changes)

**Key outcomes:**

- Identify true bottleneck (vs assumed bottleneck)
- Quantify each stage's contribution to total time
- Prioritize optimizations by impact (P0/P1/P2)
- Avoid premature optimization of non-bottlenecks

## Core Methodology

### 1. Multi-Layer Profiling Model (5-Agent Pattern)

**Agent 1: Profiling (Instrumentation)**

- Empirical timing of each pipeline stage
- Phase-boundary instrumentation with time.perf_counter()
- Memory profiling (peak usage, allocations)
- Bottleneck identification (% of total time)

**Agent 2: Database Configuration Analysis**

- Server settings review (WAL, heap, commit intervals)
- Production vs development config comparison
- Expected impact quantification (<5%, 10%, 50%)

**Agent 3: Client Library Analysis**

- API usage patterns (dataframe vs row-by-row)
- Buffer size tuning opportunities
- Auto-flush behavior analysis

**Agent 4: Batch Size Analysis**

- Current batch size validation
- Optimal batch range determination
- Memory overhead vs throughput tradeoff

**Agent 5: Integration & Synthesis**

- Consensus-building across agents
- Prioritization (P0/P1/P2) with impact quantification
- Implementation roadmap creation

### 2. Agent Orchestration Pattern

**Parallel Execution** (all 5 agents run simultaneously):

```
Agent 1 (Profiling)          → [PARALLEL]
Agent 2 (DB Config)          → [PARALLEL]
Agent 3 (Client Library)     → [PARALLEL]
Agent 4 (Batch Size)         → [PARALLEL]
Agent 5 (Integration)        → [PARALLEL - reads tmp/ outputs from others]
```

**Key Principle**: No dependencies between investigation agents (1-4). Integration agent synthesizes findings.

**Dynamic Todo Management:**

- Start with investigation plan (5 agents)
- Spawn agents in parallel using single message with multiple Task tool calls
- Update todos as each agent completes
- Integration agent waits for all findings before synthesizing

### 3. Profiling Script Structure

Each agent produces:

1. **Investigation Script** (e.g., `profile_pipeline.py`)
   - time.perf_counter() instrumentation at phase boundaries
   - Memory profiling with tracemalloc
   - Structured output (phase, duration, % of total)
2. **Report** (markdown with findings, recommendations, impact quantification)
3. **Evidence** (benchmark results, config dumps, API traces)

**Example Profiling Code:**

```python
import time

# Profile multi-stage pipeline
def profile_pipeline():
    results = {}

    # Phase 1: Download
    start = time.perf_counter()
    data = download_from_cdn(url)
    results["download"] = time.perf_counter() - start

    # Phase 2: Extract
    start = time.perf_counter()
    csv_data = extract_zip(data)
    results["extract"] = time.perf_counter() - start

    # Phase 3: Parse
    start = time.perf_counter()
    df = parse_csv(csv_data)
    results["parse"] = time.perf_counter() - start

    # Phase 4: Ingest
    start = time.perf_counter()
    ingest_to_db(df)
    results["ingest"] = time.perf_counter() - start

    # Analysis
    total = sum(results.values())
    for phase, duration in results.items():
        pct = (duration / total) * 100
        print(f"{phase}: {duration:.3f}s ({pct:.1f}%)")

    return results
```

### 4. Impact Quantification Framework

**Priority Levels:**

- **P0 (Critical)**: >5x improvement, addresses primary bottleneck
- **P1 (High)**: 2-5x improvement, secondary optimizations
- **P2 (Medium)**: 1.2-2x improvement, quick wins
- **P3 (Low)**: <1.2x improvement, minor tuning

**Impact Reporting Format:**

```markdown
### Recommendation: [Optimization Name] (P0/P1/P2) - [IMPACT LEVEL]

**Impact**: 🔴/🟠/🟡 **Nx improvement**
**Effort**: High/Medium/Low (N days)
**Expected Improvement**: CurrentK → TargetK rows/sec

**Rationale**:

- [Why this matters]
- [Supporting evidence from profiling]
- [Comparison to alternatives]

**Implementation**:
[Code snippet or architecture description]
```

### 5. Consensus-Building Pattern

**Integration Agent Responsibilities:**

1. Read all investigation reports (Agents 1-4)
2. Identify consensus recommendations (all agents agree)
3. Flag contradictions (agents disagree)
4. Synthesize master integration report
5. Create implementation roadmap (P0 → P1 → P2)

**Consensus Criteria:**

- ≥3/4 agents recommend same optimization → Consensus
- 2/4 agents recommend, 2/4 neutral → Investigate further
- Agents contradict (one says "optimize X", another says "X is not bottleneck") → Run tie-breaker experiment

## Workflow: Step-by-Step

### Step 1: Define Performance Problem

**Input**: Performance metric below SLO
**Output**: Problem statement with baseline metrics

**Example Problem Statement:**

```
Performance Issue: BTCUSDT 1m ingestion at 47K rows/sec
Target SLO: >100K rows/sec
Gap: 53% below target
Pipeline: CloudFront download → ZIP extract → CSV parse → QuestDB ILP ingest
```

### Step 2: Create Investigation Plan

**Directory Structure:**

```
tmp/perf-optimization/
  profiling/              # Agent 1
    profile_pipeline.py
    PROFILING_REPORT.md
  questdb-config/         # Agent 2
    CONFIG_ANALYSIS.md
  python-client/          # Agent 3
    CLIENT_ANALYSIS.md
  batch-size/             # Agent 4
    BATCH_ANALYSIS.md
  MASTER_INTEGRATION_REPORT.md  # Agent 5
```

**Agent Assignment:**

- Agent 1: Empirical profiling (instrumentation)
- Agent 2: Database configuration analysis
- Agent 3: Client library usage analysis
- Agent 4: Batch size optimization analysis
- Agent 5: Synthesis and integration

### Step 3: Spawn Agents in Parallel

**IMPORTANT**: Use single message with multiple Task tool calls for true parallelism

**Example:**

```
I'm going to spawn 5 parallel investigation agents:

[Uses Task tool 5 times in a single message]
- Agent 1: Profiling
- Agent 2: QuestDB Config
- Agent 3: Python Client
- Agent 4: Batch Size
- Agent 5: Integration (depends on others completing)
```

**Execution:**

```bash
# All agents run simultaneously (user observes 5 parallel tool calls)
# Each agent writes to its own tmp/ subdirectory
# Integration agent polls for completed reports
```

### Step 4: Wait for All Agents to Complete

**Progress Tracking:**

- Update todo list as each agent completes
- Integration agent polls tmp/ directory for report files
- Once 4/4 investigation reports exist → Integration agent synthesizes

**Completion Criteria:**

- All 4 investigation reports written
- Integration report synthesizes findings
- Master recommendations list created

### Step 5: Review Master Integration Report

**Report Structure:**

```markdown
# Master Performance Optimization Integration Report

## Executive Summary

- Critical discovery (what is/isn't the bottleneck)
- Key findings from each agent (1-sentence summary)

## Top 3 Recommendations (Consensus)

1. [P0 Optimization] - HIGHEST IMPACT
2. [P1 Optimization] - HIGH IMPACT
3. [P2 Optimization] - QUICK WIN

## Agent Investigation Summary

### Agent 1: Profiling

### Agent 2: Database Config

### Agent 3: Client Library

### Agent 4: Batch Size

## Implementation Roadmap

### Phase 1: P0 Optimizations (Week 1)

### Phase 2: P1 Optimizations (Week 2)

### Phase 3: P2 Quick Wins (As time permits)
```

### Step 6: Implement Optimizations (P0 First)

**For each recommendation:**

1. Implement highest-priority optimization (P0)
2. Re-run profiling script
3. Verify expected improvement achieved
4. Update report with actual results
5. Move to next priority (P1, P2, P3)

**Example Implementation:**

```bash
# Before optimization
uv run python tmp/perf-optimization/profiling/profile_pipeline.py
# Output: 47K rows/sec, download=857ms (90%)

# Implement P0 recommendation (concurrent downloads)
# [Make code changes]

# After optimization
uv run python tmp/perf-optimization/profiling/profile_pipeline.py
# Output: 450K rows/sec, download=90ms per symbol * 10 concurrent (90%)
```

## Real-World Example: QuestDB Refactor Performance Investigation

**Context**: Pipeline achieving 47K rows/sec, target 100K rows/sec (53% below SLO)

**Assumptions Before Investigation:**

- QuestDB ILP ingestion is the bottleneck (4% of time)
- Need to tune database configuration
- Need to optimize Sender API usage

**Findings After 5-Agent Investigation:**

1. **Profiling Agent**: CloudFront download is 90% of time (857ms), ILP ingest only 4% (40ms)
2. **QuestDB Config Agent**: Database already optimal, tuning provides <5% improvement
3. **Python Client Agent**: Sender API already optimal (using dataframe() bulk ingestion)
4. **Batch Size Agent**: 44K batch size is within optimal range
5. **Integration Agent**: Consensus recommendation - optimize download, NOT database

**Top 3 Recommendations:**

1. 🔴 **P0**: Concurrent multi-symbol downloads (10-20x improvement)
2. 🟠 **P1**: Multi-month pipeline parallelism (2x improvement)
3. 🟡 **P2**: Streaming ZIP extraction (1.3x improvement)

**Impact**: Discovered database ingests at 1.1M rows/sec (11x faster than target) - proving database was never the bottleneck

**Outcome**: Avoided wasting 2-3 weeks optimizing database when download was the real bottleneck

## Common Pitfalls

### 1. Profiling Only One Layer

❌ **Bad**: Profile database only, assume it's the bottleneck
✅ **Good**: Profile entire pipeline (download → extract → parse → ingest)

### 2. Serial Agent Execution

❌ **Bad**: Run Agent 1, wait, then run Agent 2, wait, etc.
✅ **Good**: Spawn all 5 agents in parallel using single message with multiple Task calls

### 3. Optimizing Without Profiling

❌ **Bad**: "Let's optimize the database config first" (assumption-driven)
✅ **Good**: Profile first, discover database is only 4% of time, optimize download instead

### 4. Ignoring Low-Hanging Fruit

❌ **Bad**: Only implement P0 (highest impact, highest effort)
✅ **Good**: Implement P2 quick wins (1.3x for 4-8 hours effort) while planning P0

### 5. Not Re-Profiling After Changes

❌ **Bad**: Implement optimization, assume it worked
✅ **Good**: Re-run profiling script, verify expected improvement achieved

## Resources

### scripts/

Not applicable - profiling scripts are project-specific (stored in `tmp/perf-optimization/`)

### references/

- `profiling_template.py` - Template for phase-boundary instrumentation
- `integration_report_template.md` - Template for master integration report
- `impact_quantification_guide.md` - How to assess P0/P1/P2 priorities

### assets/

Not applicable - profiling artifacts are project-specific

---

## Troubleshooting

| Issue                       | Cause                     | Solution                                            |
| --------------------------- | ------------------------- | --------------------------------------------------- |
| Agents running sequentially | Using separate messages   | Spawn all agents in single message with multi-Task  |
| Integration report empty    | Agent reports not written | Wait for all 4 investigation agents to complete     |
| Wrong bottleneck identified | Single-layer profiling    | Profile entire pipeline, not just assumed layer     |
| Profiling results vary      | No warmup runs            | Run 3-5 warmup iterations before measuring          |
| Memory not profiled         | Missing tracemalloc       | Add tracemalloc instrumentation to profiling script |
| P0/P1 priority unclear      | No impact quantification  | Include expected Nx improvement for each finding    |
| Consensus missing           | Agents not compared       | Integration agent must synthesize all 4 reports     |
| Re-profile shows no change  | Caching effects           | Clear caches, restart services before re-profiling  |


## Post-Execution Reflection

After this skill completes, reflect before closing the task:

0. **Locate yourself.** — Find this SKILL.md's canonical path before editing.
1. **What failed?** — Fix the instruction that caused it.
2. **What worked better than expected?** — Promote to recommended practice.
3. **What drifted?** — Fix any script, reference, or dependency that no longer matches reality.
4. **Log it.** — Evolution-log entry with trigger, fix, and evidence.

Do NOT defer. The next invocation inherits whatever you leave behind.

© terrylica · MIT · terrylica/cc-skills

Similar skills

All skills for Agent setup: memory, context, orchestration →

From the same set79

The whole set →
asciinema-player asciinema-player — a skill for Claude Code and five other platforms that plays .cast terminal recordings inside a dedicated iTerm2 window. Files over 100 MB crash browser players; this skill streams them from disk. CX Codex +1 126installs impl-standards impl-standards — a set of coding rules: error propagation, constant extraction, progress logging for long operations, and dependency injection patterns. CX Codex +1 126installs bootstrap-monorepo bootstrap-monorepo scaffolds a polyglot monorepo using moon, proto, and Bun with a TypeScript control plane and strict boundary rules. CX Codex +1 126installs asciinema-converter asciinema-converter turns asciinema .cast recordings into clean .txt files for reading and search. It handles one file or a directory batch. CX Codex +1 126installs clickhouse-cloud-management clickhouse-cloud-management manages ClickHouse Cloud database users and access permissions using SQL statements over HTTPS. CX Codex +1 126installs booking-notify booking-notify sends notifications for new bookings, cancellations and reschedules via Telegram and Pushover. 6-hour polling sync, real-time webhook for urgent events. CX Codex +1 125installs implement-plan-preflight implement-plan-preflight runs the ITP workflow preflight phase to generate MADR 4.0 architecture records, design specs, and linked diagrams. CX Codex +1 125installs issues-workflow issues-workflow puts plan, hierarchy and status into GitHub Issues: sub-issues under a parent, an issue linked to its branch and pull request, and a close on merge. CX Codex +1 124installs

Project sponsors

Companies that keep this analytics open