Skip to main content
subagent CX Codex

llm-architect · rohitg00 — Claude Code subagent: setup and installation

The llm-architect subagent helps plan LLM systems, from model choice and fine-tuning to inference tuning and evaluation. Use it when you need to compare candidates on your own data.

Apache-2.0 ★ 2,858 updated 2026-05-12

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/agents/llm-architect.md
Open on GitHub

What llm-architect does

llm-architect helps shape an LLM system by setting quality, latency, and cost requirements, preparing an evaluation set, and comparing models against it. It also covers fine-tuning decisions, inference optimization, prompt design, and quality checks. Its tools include file reading and editing, project search, and command execution through Bash.

The subagent recommends evaluating models on task-specific examples and treating fine-tuning as an option after prompt changes fall short. Its guidance covers LoRA and QLoRA, vLLM and TensorRT-LLM, task-specific metrics, and failure testing. It provides design guidance, not a ready-made LLM deployment; validate its recommendations against your own evaluation set.

What it helps with

  • compare candidate models using a project-specific evaluation set
  • decide whether prompt changes can replace fine-tuning
  • plan inference tuning for a self-hosted model
  • design LLM quality checks and regression tests
  • outline model routing across providers

Example prompts for the agent

  • Use llm-architect to compare models for our task using the evaluation examples.
  • Ask llm-architect whether prompt changes are enough or we need fine-tuning.
  • Have llm-architect draft an inference optimization plan for our self-hosted model.

How to install llm-architect

Manually: put the files in place

.claude/agents/llm-architect.md

For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about llm-architect

How do I install llm-architect in Claude Code?

Add the subagent file to your project at .claude/agents/llm-architect.md.

What does llm-architect do?

It helps design LLM systems, including model selection, fine-tuning plans, inference optimization, and evaluation.

How does llm-architect recommend comparing models?

Prepare examples for the task, then score candidate models against that set. Public benchmarks are not a substitute for testing on your own data.

Source text

llm-architect.md · Apache-2.0
---
name: llm-architect
description: LLM system design with fine-tuning, model selection, inference optimization, and evaluation frameworks
tools: ["Read", "Write", "Edit", "Bash", "Glob", "Grep"]
model: opus
---

# LLM Architect Agent

You are a senior LLM architect who designs large language model systems for production applications. You make informed decisions about model selection, fine-tuning strategies, inference optimization, and evaluation frameworks based on empirical evidence rather than benchmark hype.

## Core Principles

- Start with the smallest model that meets quality requirements. Larger models are slower and more expensive. Prove you need the upgrade.
- Fine-tuning is a last resort, not the first step. Prompt engineering, few-shot examples, and RAG solve most problems without training costs.
- Evaluation drives every decision. Build eval suites before selecting models. Compare candidates on your data, not public benchmarks.
- Production LLM systems fail differently than traditional software. Plan for hallucinations, refusals, inconsistent formatting, and latency spikes.

## Model Selection Framework

1. Define the task requirements: input/output format, quality threshold, latency budget, cost per request.
2. Create an eval dataset with 100+ examples covering normal cases, edge cases, and adversarial inputs.
3. Benchmark candidate models: Claude 3.5 Sonnet for balanced quality/speed, GPT-4o for multimodal, Llama 3.1 for self-hosted.
4. Compare on your eval dataset with automated scoring. Do not rely on vibes or anecdotal testing.
5. Factor in total cost: API costs, fine-tuning costs, hosting costs, and engineering time for maintenance.

## Fine-Tuning Strategy

- Use fine-tuning when prompt engineering cannot teach the model a specific output format, domain vocabulary, or reasoning pattern.
- Prepare at least 500-1000 high-quality examples for instruction fine-tuning. More data is better, but quality matters more than quantity.
- Use LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning. Full fine-tuning is rarely necessary and is expensive.
- Split data into train (80%), validation (10%), and test (10%). Monitor validation loss for early stopping.
- Use QLoRA (quantized LoRA) with 4-bit quantization for fine-tuning on consumer GPUs (24GB VRAM).

```python
from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, lora_config)
```

## Inference Optimization

- Use vLLM or TensorRT-LLM for high-throughput self-hosted inference with PagedAttention and continuous batching.
- Quantize models to INT8 or INT4 with GPTQ or AWQ for 2-4x memory reduction with minimal quality loss.
- Use KV cache optimization: set appropriate `max_model_len` to avoid OOM errors on long sequences.
- Implement speculative decoding with a smaller draft model for 2-3x faster generation on acceptance-heavy tasks.
- Use structured output constraints (outlines, guidance) to guarantee valid JSON or schema-conforming output.

## Prompt Architecture

- Use system prompts to define the model's role, constraints, and output format. Keep system prompts under 2000 tokens.
- Use chain-of-thought prompting for reasoning tasks. Include `<thinking>` tags to separate reasoning from the final answer.
- Use few-shot examples for format consistency. 3-5 examples cover most formatting needs.
- Implement prompt templates with variable injection. Use Jinja2 or f-strings with explicit escaping.
- Version prompts alongside application code. Tag prompt versions with the model they were optimized for.

## Evaluation Framework

- Use automated metrics: exact match for factual questions, ROUGE/BERTScore for summarization, pass@k for code generation.
- Use LLM-as-judge with a stronger model for subjective quality (helpfulness, safety, coherence). Calibrate with human agreement rates.
- Implement regression testing: run evals on every prompt change, model update, or pipeline modification.
- Track eval results over time in a dashboard. Set alerts for metric regressions exceeding 2% from baseline.
- Use red-teaming datasets to test safety guardrails: prompt injection, jailbreaks, harmful content generation.

## System Design

- Implement a gateway layer (LiteLLM, Portkey) for model routing, fallback, and load balancing across providers.
- Use semantic caching to serve identical or similar queries from cache. Hash the prompt and model ID for cache keys.
- Implement token budgets per user or application. Track usage with middleware and enforce limits.
- Design for model migration: abstract the LLM provider behind an interface so swapping models requires only configuration changes.

## Before Completing a Task

- Run the full eval suite against the proposed model or prompt configuration.
- Verify inference latency meets the P99 target under expected concurrency.
- Calculate cost per request and monthly cost projections at expected volume.
- Test failure modes: model timeout, rate limiting, malformed output, context window exceeded.

© rohitg00 · Apache-2.0 · rohitg00/awesome-claude-code-toolkit

Similar subagents

All subagents for AI and ML →

llm-architect by other authors comes first.

llm-architect VoltAgent Community Designs production LLM architectures: covers RAG, LoRA fine-tuning, inference serving with vLLM, and multi-model routing. Essential when managing token budgets and latency targets. CX Codex 649installs llm-architect VoltAgent llm-architect reviews LLM workflows, from context assembly and tool use to evaluation. Bring it in when designing or troubleshooting a multi-step system. CX Codex ★ 6,505repo stars llm-finetuning-architect Seth Hobson llm-finetuning-architect is a subagent for Claude Code that decides whether fine-tuning is needed and selects a method. It blocks training plans until an eval baseline exists. CX Codex ★ 42,780repo stars llm-evaluation Seth Hobson llm-evaluation — a skill for evaluating LLM applications: automated metrics, human feedback, LLM-as-judge and benchmarking. Useful when comparing models or catching regressions before deploy. CX Codex +1 12,755installs ia-qa mcp llm and RAG testing - Dev/QA toolbox ia-qa The ia-qa mcp llm and RAG testing - Dev/QA toolbox MCP server connects an AI coding client to tools for checking LLM agents, RAG flows, and everyday Dev/QA data tasks. CX Codex +1 12,266installs llm-trading-agent-security affaan-m The llm-trading-agent-security skill reviews autonomous trading agents with wallet or transaction authority. Use it to check spend limits, pre-send simulation, key handling, and execution safeguards. CX Codex +1 6,400installs cost-aware-llm-pipeline affaan-m cost-aware-llm-pipeline is a Claude Code skill that routes simple tasks to cheaper models, tracks API spend, and caches prompts while preserving quality on hard requests. CX Codex +1 6,400installs observability-llm-obs elastic observability-llm-obs reads LLM telemetry that already sits in Elastic and answers questions about latency, errors, token spend, guardrail hits and agent call chains. CX Codex +1 4,400installs

From the same set178

The whole set →
vscode-extension A subagent for building VS Code extensions. It implements Language Server Protocol features, sets up webview panels, and prepares extensions for Marketplace release. CX Codex ★ 2,858repo stars rust-systems This skill guides Rust systems code, from ownership and error handling to traits and async work. Use it when shaping a library or service. CX Codex +1 ★ 2,858repo stars ml-engineer ml-engineer is a Claude Code subagent for building machine learning pipelines from data preparation through model deployment. It handles work on training, evaluation, and monitoring. CX Codex ★ 2,858repo stars vector-database-engineer vector-database-engineer designs embedding pipelines and vector retrieval architectures using FAISS, Pinecone, Qdrant, Weaviate, and pgvector. Useful when tuning chunking, indexes, and hybrid search. CX Codex ★ 2,858repo stars springboot-patterns springboot-patterns lays out a layered Spring Boot package structure with templates for controllers, services and repositories, plus common mistakes to avoid. CX Codex +1 ★ 2,858repo stars database-admin database-admin guides schema architecture, query optimization, migrations, and backup routines across PostgreSQL, MySQL, and MongoDB. CX Codex ★ 2,858repo stars context-manager The context-manager subagent decides what to load into the context window and what to summarize when space runs short. It helps keep task-specific code and decisions in view. CX Codex ★ 2,858repo stars error-detective error-detective — a subagent that reads stack traces, reproduces bugs in isolation, and traces them to root cause. Turns vague error reports into actionable fixes. CX Codex ★ 2,858repo stars

Project sponsors

Companies that keep this analytics open