Skip to main content
plugin

llm-application-dev · ormus-solutions — Claude Code plugin: features and install

llm-application-dev adds an LLM architect agent, RAG scaffolding, and reference patterns for prompt engineering and LangChain orchestration to Claude Code.

MIT ★ 126 updated 2026-10-04

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add Ormus-Solutions/LibreUIUX-Claude-Code
/plugin install llm-application-dev@libreuiux
Open on GitHub

What llm-application-dev does

The plugin introduces the @llm-architect agent alongside a /rag-scaffold slash command and four structured skills. These skills guide developers through RAG architectures, prompt strategies like few-shot examples or chain-of-thought, LangChain workflows, and LLM output evaluation.

For quality testing, it outlines evaluation approaches across metrics such as faithfulness, relevance, latency, and cost, mentioning frameworks like RAGAS, DeepEval, and Promptfoo. You can quickly generate project boilerplate by pairing /rag-scaffold with vector databases like Pinecone or Chroma.

Because this tooling focuses strictly on text generation and vector retrieval, it is explicitly not designed for traditional ML, computer vision, or audio processing tasks.

What it helps with

  • design a hybrid-search RAG pipeline with source citations
  • structure prompt templates for predictable JSON outputs
  • scaffold document Q&A projects via /rag-scaffold
  • set up evaluation metrics for response faithfulness and hallucination rates

Example prompts for the agent

  • @llm-architect design a RAG system for internal documentation
  • /rag-scaffold document-qa --vector-db pinecone
  • @llm-architect suggest evaluation metrics for this chatbot

How to install llm-application-dev

Plugin marketplace

/plugin marketplace add Ormus-Solutions/LibreUIUX-Claude-Code
/plugin install llm-application-dev@libreuiux

Step by step, with commands for each platform: Claude Code plugins: marketplaces, installing, and building your own

Questions about llm-application-dev

How to install llm-application-dev in Claude Code?

Run both marketplace commands in sequence: first /plugin marketplace add Ormus-Solutions/LibreUIUX-Claude-Code, then /plugin install llm-application-dev@libreuiux.

What is included with llm-application-dev?

It bundles the @llm-architect specialist agent, the /rag-scaffold command, and four skill guides covering RAG design, prompt patterns, LangChain, and LLM evaluation.

When should you avoid using this plugin?

It does not cover classical machine learning tasks like tabular regression, nor does it support computer vision or audio and speech processing.

Source text

llm-application-dev · MIT
# LLM Application Development Plugin

> Build production-grade AI applications with RAG, prompt engineering, and LLM orchestration.

**LLM Application Dev** provides:
- **4 Skills** - RAG, prompt engineering, LangChain, LLM evaluation
- **1 Agent** - LLM application architect
- **1 Command** - RAG project scaffolding

**Result**: AI-powered applications that are accurate, reliable, and maintainable.

---

## Quick Start

### Invoke the Agent
```
@llm-architect design a RAG system for internal documentation
@llm-architect review my prompt for edge cases
@llm-architect suggest evaluation metrics for this chatbot
```

### Use Commands
```
/rag-scaffold document-qa --vector-db pinecone
```

---

## The 4 Skills

### 1. RAG Implementation
> Build knowledge-grounded AI that cites sources

Retrieval-Augmented Generation connects LLMs to external knowledge bases, reducing hallucinations and enabling domain-specific answers.

**Key Components:**
| Component | Purpose | Options |
|-----------|---------|---------|
| Vector DB | Store embeddings | Pinecone, Weaviate, Chroma, Qdrant |
| Embeddings | Text to vectors | OpenAI, Sentence Transformers, Cohere |
| Retriever | Find relevant docs | Dense, sparse, hybrid search |
| Generator | Produce answers | GPT-4, Claude, local LLMs |

**Architecture:**
```
Query -> Embedding -> Vector Search -> Relevant Chunks -> LLM with Context -> Answer
```

**Best Practices:**
- Chunk documents at semantic boundaries (not arbitrary splits)
- Overlap chunks to preserve context (10-20%)
- Use hybrid search (dense + sparse) for robustness
- Include source citations in responses
- Implement answer verification

---

### 2. Prompt Engineering Patterns
> Craft prompts that get consistent, high-quality results

**Core Patterns:**

| Pattern | When to Use |
|---------|-------------|
| **Few-shot** | Show examples of desired output |
| **Chain-of-thought** | Complex reasoning tasks |
| **System prompts** | Set consistent persona/rules |
| **Output formatting** | Structured responses (JSON, etc.) |
| **Self-consistency** | Sample multiple, take majority |

**Example - Structured Output:**
```
You are a product classifier. Given a product description, output JSON:
{
  "category": "string",
  "subcategory": "string",
  "confidence": 0-100
}

Product: "Wireless Bluetooth earbuds with noise cancellation"
```

**Anti-patterns:**
- Vague instructions ("do a good job")
- No examples for complex tasks
- Ignoring edge cases
- No output format specification

---

### 3. LangChain Architecture
> Orchestrate LLM chains, agents, and tools

**Key Concepts:**

| Concept | Purpose |
|---------|---------|
| **Chains** | Sequence of operations |
| **Agents** | LLM decides which tools to use |
| **Tools** | Functions the agent can call |
| **Memory** | Conversation history management |
| **Retrievers** | Document retrieval for RAG |

**Common Patterns:**
```python
# Simple chain
chain = prompt | llm | output_parser

# RAG chain
chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Agent with tools
agent = create_react_agent(llm, tools, prompt)
```

**When to Use What:**
- **Chains**: Predictable, fixed workflows
- **Agents**: Dynamic, decision-based workflows
- **RAG**: Knowledge-intensive applications

---

### 4. LLM Evaluation
> Measure and improve LLM application quality

**Evaluation Dimensions:**

| Metric | What It Measures |
|--------|------------------|
| **Relevance** | Does the answer address the query? |
| **Faithfulness** | Is the answer grounded in retrieved context? |
| **Harmlessness** | Are responses safe and appropriate? |
| **Helpfulness** | Does it actually help the user? |
| **Latency** | Response time |
| **Cost** | Token usage and API costs |

**Evaluation Approaches:**
1. **LLM-as-Judge**: Use GPT-4 to score outputs
2. **Human Evaluation**: Gold standard but slow
3. **Reference-based**: Compare to ground truth (BLEU, ROUGE)
4. **Retrieval Metrics**: Precision@k, Recall@k, MRR

**Tools:**
- RAGAS - RAG evaluation framework
- DeepEval - Unit tests for LLMs
- Promptfoo - Prompt testing
- LangSmith - Tracing and evaluation

---

## Plugin Structure

```
llm-application-dev/
├── README.md                              # This file
│
├── skills/                                # Knowledge base
│   ├── rag-implementation/
│   │   └── SKILL.md                       # RAG patterns
│   ├── prompt-engineering-patterns/
│   │   └── SKILL.md                       # Prompt patterns
│   ├── langchain-architecture/
│   │   └── SKILL.md                       # LangChain patterns
│   └── llm-evaluation/
│       └── SKILL.md                       # Evaluation patterns
│
├── commands/                              # Slash commands
│   └── rag-scaffold.md                    # RAG project setup
│
└── agents/                                # Specialist agents
    └── llm-architect.md                   # LLM application expert
```

---

## Usage Examples

### Example 1: Document Q&A System
```
User: Design a RAG system for our internal wiki

@llm-architect:
1. Ingestion Pipeline
   - Crawl wiki pages
   - Chunk by headings/sections
   - Embed with text-embedding-3-large
   - Store in Pinecone with metadata

2. Query Pipeline
   - Embed query
   - Hybrid search (dense + BM25)
   - Rerank top 10 to top 3
   - Generate answer with sources

3. Evaluation
   - Track faithfulness (hallucination rate)
   - Measure relevance scores
   - Monitor latency p50/p95
```

### Example 2: Prompt Optimization
```
User: My classification prompt is inconsistent

@llm-architect:
1. Add few-shot examples (3-5)
2. Specify output format explicitly
3. Add edge case handling
4. Use temperature=0 for consistency
5. Implement self-consistency (3 samples)

Before: 60% accuracy
After: 92% accuracy
```

### Example 3: Evaluation Setup
```
User: How do I evaluate my chatbot quality?

@llm-architect:
1. Define metrics
   - Relevance: LLM-as-judge scoring
   - Faithfulness: Check against sources
   - Helpfulness: User satisfaction ratings

2. Create test dataset
   - 100+ representative queries
   - Include edge cases
   - Cover all categories

3. Implement evaluation pipeline
   - Automated nightly runs
   - Dashboard with trends
   - Alerts on degradation
```

---

## When to Use This Plugin

### Perfect For:
- Building chatbots and Q&A systems
- Creating document search/analysis tools
- Developing AI-powered assistants
- Implementing content generation
- Building agent systems with tools

### Not Ideal For:
- Traditional ML (classification, regression)
- Computer vision applications
- Audio/speech processing

### Combines Well With:
- **python-development** - Python patterns for LLM apps
- **backend-development** - API design for AI services
- **machine-learning-ops** - MLOps for LLM deployment
- **observability-monitoring** - LLM observability

---

## Technology Stack

| Category | Technologies |
|----------|--------------|
| Orchestration | LangChain, LlamaIndex, Haystack |
| Vector DBs | Pinecone, Weaviate, Chroma, Qdrant |
| LLM Providers | OpenAI, Anthropic, Cohere, local (Ollama) |
| Evaluation | RAGAS, DeepEval, Promptfoo |
| Observability | LangSmith, Langfuse, Phoenix |

---

## Key Concepts Quick Reference

### RAG Pipeline
```
1. Ingest: Documents -> Chunks -> Embeddings -> Vector DB
2. Query: Question -> Embedding -> Search -> Context -> LLM -> Answer
```

### Prompt Structure
```
System: [Persona and rules]
Context: [Retrieved documents]
Examples: [Few-shot demonstrations]
Query: [User question]
Format: [Output specification]
```

### Evaluation Loop
```
1. Define metrics
2. Create test dataset
3. Run evaluations
4. Analyze results
5. Improve prompts/retrieval
6. Repeat
```

---

## License

Part of the LibreUIUX project - MIT License

---

*"The goal isn't to replace human intelligence, but to augment it with knowledge retrieval and reasoning at scale."*

© Seth Hobson · MIT · Ormus-Solutions/LibreUIUX-Claude-Code

Similar plugins

All plugins for AI and ML →

llm-application-dev by other authors comes first.

llm-application-dev Seth Hobson llm-application-dev — a plugin for building LLM apps, RAG systems, and LangGraph agents. Use it for vector search, embeddings, prompt work, and model evaluation. CX Codex ★ 42,780repo stars llm-application-dev-prompt-optimize sickn33 llm-application-dev-prompt-optimize refines raw prompts into production instructions using regression checks and structured evaluation. Helps balance latency, cost, and answer quality. CX Codex +1 1installs developer-kit-ai Giuseppe Trisciuoglio The developer-kit-ai plugin adds skills for prompt design, RAG pipelines, and document chunking. Use it when building applications that rely on LLMs. ★ 372repo stars llm-app-development travisjneuman llm-app-development — a skill for Claude Code guides the design of RAG pipelines, autonomous agents, and structured schemas. Triggers when building chatbots, retrieval systems, or LLM-driven tools. CX Codex +1 139installs langchain-orchestration manutej langchain-orchestration — a LangChain reference skill for your coding agent: chains, agents, memory, RAG and orchestration. Useful when the agent writes LangChain code and should stick to working patterns. CX Codex +1 453installs ai-engineer qdhenry ai-engineer designs RAG pipelines, configures vector databases, and writes agent workflows for LLM applications. Built for developers adding AI features. CX Codex ★ 1,357repo stars agent alirezarezvani agent — a prompt for LLM engineers. Builds and refines prompts across Claude, GPT and open models — from system instructions to reasoning chains, with model-specific adaptation. GP ChatGPT ★ 788repo stars

From the same set42

The whole set →
vibe-coding The vibe-coding skill helps create throwaway UI prototypes, compare design directions, and decide when a concept is ready for production work. Use it while validating interface ideas or user flows. ★ 126repo stars mcp-integrations mcp-integrations is a Claude Code plugin that uses MCP browser tools to inspect live interfaces. It can turn screenshots into UI code and extract design tokens. ★ 126repo stars archetypal-alchemy archetypal-alchemy transforms Jungian archetypes and Major Arcana symbolism into UI layouts, design tokens, and Tailwind color palettes inside Claude Code. ★ 126repo stars design-mastery design-mastery is a Claude Code plugin that applies established design principles to UI work. It supports layout analysis, design audits, style guides, and brand identity tasks. ★ 126repo stars julia-development julia-development adds julia-pro sub-agent that writes Julia 1.10+ code, optimizes numerical algorithms, designs multiple dispatch type hierarchies, and builds packages. ★ 126repo stars python-development Plugin for Python development: skills for asyncio, pytest, packaging and uv, an expert sub-agent, and a command to scaffold projects. ★ 126repo stars cloud-infrastructure cloud-infrastructure designs AWS, GCP, and Azure setups, prepares reusable Terraform modules, and optimizes cloud spending using six specialized subagents. ★ 126repo stars code-review-ai code-review-ai ships the /ai-review command for layered pull-request checks and an architect agent that joins design discussions on its own. ★ 126repo stars

Project sponsors

Companies that keep this analytics open