llm-integration · yonatangross — Claude Code skill: what it does, how to install
llm-integration — a collection of 20 production rules for LLM workflows: strict function calling, SSE streaming endpoints, local Ollama deployments, and LoRA tuning.
MIT 218 installs ★ 305 updated 2026-11-21
npx skills add yonatangross/orchestkit --skill llm-integration -a claude-codeWhat llm-integration does
This skill bundles engineering blueprints for embedding LLMs into backend services. It organizes 20 rules across seven domains: function calling with strict schemas, Server-Sent Events streaming in FastAPI, local model hosting via Ollama, and QLoRA fine-tuning with Unsloth. It also includes rules for context window management, prompt caching, and evaluation using RAGAS and LLM-as-judge setups.
Individual rules live in the rules/ directory and load on demand. For tool execution, the patterns enforce strict input validation using Pydantic or Zod and require execution errors to return as tool output rather than unhandled server exceptions.
The skill explicitly restricts fine-tuning: it should be used only after exhausting prompt engineering and RAG. Training requires a curated dataset of at least 1000 quality records. Integration requires Claude Code version 2.1.277 or newer.
What it helps with
- define strict schemas for parallel and single function calling
- build SSE streaming endpoints with backpressure handling
- connect local models in Ollama with LangChain bindings
- set up QLoRA fine-tuning pipelines using Unsloth
- manage token budgets and context caching across conversation turns
Example prompts for the agent
- Configure a FastAPI endpoint with SSE streaming for model responses
- Draft a strict mode tool schema for function calling with input validation
- Set up a QLoRA fine-tuning script for Llama 3 using Unsloth
How to install llm-integration
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill llm-integration -a claude-code2Manually: put the files in place
~/.claude/skills/llm-integration/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/llm-integration/
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill llm-integration -a codex2Manually: put the files in place
.agents/skills/llm-integration/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill llm-integration -a cursor2Manually: put the files in place
.cursor/skills/llm-integration/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install yonatangross/orchestkit src/skills/llm-integration2Manually: put the files in place
.github/skills/llm-integration/SKILL.md1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill llm-integration -a gemini-cli2Platform CLI
gemini skills install https://github.com/yonatangross/orchestkit --path src/skills/llm-integrationManually: put the files in place
.devin/skills/llm-integration/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill llm-integration -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about llm-integration
How do I install llm-integration in Claude Code?
Run npx skills add yonatangross/orchestkit --skill llm-integration -a claude-code.
What does the llm-integration skill do?
It provides 20 modular rules covering tool definitions, streaming responses, local inference, context optimization, prompt patterns, and fine-tuning.
What prerequisite does the skill enforce for fine-tuning?
It requires exhausting prompt design and RAG first, along with a prepared dataset of at least 1000 high-quality training examples.
Source text
SKILL.md · MIT
---
name: llm-integration
license: MIT
compatibility: "Claude Code 2.1.277+."
description: LLM integration patterns for function calling, streaming responses, local inference with Ollama, and fine-tuning customization. Use when implementing tool use, SSE streaming, local model deployment, LoRA/QLoRA fine-tuning, or multi-provider LLM APIs.
user-invocable: false
metadata:
archive-candidate: "pending an outside usage signal"
owner-agent: "llm-integrator"
category: mcp-enhancement
version: "2.0.0"
author: "OrchestKit"
complexity: "medium"
tags: "llm, function-calling, streaming, ollama, fine-tuning, lora, tool-use, local-inference"
allowed-tools: "Read Glob Grep WebFetch WebSearch"
---
# LLM Integration
Patterns for integrating LLMs into production applications: tool use, streaming, local inference, and fine-tuning. Each category has individual rule files in `rules/` loaded on-demand.
## Quick Reference
| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [Function Calling](#function-calling) | 3 | CRITICAL | Tool definitions, parallel execution, input validation |
| [Streaming](#streaming) | 3 | HIGH | SSE endpoints, structured streaming, backpressure handling |
| [Local Inference](#local-inference) | 3 | HIGH | Ollama setup, model selection, GPU optimization |
| [Fine-Tuning](#fine-tuning) | 3 | HIGH | LoRA/QLoRA training, dataset preparation, evaluation |
| [Context Optimization](#context-optimization) | 2 | HIGH | Window management, compression, caching, budget scaling |
| [Evaluation](#evaluation) | 2 | HIGH | LLM-as-judge, RAGAS metrics, quality gates, benchmarks |
| [Prompt Engineering](#prompt-engineering) | 4 | HIGH | CoT, few-shot, versioning, DSPy optimization, ReAct, cost optimization |
**Total: 20 rules across 7 categories**
## Quick Start
```python
# Function calling: strict mode tool definition
tools = [{
"type": "function",
"function": {
"name": "search_documents",
"description": "Search knowledge base",
"strict": True,
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"limit": {"type": "integer", "description": "Max results"}
},
"required": ["query", "limit"],
"additionalProperties": False
}
}
}]
```
```python
# Streaming: SSE endpoint with FastAPI
@app.get("/chat/stream")
async def stream_chat(prompt: str):
async def generate():
async for token in async_stream(prompt):
yield {"event": "token", "data": token}
yield {"event": "done", "data": ""}
return EventSourceResponse(generate())
```
```python
# Local inference: Ollama with LangChain
llm = ChatOllama(
model="deepseek-r1:70b",
base_url="http://localhost:11434",
temperature=0.0,
num_ctx=32768,
)
```
```python
# Fine-tuning: QLoRA with Unsloth
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Meta-Llama-3.1-8B",
max_seq_length=2048, load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=32)
```
## Function Calling
Enable LLMs to use external tools and return structured data. Use strict mode schemas (2026 best practice) for reliability. On Claude keep the `tools` array stable across turns so the prompt cache holds; add tools mid-session with an inline `tool_addition` (beta `inline-tools-2026-09-15`, Claude API only, available on Sonnet 5.5 but not on Sonnet 5) instead of swapping subsets (https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages#mid-conversation-tool-changes). For large tool sets, use a non-deferred tool search tool with `defer_loading: true` on the rest, never `defer_loading` alone (a request cannot have every tool deferred). Forced `tool_choice` (`any` or a named tool) returns a 400 on Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1; use `auto` with `strict: true` tools. Validate all inputs with Pydantic/Zod, and return errors as tool results. <!-- model-recency-ok: per-model contrast, Sonnet 5 lacks what Sonnet 5.5 has (platform docs 2026-09-28) -->
- `calling-tool-definition.md` -- Strict mode schemas, OpenAI/Anthropic formats, LangChain binding
- `calling-parallel.md` -- Parallel tool execution, asyncio.gather, strict mode constraints
- `calling-validation.md` -- Input validation, error handling, tool execution loops
## Streaming
Deliver LLM responses in real-time for better UX. Use SSE for web, WebSocket for bidirectional. Handle backpressure with bounded queues.
- `streaming-sse.md` -- FastAPI SSE endpoints, frontend consumers, async iterators
- `streaming-structured.md` -- Streaming with tool calls, partial JSON parsing, chunk accumulation
- `streaming-backpressure.md` -- Backpressure handling, bounded buffers, cancellation
## Local Inference
Run LLMs locally with Ollama for cost savings (93% vs cloud), privacy, and offline development. Pre-warm models, use provider factory for cloud/local switching.
- `local-ollama-setup.md` -- Installation, model pulling, environment configuration
- `local-model-selection.md` -- Model comparison by task, hardware profiles, quantization
- `local-gpu-optimization.md` -- Apple Silicon tuning, keep-alive, CI integration
## Fine-Tuning
Customize LLMs with parameter-efficient techniques. Fine-tune ONLY after exhausting prompt engineering and RAG. Requires 1000+ quality examples.
- `tuning-lora.md` -- LoRA/QLoRA configuration, Unsloth training, adapter merging
- `tuning-dataset-prep.md` -- Synthetic data generation, quality validation, deduplication
- `tuning-evaluation.md` -- DPO alignment, evaluation metrics, anti-patterns
## Context Optimization
Manage context windows, compression, and attention-aware positioning. Optimize for tokens-per-task.
- `context-window-management.md` -- Five-layer architecture, anchored summarization, compression triggers
- `context-caching.md` -- Just-in-time loading, budget scaling, probe evaluation, CC 2.1.32+
## Evaluation
Evaluate LLM outputs with multi-dimension scoring, quality gates, and benchmarks.
- `evaluation-metrics.md` -- LLM-as-judge, RAGAS metrics, hallucination detection
- `evaluation-benchmarks.md` -- Quality gates, batch evaluation, pairwise comparison
## Prompt Engineering
Design, version, and optimize prompts for production LLM applications.
- `prompt-design.md` -- Chain-of-Thought, few-shot learning, pattern selection guide
- `prompt-testing.md` -- Langfuse versioning, DSPy optimization, A/B testing, self-consistency
- `prompt-react-pattern.md` -- ReAct loop for tool-using agents, thought-action-observation format
- `prompt-optimization.md` -- Token reduction, cost optimization, model tiering, prompt spec format
## Upstream coverage (do not restate)
These topics are covered by their vendors' own documentation. This skill points
at them instead of teaching them; the rules above keep only our floors, ceilings
and scars. Our delta on all of it is in `references/ork-delta.md`.
| Topic | First-party source |
|-------|--------------------|
| Strict-mode tool schemas, structured outputs | https://platform.openai.com/docs/guides/function-calling |
| Anthropic `input_schema` / `tool_use` | https://docs.claude.com/en/docs/agents-and-tools/tool-use/overview |
| SSE client mechanics, reconnection, cancellation | https://developer.mozilla.org/en-US/docs/Web/API/EventSource |
| LoRA / QLoRA config, target modules, adapter merging | https://huggingface.co/docs/peft/developer_guides/lora |
| Unsloth training loop, 4-bit loading | https://docs.unsloth.ai/get-started/fine-tuning-llms-guide |
| DPO, preference pairs, beta tuning, RLHF comparison | https://huggingface.co/docs/trl/dpo_trainer |
| SFT dataset formats (Alpaca, ChatML) | https://huggingface.co/docs/trl/sft_trainer |
| Embedding similarity for dataset dedup | https://sbert.net/ |
| Fine-tune vs prompt vs RAG decision framework | https://platform.openai.com/docs/guides/optimizing-llm-accuracy |
| Vendor token pricing (never hardcode it here) | https://platform.openai.com/docs/pricing |
## Supporting Files
- `references/ork-delta.md` -- our delta: scars, house ceilings, retired-file provenance
- `references/model-selection.md` -- local model comparison by task and hardware
- `scripts/create-lora-config.md` -- LoRA config scaffold with auto-detected model type
## Key Decisions
| Decision | Recommendation |
|----------|----------------|
| Tool schema mode | `strict: true` (2026 best practice) |
| Tool count | Load roughly 5-15 up front; defer the rest behind a non-deferred tool search tool |
| Streaming protocol | SSE for web, WebSocket for bidirectional |
| Buffer size | 50-200 tokens |
| Local model (reasoning) | `deepseek-r1:70b` |
| Local model (coding) | `qwen2.5-coder:32b` |
| Fine-tuning approach | LoRA/QLoRA (try prompting first) |
| LoRA rank | 16-64 typical |
| Training epochs | 1-3 (more risks overfitting) |
| Context compression | Anchored iterative (60-80%) |
| Compress trigger | 70% utilization, target 50% |
| Judge model | `claude-haiku-5-5` (cost tier), `gpt-5.5`, or `gemini-3.8-flash` (Google's GA general-purpose model and a third vendor for the different-model rule; price lives in `models.vocab.json`, not here) |
| Quality threshold | 0.7 production, 0.6 drafts |
| Few-shot examples | 3-5 diverse, representative |
| Prompt versioning | Langfuse with labels |
| Auto-optimization | DSPy MIPROv2 |
## Related Skills
- `ork-extras:rag-retrieval` -- Embedding patterns, when RAG is better than fine-tuning
- `agent-loops` -- Multi-step tool use with reasoning
- `llm-evaluation` -- Evaluate fine-tuned and local models
- `langfuse-observability` -- Track training experiments
## Capability Details
### function-calling
**Keywords:** tool, function, define tool, tool schema, function schema, strict mode, parallel tools
**Solves:**
- Define tools with clear descriptions and strict schemas
- Execute tool calls in parallel with asyncio.gather
- Validate inputs and handle errors in tool execution loops
### streaming
**Keywords:** streaming, SSE, Server-Sent Events, real-time, backpressure, token stream
**Solves:**
- Stream LLM tokens via SSE endpoints
- Handle tool calls within streams
- Manage backpressure with bounded queues
### local-inference
**Keywords:** Ollama, local, self-hosted, model selection, GPU, Apple Silicon
**Solves:**
- Set up Ollama for local LLM inference
- Select models based on task and hardware
- Optimize GPU usage and CI integration
### fine-tuning
**Keywords:** LoRA, QLoRA, fine-tune, DPO, synthetic data, PEFT, alignment
**Solves:**
- Configure LoRA/QLoRA for parameter-efficient training
- Generate and validate synthetic training data
- Align models with DPO and evaluate results
© OrchestKit · MIT · yonatangross/orchestkit
Similar skills
All skills for AI and ML →llm-integration by other authors comes first.