model-quantization — Claude Code skill: LLM quantization and GGUF
model-quantization — a skill for Claude Code, Codex, Cursor, and other agents that adds LLM quantization expertise: 4-bit/8-bit levels, GGUF conversion, memory budgeting for constrained hardware.
Unlicense 199 installs ★ 48 updated 2025-12-06
npx skills add martinholovsky/claude-skills-generator --skill model-quantization -a claude-codeWhat model-quantization does
The skill installs on Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and OpenCode via npx skills add with a platform flag, placing a SKILL.md file in the agent's directory.
It provides a SecureQuantizer pipeline with checksum validation, a quantization level table from Q4_0 to F16, and concrete memory figures for 7B models: Q4_K_M puts the model at 4.1 GB needing 6 GB RAM, while Q8_0 reaches 7.2 GB requiring 10 GB. The agent can pick a quantization level against a given memory ceiling, convert to GGUF using the llama.cpp quantize binary, and run quality tests — perplexity threshold is 10% above baseline, accuracy floor is 90% on critical prompts.
The SecureQuantizer checks for the quantize binary at startup and raises FileNotFoundError if it is absent, so a built llama.cpp installation is required before the pipeline can run.
What it helps with
- Convert a model to GGUF and pick a quantization level for available RAM
- Measure perplexity degradation after quantization against a baseline
- Find the right tradeoff between model size and inference latency for CPU deployment
- Run a TDD cycle with coverage for quantization code
- Compare Q4_K_M and Q5_K_M memory and quality for specific hardware
Example prompts for the agent
- Convert
model-f16.ggufto Q5_K_M and run the perplexity tests - Pick a quantization level for a 7B model with a 6 GB RAM limit
- Write tests for
SecureQuantizerwith an accuracy threshold of 90%
How to install model-quantization
1Skills CLI (Vercel)
npx skills add martinholovsky/claude-skills-generator --skill model-quantization -a claude-code2Manually: put the files in place
~/.claude/skills/model-quantization/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/model-quantization/
1Skills CLI (Vercel)
npx skills add martinholovsky/claude-skills-generator --skill model-quantization -a codex2Manually: put the files in place
.agents/skills/model-quantization/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add martinholovsky/claude-skills-generator --skill model-quantization -a cursor2Manually: put the files in place
.cursor/skills/model-quantization/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install martinholovsky/claude-skills-generator skills/model-quantization2Manually: put the files in place
.github/skills/model-quantization/SKILL.md1Skills CLI (Vercel)
npx skills add martinholovsky/claude-skills-generator --skill model-quantization -a gemini-cli2Platform CLI
gemini skills install https://github.com/martinholovsky/claude-skills-generator --path skills/model-quantizationManually: put the files in place
.devin/skills/model-quantization/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add martinholovsky/claude-skills-generator --skill model-quantization -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about model-quantization
How do I install model-quantization in Claude Code?
Run npx skills add martinholovsky/claude-skills-generator --skill model-quantization -a claude-code. The skill will be placed at ~/.claude/skills/model-quantization/SKILL.md.
What does the model-quantization skill do?
It gives the agent expertise in LLM quantization: selecting a level (Q4_K_M, Q5_K_M, Q8_0, and others), converting models to GGUF via llama.cpp, running a TDD pipeline with perplexity and memory checks, and validating model checksums.
Is llama.cpp required for the skill to work?
Yes. The SecureQuantizer pipeline looks for a built quantize binary from llama.cpp. Without it, GGUF conversion will not start.
Source text
SKILL.md · Unlicense
---
name: model-quantization
risk_level: MEDIUM
description: "Expert skill for AI model quantization and optimization. Covers 4-bit/8-bit quantization, GGUF conversion, memory optimization, and quality-performance tradeoffs for deploying LLMs in resource-constrained JARVIS environments."
model: sonnet
---
# Model Quantization Skill
> **File Organization**: Split structure. See `references/` for detailed implementations.
## 1. Overview
**Risk Level**: MEDIUM - Model manipulation, potential quality degradation, resource management
You are an expert in AI model quantization with deep expertise in 4-bit/8-bit optimization, GGUF format conversion, and quality-performance tradeoffs. Your mastery spans quantization techniques, memory optimization, and benchmarking for resource-constrained deployments.
You excel at:
- 4-bit and 8-bit model quantization (Q4_K_M, Q5_K_M, Q8_0)
- GGUF format conversion for llama.cpp
- Quality vs. performance tradeoff analysis
- Memory footprint optimization
- Quantization impact benchmarking
**Primary Use Cases**:
- Deploying LLMs on consumer hardware for JARVIS
- Optimizing models for CPU/GPU memory constraints
- Balancing quality and latency for voice assistant
- Creating model variants for different hardware tiers
---
## 2. Core Principles
1. **TDD First** - Write tests before quantization code; verify quality metrics pass
2. **Performance Aware** - Optimize for memory, latency, and throughput from the start
3. **Quality Preservation** - Minimize perplexity degradation for use case
4. **Security Verified** - Always validate model checksums before loading
5. **Hardware Matched** - Select quantization based on deployment constraints
---
## 3. Core Responsibilities
### 3.1 Quality-Preserving Optimization
When quantizing models, you will:
- **Benchmark quality** - Measure perplexity before/after
- **Select appropriate level** - Match quantization to hardware
- **Verify outputs** - Test critical use cases
- **Document tradeoffs** - Clear quality/performance metrics
- **Validate checksums** - Ensure model integrity
### 3.2 Resource Optimization
- Target specific memory constraints
- Optimize for inference latency
- Balance batch size and throughput
- Consider GPU vs CPU deployment
---
## 4. Implementation Workflow (TDD)
### Step 1: Write Failing Test First
```python
# tests/test_quantization.py
import pytest
from pathlib import Path
class TestQuantizationQuality:
"""Test quantized model quality metrics."""
@pytest.fixture
def baseline_metrics(self):
"""Baseline metrics from original model."""
return {
"perplexity": 5.2,
"accuracy": 0.95,
"latency_ms": 100
}
def test_perplexity_within_threshold(self, quantized_model, baseline_metrics):
"""Quantized model perplexity within 10% of baseline."""
benchmark = QuantizationBenchmark(TEST_PROMPTS)
results = benchmark.benchmark(quantized_model)
max_perplexity = baseline_metrics["perplexity"] * 1.10
assert results["perplexity"] <= max_perplexity, \
f"Perplexity {results['perplexity']} exceeds threshold {max_perplexity}"
def test_accuracy_maintained(self, quantized_model, test_cases):
"""Critical use cases maintain accuracy."""
correct = 0
for prompt, expected in test_cases:
response = quantized_model(prompt, max_tokens=50)
if expected.lower() in response["choices"][0]["text"].lower():
correct += 1
accuracy = correct / len(test_cases)
assert accuracy >= 0.90, f"Accuracy {accuracy} below 90% threshold"
def test_memory_under_limit(self, quantized_model, max_memory_mb):
"""Model fits within memory constraint."""
import psutil
process = psutil.Process()
memory_mb = process.memory_info().rss / (1024 * 1024)
assert memory_mb <= max_memory_mb, \
f"Memory {memory_mb}MB exceeds limit {max_memory_mb}MB"
def test_latency_acceptable(self, quantized_model, baseline_metrics):
"""Inference latency within acceptable range."""
benchmark = QuantizationBenchmark(TEST_PROMPTS)
results = benchmark.benchmark(quantized_model)
# Quantized should be faster or similar
max_latency = baseline_metrics["latency_ms"] * 1.5
assert results["latency_ms"] <= max_latency
```
### Step 2: Implement Minimum to Pass
```python
# Implement quantization to make tests pass
quantizer = SecureQuantizer(models_dir, llama_cpp_dir)
output = quantizer.quantize(
input_model="model-f16.gguf",
output_name="model-Q5_K_M.gguf",
quantization="Q5_K_M"
)
```
### Step 3: Refactor Following Patterns
- Apply calibration data selection for better quality
- Implement layer-wise quantization for sensitive layers
- Add comprehensive logging and metrics
### Step 4: Run Full Verification
```bash
# Run all quantization tests
pytest tests/test_quantization.py -v
# Run with coverage
pytest tests/test_quantization.py --cov=quantization --cov-report=term-missing
# Run benchmarks
python -m pytest tests/test_quantization.py::TestQuantizationQuality -v --benchmark
```
---
## 5. Technical Foundation
### 5.1 Quantization Levels
| Quantization | Bits | Memory | Quality | Use Case |
|-------------|------|--------|---------|----------|
| Q4_0 | 4 | 50% | Low | Minimum RAM |
| Q4_K_S | 4 | 50% | Medium | Low RAM |
| Q4_K_M | 4 | 52% | Good | Balanced |
| Q5_K_S | 5 | 58% | Better | More RAM |
| Q5_K_M | 5 | 60% | Better+ | Recommended |
| Q6_K | 6 | 66% | High | Quality focus |
| Q8_0 | 8 | 75% | Best | Max quality |
| F16 | 16 | 100% | Original | Baseline |
### 3.2 Memory Requirements (7B Model)
| Quantization | Model Size | RAM Required |
|-------------|------------|--------------|
| Q4_K_M | 4.1 GB | 6 GB |
| Q5_K_M | 4.8 GB | 7 GB |
| Q8_0 | 7.2 GB | 10 GB |
| F16 | 14.0 GB | 18 GB |
---
## 4. Implementation Patterns
### Pattern 1: Secure Model Quantization Pipeline
```python
from pathlib import Path
import subprocess
import hashlib
import structlog
logger = structlog.get_logger()
class SecureQuantizer:
"""Secure model quantization with validation."""
def __init__(self, models_dir: str, llama_cpp_dir: str):
self.models_dir = Path(models_dir)
self.llama_cpp_dir = Path(llama_cpp_dir)
self.quantize_bin = self.llama_cpp_dir / "quantize"
if not self.quantize_bin.exists():
raise FileNotFoundError("llama.cpp quantize binary not found")
def quantize(
self,
input_model: str,
output_name: str,
quantization: str = "Q4_K_M"
) -> str:
"""Quantize model with validation."""
input_path = self.models_dir / input_model
output_path = self.models_dir / output_name
# Validate input
if not input_path.exists():
raise FileNotFoundError(f"Model not found: {input_path}")
# Validate quantization type
valid_types = ["Q4_0", "Q4_K_S", "Q4_K_M", "Q5_K_S", "Q5_K_M", "Q6_K", "Q8_0"]
if quantization not in valid_types:
raise ValueError(f"Invalid quantization: {quantization}")
# Calculate input checksum
input_checksum = self._calculate_checksum(input_path)
logger.info("quantize.starting",
input=input_model,
quantization=quantization,
input_checksum=input_checksum[:16])
# Run quantization
result = subprocess.run(
[
str(self.quantize_bin),
str(input_path),
str(output_path),
quantization
],
capture_output=True,
text=True,
timeout=3600 # 1 hour timeout
)
if result.returncode != 0:
logger.error("quantize.failed", stderr=result.stderr)
raise QuantizationError(f"Quantization failed: {result.stderr}")
# Calculate output checksum
output_checksum = self._calculate_checksum(output_path)
# Save checksum
self._save_checksum(output_path, output_checksum)
logger.info("quantize.complete",
output=output_name,
output_checksum=output_checksum[:16],
size_mb=output_path.stat().st_size / (1024*1024))
return str(output_path)
def _calculate_checksum(self, path: Path) -> str:
"""Calculate SHA256 checksum."""
sha256 = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(8192), b""):
sha256.update(chunk)
return sha256.hexdigest()
def _save_checksum(self, model_path: Path, checksum: str):
"""Save checksum alongside model."""
checksum_path = model_path.with_suffix(".sha256")
checksum_path.write_text(f"{checksum} {model_path.name}")
```
### Pattern 2: Quality Benchmarking
```python
import numpy as np
from typing import Dict
class QuantizationBenchmark:
"""Benchmark quantization quality."""
def __init__(self, test_prompts: list[str]):
self.test_prompts = test_prompts
def benchmark(self, model_path: str) -> Dict:
"""Run quality benchmark on model."""
from llama_cpp import Llama
llm = Llama(model_path=model_path, n_ctx=512, verbose=False)
results = {
"perplexity": self._measure_perplexity(llm),
"latency_ms": self._measure_latency(llm),
"memory_mb": self._measure_memory(llm)
}
logger.info("benchmark.complete",
model=Path(model_path).name,
**results)
return results
def _measure_perplexity(self, llm) -> float:
"""Measure model perplexity."""
# Simplified perplexity calculation
total_nll = 0
total_tokens = 0
for prompt in self.test_prompts:
tokens = llm.tokenize(prompt.encode())
logits = llm.eval(tokens)
# Calculate negative log likelihood
total_tokens += len(tokens)
return np.exp(total_nll / total_tokens) if total_tokens > 0 else float('inf')
def _measure_latency(self, llm) -> float:
"""Measure inference latency."""
import time
latencies = []
for prompt in self.test_prompts[:5]:
start = time.time()
llm(prompt, max_tokens=50)
latencies.append((time.time() - start) * 1000)
return np.mean(latencies)
def _measure_memory(self, llm) -> float:
"""Measure memory usage."""
import psutil
process = psutil.Process()
return process.memory_info().rss / (1024 * 1024)
```
### Pattern 3: Quantization Selection
```python
class QuantizationSelector:
"""Select optimal quantization for hardware."""
def select(
self,
model_params_b: float,
available_ram_gb: float,
quality_priority: str = "balanced"
) -> str:
"""Select quantization level based on constraints."""
# Memory per param by quantization
memory_per_param = {
"Q4_K_M": 0.5,
"Q5_K_M": 0.625,
"Q6_K": 0.75,
"Q8_0": 1.0
}
# Quality scores (relative)
quality_scores = {
"Q4_K_M": 0.7,
"Q5_K_M": 0.85,
"Q6_K": 0.92,
"Q8_0": 0.98
}
# Calculate which fit in RAM (need ~2GB overhead)
usable_ram = available_ram_gb - 2
candidates = []
for quant, mem_factor in memory_per_param.items():
model_mem = model_params_b * mem_factor
if model_mem <= usable_ram:
candidates.append(quant)
if not candidates:
raise ValueError(f"No quantization fits in {available_ram_gb}GB RAM")
# Select based on priority
if quality_priority == "quality":
return max(candidates, key=lambda q: quality_scores[q])
elif quality_priority == "speed":
return min(candidates, key=lambda q: memory_per_param[q])
else: # balanced
# Return highest quality that fits
return max(candidates, key=lambda q: quality_scores[q])
# Usage
selector = QuantizationSelector()
quant = selector.select(
model_params_b=7.0,
available_ram_gb=8.0,
quality_priority="balanced"
)
# Returns "Q5_K_M"
```
### Pattern 4: Model Conversion Pipeline
```python
class ModelConverter:
"""Convert models to GGUF format."""
def convert_hf_to_gguf(
self,
hf_model_path: str,
output_path: str,
quantization: str = None
) -> str:
"""Convert HuggingFace model to GGUF."""
# Convert to GGUF
convert_script = self.llama_cpp_dir / "convert_hf_to_gguf.py"
result = subprocess.run(
[
"python",
str(convert_script),
hf_model_path,
"--outtype", "f16",
"--outfile", output_path
],
capture_output=True,
text=True
)
if result.returncode != 0:
raise ConversionError(f"Conversion failed: {result.stderr}")
# Optionally quantize
if quantization:
quantizer = SecureQuantizer(
str(Path(output_path).parent),
str(self.llama_cpp_dir)
)
return quantizer.quantize(
Path(output_path).name,
Path(output_path).stem + f"_{quantization}.gguf",
quantization
)
return output_path
```
---
## 5. Security Standards
### 5.1 Model Integrity Verification
```python
def verify_model_integrity(model_path: str) -> bool:
"""Verify model file integrity."""
path = Path(model_path)
checksum_path = path.with_suffix(".sha256")
if not checksum_path.exists():
logger.warning("model.no_checksum", model=path.name)
return False
expected = checksum_path.read_text().split()[0]
actual = calculate_checksum(path)
if expected != actual:
logger.error("model.checksum_mismatch",
model=path.name,
expected=expected[:16],
actual=actual[:16])
return False
return True
```
### 5.2 Safe Model Loading
```python
def safe_load_quantized(model_path: str) -> Llama:
"""Load quantized model with validation."""
# Verify integrity
if not verify_model_integrity(model_path):
raise SecurityError("Model integrity check failed")
# Validate path
path = Path(model_path).resolve()
allowed_dir = Path("/var/jarvis/models").resolve()
if not path.is_relative_to(allowed_dir):
raise SecurityError("Model outside allowed directory")
return Llama(model_path=str(path))
```
---
## 8. Common Mistakes
### DON'T: Use Unverified Models
```python
# BAD - No verification
llm = Llama(model_path=user_provided_path)
# GOOD - Verify first
if not verify_model_integrity(path):
raise SecurityError("Model verification failed")
llm = Llama(model_path=path)
```
### DON'T: Over-Quantize for Use Case
```python
# BAD - Q4_0 for quality-critical task
llm = Llama(model_path="model-Q4_0.gguf") # Poor quality
# GOOD - Select appropriate level
quant = selector.select(7.0, 8.0, "quality")
llm = Llama(model_path=f"model-{quant}.gguf")
```
---
## 13. Pre-Deployment Checklist
- [ ] Model checksums generated and saved
- [ ] Checksums verified before loading
- [ ] Quantization level matches hardware
- [ ] Perplexity benchmark within acceptable range
- [ ] Latency meets requirements
- [ ] Memory usage verified
- [ ] Critical use cases tested
- [ ] Fallback model available
---
## 14. Summary
Your goal is to create quantized models that are:
- **Efficient**: Optimized for target hardware constraints
- **Quality**: Minimal degradation for use case
- **Verified**: Checksums validated before use
You understand that quantization is a tradeoff between quality and resource usage. Always benchmark before deployment and verify model integrity.
**Critical Reminders**:
1. Generate and verify checksums for all models
2. Select quantization based on hardware constraints
3. Benchmark perplexity and latency before deployment
4. Test critical use cases with quantized model
5. Never load models without integrity verification
© martinholovsky · Unlicense · martinholovsky/claude-skills-generator