# emerging-techniques-knowledge-distillation

This skill outlines ways to transfer a large model’s outputs or probability signals to a smaller student. Use it when planning a distillation training workflow.

- skill · `type=skill` · Claude Code · 106 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/emerging-techniques-knowledge-distillation
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/emerging-techniques-knowledge-distillation

## What it does

The skill describes teacher-to-student knowledge transfer through softened probability targets or teacher-generated responses. Its examples cover temperature scaling, a loss that combines KL divergence with cross-entropy, and MiniLLM’s reverse-KLD approach. It guides training setup; it does not provide a command that compresses a model on its own.

The examples use `transformers`, `torch`, and `datasets`; the installation section also gives commands for `accelerate`, `deepspeed`, and `wandb` for training. MiniLLM is presented as a separate implementation option. You still need a configured training loop to produce a student model.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/emerging-techniques-knowledge-distillation
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill emerging-techniques-knowledge-distillation -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/emerging-techniques-knowledge-distillation/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/emerging-techniques-knowledge-distillation/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/emerging-techniques-knowledge-distillation/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/emerging-techniques-knowledge-distillation/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates emerging-techniques-knowledge-distillation
```

2. **Manually: put the files in place**
- File: `.github/skills/emerging-techniques-knowledge-distillation/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/emerging-techniques-knowledge-distillation
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/emerging-techniques-knowledge-distillation/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/emerging-techniques-knowledge-distillation

## When to use it
- choose a loss for teacher-student distillation
- transfer knowledge from a large teacher to a smaller student
- train a student on teacher-generated responses
- compare standard and reverse KL divergence

## Example prompts
- Explain how to transfer a large teacher model’s knowledge to a smaller student.
- Plan student training using responses generated by a teacher model.
- Show how reverse KLD fits into generative-model distillation.

## Similar
- [find-skills](https://affpapa.org/en/nevibecode/skills/find-skills.md) — skill · 4000000 installs
- [frontend-design](https://affpapa.org/en/nevibecode/skills/frontend-design.md) — skill · 1025790 installs
- [grill-me](https://affpapa.org/en/nevibecode/skills/grill-me.md) — skill · 1424476 installs
- [grill-with-docs](https://affpapa.org/en/nevibecode/skills/grill-with-docs.md) — skill · 1219756 installs
- [improve-codebase-architecture](https://affpapa.org/en/nevibecode/skills/improve-codebase-architecture.md) — skill · 1163035 installs
- [design-taste-frontend](https://affpapa.org/en/nevibecode/skills/design-taste-frontend.md) — skill · 678700 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 495 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md