# flash-moe-inference

Runs the Qwen3.5-397B Mixture-of-Experts model on Apple Silicon by streaming expert weights directly from an SSD in pure C and Metal. Designed for Macs with 48GB of memory.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1100 installs · ★ 88
- Author: reason-machines
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/reason-machines/trending-skills/tree/HEAD/skills/flash-moe-inference/SKILL.md
- Updated: 2026-10-09
- HTML: https://affpapa.org/en/nevibecode/skills/flash-moe-inference

## What it does

This skill guides coding assistants through building and running the Flash-MoE execution runtime. Instead of attempting to fit the entire 209GB parameter set into memory, the engine streams four active experts per layer from NVMe storage on demand. Hand-written Metal compute kernels handle matrix multiplication directly on the GPU, achieving speeds above 4.4 tokens per second without Python runtime overhead.

The project compiles `infer.m`, an interactive terminal chat `chat.m`, and an isolated MoE benchmark `main.m` using a standard `Makefile`. Python is utilized strictly offline to extract tensors from safetensors archives into binary files. Requantizing experts down to 2-bit saves roughly 89GB of storage, though doing so compromises structured JSON generation and tool calls.

Execution strictly requires an Apple Silicon machine like M3 Max, 48GB unified RAM, macOS 26 or Darwin 25, and at least 210GB free SSD space. Systems missing these hardware prerequisites cannot run the pipeline.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/trending-skills --skill flash-moe-inference -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/flash-moe-inference/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/flash-moe-inference/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/trending-skills --skill flash-moe-inference -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/flash-moe-inference/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/trending-skills --skill flash-moe-inference -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/flash-moe-inference/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install reason-machines/trending-skills flash-moe-inference
```

2. **Manually: put the files in place**
- File: `.github/skills/flash-moe-inference/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/trending-skills --skill flash-moe-inference -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/reason-machines/trending-skills --path skills/flash-moe-inference
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/flash-moe-inference/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add reason-machines/trending-skills --skill flash-moe-inference -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/reason-machines/trending-skills/tree/HEAD/skills/flash-moe-inference/SKILL.md

## When to use it
- compile the C and Metal inference engine using make
- pack safetensors files into quantized binary slices for SSD streaming
- start a local terminal chat with function calling support
- benchmark forward-pass expert throughput on Apple Silicon hardware

## Example prompts
- Build the Flash-MoE engine with make and inspect the compiled binaries
- Help unpack the Qwen3.5 safetensors directory into 4-bit expert chunks
- Run infer with the prompt 'Explain quantum computing' for 100 tokens

## Similar
- [skill-creator](https://affpapa.org/en/nevibecode/skills/skill-creator.md) — skill · 414180 installs
- [writing-for-agents](https://affpapa.org/en/nevibecode/skills/writing-for-agents.md) — skill · 466719 installs
- [media-use](https://affpapa.org/en/nevibecode/skills/media-use.md) — skill · 1100000 installs
- [mcp-builder](https://affpapa.org/en/nevibecode/skills/mcp-builder.md) — skill · 130949 installs
- [subagent-driven-development](https://affpapa.org/en/nevibecode/skills/subagent-driven-development.md) — skill · 237746 installs
- [orchestration](https://affpapa.org/en/nevibecode/skills/orchestration.md) — skill · 349800 installs

From the same set: [reason-machines/trending-skills](https://affpapa.org/en/nevibecode/by/reason-machines/trending-skills.md) — 139 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md