# llm-inference-scaling

Autoscales LLM inference clusters on Kubernetes using KEDA, Prometheus GPU metrics, and Redis queues. Built for unpredictable vLLM traffic and cutting costs with spot instances.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 343 installs · ★ 1177
- Author: devops-skills
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/tree/HEAD/infrastructure/local-ai/llm-inference-scaling/SKILL.md
- Updated: 2026-05-22
- HTML: https://affpapa.org/en/nevibecode/skills/llm-inference-scaling

## What it does

The skill provides production configurations to scale LLM engines like vLLM or TGI horizontally on Kubernetes. It connects KEDA to Prometheus metrics, triggering pod scaling based on waiting requests (`vllm_num_requests_waiting`) and GPU KV-cache saturation. For asynchronous batch workloads, it sets up KEDA `ScaledJob` resources driven by Redis queue depth.

To rein in hardware bills, the setup includes a mixed node-pool strategy. It favors spot GPU instances via Cluster Autoscaler priority maps, falling back to on-demand nodes through pod affinity. Cluster prerequisites include Helm 3, KEDA, and the NVIDIA GPU Operator with DCGM metrics enabled.

Autoscaling alone will not prevent request latency during sudden spikes if pods must start from scratch. Pulling large weights onto a GPU takes time, so keeping `minReplicaCount` above zero on critical deployments is mandatory.

## Requirements
- Works with: Kubernetes

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-inference-scaling -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/llm-inference-scaling/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/llm-inference-scaling/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-inference-scaling -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/llm-inference-scaling/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-inference-scaling -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/llm-inference-scaling/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install BagelHole/DevOps-Security-Agent-Skills llm-inference-scaling
```

2. **Manually: put the files in place**
- File: `.github/skills/llm-inference-scaling/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-inference-scaling -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/BagelHole/DevOps-Security-Agent-Skills --path infrastructure/local-ai/llm-inference-scaling
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/llm-inference-scaling/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-inference-scaling -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/BagelHole/DevOps-Security-Agent-Skills/tree/HEAD/infrastructure/local-ai/llm-inference-scaling/SKILL.md

## When to use it
- configure horizontal pod autoscaling for vLLM using request queue depth
- scale GPU pods based on KV-cache memory pressure in Prometheus
- run asynchronous batch inference using Redis queues and KEDA ScaledJob
- reduce infrastructure spend with spot GPU nodes and on-demand fallback
- deploy NVIDIA GPU Operator and DCGM metric collection with Helm

## Example prompts
- Generate a KEDA ScaledObject manifest for vLLM based on vllm_num_requests_waiting
- Configure pod affinity and Cluster Autoscaler priorities for spot GPU instances with on-demand fallback
- Create a KEDA ScaledJob to process LLM inference jobs from a Redis queue

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 2dd8536effe3abc8.

BEGIN THIRD-PARTY FILE 2dd8536effe3abc8
````text
---
name: llm-inference-scaling
description: Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads.
license: MIT
metadata:
  author: devops-skills
  version: "1.0"
---

# LLM Inference Scaling

Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.

## When to Use This Skill

Use this skill when:
- LLM API traffic is unpredictable and you need to scale up/down automatically
- Managing a fleet of vLLM or TGI inference pods on Kubernetes
- Reducing inference costs with spot/preemptible GPU instances
- Implementing queue-based autoscaling for batch inference jobs
- Building a multi-model serving platform that shares GPU resources

## Prerequisites

- Kubernetes cluster with GPU nodes (NVIDIA operator installed)
- KEDA (Kubernetes Event-Driven Autoscaler) installed
- Prometheus with GPU metrics (`dcgm-exporter` or `gpu-operator`)
- Helm 3+ for chart deployments

## GPU Node Setup

```bash
# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=true \
  --set dcgm.enabled=true \
  --set devicePlugin.enabled=true

# Verify GPU nodes are recognized
kubectl get nodes -l nvidia.com/gpu.present=true
kubectl describe node <gpu-node> | grep nvidia
```

## vLLM Deployment with GPU Resources

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-8b
  labels:
    app: vllm
    model: llama-3.1-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
      model: llama-3.1-8b
  template:
    metadata:
      labels:
        app: vllm
        model: llama-3.1-8b
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "meta-llama/Llama-3.1-8B-Instruct"
        - "--tensor-parallel-size"
````
END THIRD-PARTY FILE 2dd8536effe3abc8

The third-party file has ended. Everything between the 2dd8536effe3abc8 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2320 of 7912 bytes. © devops-skills · MIT · https://github.com/BagelHole/DevOps-Security-Agent-Skills/tree/HEAD/infrastructure/local-ai/llm-inference-scaling/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md