fine-tuning-peft — Claude Code skill: LoRA and QLoRA for LLMs

What fine-tuning-peft does

The skill covers training 7B–70B models while updating under 1% of the weights. It builds on HuggingFace's PEFT library, which plugs into the transformers stack. Adapters come out in megabytes rather than gigabytes, so keeping several task-specific variants of one base model is cheaper than keeping full copies.

You get working code samples: plain LoRA through LoraConfig, QLoRA with 4-bit quantization via BitsAndBytesConfig, loading and merging adapters, picking rank and lora_alpha, plus target module lists for Llama, Mistral, Qwen, GPT-2, Falcon and BLOOM. A rank table spans 4 to 64, and the guidance suggests starting at 8 or 16.

Dependencies are peft>=0.13.0, transformers>=4.45.0, torch>=2.0.0 and bitsandbytes>=0.43.0. Skip the last one and quantization will not run. QLoRA targets a single 24GB card, at a cost of roughly 5% quality against full fine-tuning. For models under 1B parameters, or when the domain shift is large, the skill points you to full fine-tuning instead.