# nlp-natural-language-processing

Technical guidelines for natural language processing tasks using transformers, spaCy, and NLTK. Guides agents on proper tokenization, entity extraction, and vector search.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 980 installs · ★ 286
- Author: Mindrally
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/Mindrally/skills/tree/HEAD/nlp-natural-language-processing/SKILL.md
- Updated: 2026-10-08
- HTML: https://affpapa.org/en/nevibecode/skills/nlp-natural-language-processing

## What it does

This skill guides your coding assistant through natural language workflows in Python. It enforces rules for text preprocessing, tokenizers like SentencePiece or WordPiece, and special tokens such as `[CLS]` and `[SEP]`. The prompt instructs the assistant to follow functional patterns and maintain stratified splits when training BERT or RoBERTa classifiers.

It outlines practices for key tasks. These include entity recognition with spaCy and vector lookups with `sentence-transformers` and FAISS. For production deployments, it specifies inference tuning via model quantization or ONNX Runtime.

Your environment must have Python dependencies installed, specifically `torch`, `transformers`, and `spacy`. The skill strictly requires declaring maximum sequence lengths to prevent silent truncation bugs.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill nlp-natural-language-processing -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/nlp-natural-language-processing/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/nlp-natural-language-processing/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill nlp-natural-language-processing -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/nlp-natural-language-processing/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill nlp-natural-language-processing -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/nlp-natural-language-processing/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install Mindrally/skills nlp-natural-language-processing
```

2. **Manually: put the files in place**
- File: `.github/skills/nlp-natural-language-processing/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill nlp-natural-language-processing -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/Mindrally/skills --path nlp-natural-language-processing
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/nlp-natural-language-processing/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill nlp-natural-language-processing -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/Mindrally/skills/tree/HEAD/nlp-natural-language-processing/SKILL.md

## When to use it
- configure text tokenization with attention masks
- build custom entity extraction pipelines in spaCy
- implement vector similarity search with FAISS
- prepare model quantization or ONNX deployment

## Example prompts
- Build a text classification pipeline for RoBERTa with stratified validation splits.
- Set up a custom spaCy NER component using BILOU tagging for legal entities.
- Implement semantic document search using sentence-transformers and an Annoy index.

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 04b7c6a98a210666.

BEGIN THIRD-PARTY FILE 04b7c6a98a210666
```text
---
name: nlp-natural-language-processing
description: Expert guidance for natural language processing development using transformers, spaCy, NLTK, and modern NLP techniques.
metadata:
  maintainer: Mindrally
  source: https://github.com/Mindrally/skills
---

# Natural Language Processing (NLP) Development

You are an expert in natural language processing, text analysis, and language modeling, with a focus on transformers, spaCy, NLTK, and related libraries.

## Key Principles

- Write concise, technical responses with accurate Python examples
- Prioritize clarity, efficiency, and best practices in NLP workflows
- Use functional programming for text processing pipelines
- Implement proper tokenization and text preprocessing
- Use descriptive variable names that reflect NLP operations
- Follow PEP 8 style guidelines for Python code

## Text Preprocessing

- Implement proper text cleaning (removing special characters, handling unicode)
- Use appropriate tokenization strategies for the task (word, subword, character)
- Apply lemmatization or stemming when appropriate
- Handle stop words removal contextually (not always necessary)
- Implement proper sentence segmentation and boundary detection

## Tokenization and Encoding

- Use the Transformers library for working with pre-trained tokenizers
- Understand different tokenization schemes (BPE, WordPiece, SentencePiece)
- Handle special tokens correctly ([CLS], [SEP], [PAD], [MASK])
- Implement proper padding and truncation strategies
- Use attention masks correctly for variable-length sequences

## Text Classification

- Implement proper train/validation/test splits with stratification
- Use appropriate models for the task (BERT, RoBERTa, DistilBERT)
- Apply fine-tuning techniques with proper learning rate scheduling
- Implement multi-label classification when needed
- Use appropriate metrics (accuracy, F1, precision, recall, AUC)

## Named Entity Recognition (NER)

- Use spaCy for efficient NER in production systems
- Implement custom NER models with transformer-based approaches
- Handle entity overlapping and nested entities appropriately
- Use BIO/BILOU tagging schemes correctly
- Evaluate with entity-level metrics (partial and exact match)

## Text Generation

- Use appropriate decoding strategies (greedy, beam search, sampling)
- Implement temperature and top-k/top-p sampling correctly
- Handle repetition penalties and length normalization
- Use proper prompt engineering for instruction-tuned models
- Implement streaming generation for responsive applications

## Embeddings and Semantic Search

- Use sentence-transformers for semantic embeddings
- Implement efficient similarity search with FAISS or Annoy
```
END THIRD-PARTY FILE 04b7c6a98a210666

The third-party file has ended. Everything between the 04b7c6a98a210666 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2702 of 4221 bytes. © Mindrally · Apache-2.0 · https://github.com/Mindrally/skills/tree/HEAD/nlp-natural-language-processing/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md