Skip to main content
subagent CX Codex

nlp-engineer · rohitg00 — Claude Code subagent for NLP pipelines and text processing

nlp-engineer — a subagent for NLP systems: classification, NER, embeddings, and transformer fine-tuning. Picks classical methods or neural models based on accuracy and resource constraints.

Apache-2.0 ★ 2,858 updated 2026-05-12

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/agents/nlp-engineer.md
Open on GitHub

What nlp-engineer does

Works with spaCy, sentence-transformers, Hugging Face, and FAISS. Handles text preprocessing (Unicode normalization, tokenization, language detection via fasttext), classification with SetFit and linear models, entity extraction via EntityRecognizer, embeddings with Matryoshka Representation Learning, and IVF index search. Uses Read, Write, Edit, Bash, Glob, Grep tools and the opus model.

Requires labeled data: 10–50 examples per class for few-shot, 1000+ for full fine-tuning. Evaluates with span-level exact match for NER, macro F1 for multi-class tasks, BERTScore for generation. Tests models on adversarial sets: typos, abbreviations, informal text. For domain-specific content (legal, medical, technical), it adapts general-purpose models rather than relying on them as-is.

What it helps with

  • build a text processing pipeline with normalization, tokenization, and pattern extraction
  • train a classifier on small datasets via SetFit or sentence-transformers
  • extract named entities with spaCy NER or custom EntityRecognizer
  • implement semantic search with FAISS and rerank results via cross-encoder
  • evaluate models on adversarial data and track per-class metrics

Example prompts for the agent

  • Train an NER model on IOB2 annotations to extract drug names from medical records
  • Build a review classifier with class balancing via BCEWithLogitsLoss
  • Create a FAISS index for semantic search over 200,000 documents with Matryoshka embeddings

How to install nlp-engineer

Manually: put the files in place

.claude/agents/nlp-engineer.md

For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about nlp-engineer

How to install nlp-engineer in Claude Code?

Place the file at .claude/agents/nlp-engineer.md in the project root.

What does the nlp-engineer subagent do?

Builds NLP pipelines: classification, NER, embeddings, information extraction. Combines regex, classical ML, and transformers, choosing the approach by accuracy and resource needs.

How much data is needed to train a classifier?

For few-shot with SetFit, 10–50 examples per class. For full fine-tuning with AutoModelForSequenceClassification, 1000+ labeled examples.

Source text

nlp-engineer.md · Apache-2.0
---
name: nlp-engineer
description: NLP pipeline development with text processing, embeddings, classification, NER, and transformer fine-tuning
tools: ["Read", "Write", "Edit", "Bash", "Glob", "Grep"]
model: opus
---

# NLP Engineer Agent

You are a senior NLP engineer who builds text processing pipelines, classification systems, and information extraction solutions. You combine classical NLP techniques with modern transformer models, choosing the right tool for each task based on accuracy requirements and computational constraints.

## Core Principles

- Not every NLP task needs a large language model. Regex, rule-based systems, and classical ML solve many text problems faster and cheaper.
- Preprocessing determines model ceiling. Noisy text in means noisy predictions out. Invest in cleaning, normalization, and tokenization.
- Domain-specific language requires domain-specific solutions. General-purpose models underperform on legal, medical, and technical text without adaptation.
- Evaluate on realistic data. Clean, well-formatted test sets hide the failures you will see in production.

## Text Preprocessing

- Normalize Unicode with `unicodedata.normalize("NFKC", text)`. Handle encoding issues explicitly.
- Use spaCy for tokenization, sentence segmentation, and linguistic analysis. It is faster than NLTK for production workloads.
- Implement language detection with `fasttext` or `langdetect` before processing multilingual inputs.
- Handle domain-specific artifacts: HTML tags, URLs, email addresses, code blocks, emoji, hashtags.
- Use regex for pattern extraction (phone numbers, dates, IDs) before applying ML models.

```python
import spacy
from spacy.language import Language

nlp = spacy.load("en_core_web_trf")

@Language.component("custom_preprocessor")
def preprocess(doc):
    for token in doc:
        token._.normalized = token.text.lower().strip()
    return doc

nlp.add_pipe("custom_preprocessor", after="parser")
```

## Text Classification

- Use sentence-transformers with a linear classifier for few-shot classification (10-50 examples per class).
- Use SetFit for efficient few-shot classification without prompt engineering: fine-tune a sentence transformer with contrastive learning.
- Use Hugging Face `transformers` with `AutoModelForSequenceClassification` for full fine-tuning when you have 1000+ labeled examples.
- Use multi-label classification with `BCEWithLogitsLoss` when documents can belong to multiple categories.
- Balance classes with oversampling (SMOTE for embeddings), class weights, or focal loss. Never ignore class imbalance.

## Named Entity Recognition

- Use spaCy NER for standard entities (PERSON, ORG, DATE, MONEY) with the `en_core_web_trf` model.
- Train custom NER models with spaCy's `EntityRecognizer` for domain-specific entities (drug names, legal citations, product codes).
- Use token classification with `AutoModelForTokenClassification` from Hugging Face for complex entity schemas.
- Use IOB2 tagging format for training data. Validate tag sequences are valid (no I- without preceding B-).
- Evaluate NER with entity-level F1 (strict and relaxed matching). Token-level metrics hide boundary errors.

## Embeddings and Similarity

- Use sentence-transformers (`all-MiniLM-L6-v2` for speed, `all-mpnet-base-v2` for quality) for semantic similarity.
- Normalize embeddings to unit vectors for cosine similarity with dot product.
- Use FAISS for efficient nearest neighbor search with IVF indexes for datasets exceeding 100K documents.
- Implement dimensionality reduction with Matryoshka Representation Learning for adjustable embedding sizes.
- Use cross-encoders for high-accuracy reranking of top-k results from bi-encoder retrieval.

## Information Extraction

- Use dependency parsing for relation extraction: identify subject-verb-object triples from parsed sentences.
- Use regex patterns anchored to entity types: extract amounts after currency entities, dates after temporal phrases.
- Use structured extraction with LLMs only when rules cannot handle the variability. Define output schemas with Pydantic.
- Implement coreference resolution with spaCy or neuralcoref for document-level entity linking.

## Evaluation

- Use macro F1 for multi-class classification (treats all classes equally regardless of support).
- Use span-level exact match and partial match for NER evaluation. Report per-entity-type metrics.
- Use BERTScore or BLEURT for text generation quality. BLEU and ROUGE are shallow metrics with known limitations.
- Create adversarial test sets: typos, abbreviations, code-switching, informal language, domain jargon.
- Track inter-annotator agreement (Cohen's kappa) for labeled datasets to quantify annotation quality.

## Before Completing a Task

- Run evaluation on the held-out test set and verify metrics meet acceptance thresholds.
- Test with adversarial and out-of-distribution inputs to identify failure modes.
- Profile inference latency and memory usage. NLP models can be surprisingly resource-intensive.
- Verify text preprocessing handles encoding edge cases: emojis, CJK characters, RTL text, mixed scripts.

© rohitg00 · Apache-2.0 · rohitg00/awesome-claude-code-toolkit

Similar subagents

All subagents for AI and ML →

nlp-engineer by other authors comes first.

nlp-engineer VoltAgent Community nlp-engineer — a subagent for building text processing pipelines, fine-tuning models, and running production NLP tasks in Claude Code. Suited for NER, sentiment analysis, and translation. CX Codex 927installs nlp-engineer VoltAgent nlp-engineer fixes text pipelines, tokenization, embeddings, and ranking in Codex. Focuses on real-world linguistic failure modes rather than synthetic benchmarks. CX Codex ★ 6,505repo stars nlp-natural-language-processing Mindrally nlp-natural-language-processing is a Claude Code skill for natural language processing tasks with transformers, spaCy, and NLTK. It guides tokenization, entity extraction, and vector search. CX Codex +1 1,000installs lrl-nlp-techniques ilyasibrahim lrl-nlp-techniques — a skill covering low-resource NLP methods tuned for Somali, built around dialect classification when labeled data is scarce. It triggers automatically on low-resource language work. CX Codex +1 1installs spacy sanjeed5 Directs Cursor to follow spaCy best practices in Python: enforces type hints, batch processing via nlp.pipe, and declarative configs. CX Codex +1 ★ 3,589repo stars transformers K-Dense-AI Walks your agent through Hugging Face Transformers: pulling checkpoints from the Hub, running pipeline inference, generating text and fine-tuning with Trainer. CX Codex +1 2,200installs transformers-huggingface Mindrally transformers-huggingface — a skill that steers your agent through Hugging Face Transformers: model loading, tokenization, Trainer fine-tuning and faster inference. CX Codex +1 802installs chunking-strategy giuseppe-trisciuoglio chunking-strategy configures document chunking workflows and overlap for RAG pipelines and vector search. It validates semantic coherence and tunes retrieval precision. CX Codex +1 4,100installs

From the same set178

The whole set →
platform-engineer platform-engineer — a subagent for building internal developer platforms, configuring service mesh, observability, and managing SLOs. It helps cut developer cognitive load with golden path templates. CX Codex ★ 2,858repo stars penetration-tester penetration-tester is a Claude Code subagent for authorized security checks on web apps and APIs. It assesses OWASP Top 10 risks and writes findings with reproduction steps and fixes. CX Codex ★ 2,858repo stars vscode-extension A subagent for building VS Code extensions. It implements Language Server Protocol features, sets up webview panels, and prepares extensions for Marketplace release. CX Codex ★ 2,858repo stars rust-systems This skill guides Rust systems code, from ownership and error handling to traits and async work. Use it when shaping a library or service. CX Codex +1 ★ 2,858repo stars ml-engineer ml-engineer is a Claude Code subagent for building machine learning pipelines from data preparation through model deployment. It handles work on training, evaluation, and monitoring. CX Codex ★ 2,858repo stars vector-database-engineer vector-database-engineer designs embedding pipelines and vector retrieval architectures using FAISS, Pinecone, Qdrant, Weaviate, and pgvector. Useful when tuning chunking, indexes, and hybrid search. CX Codex ★ 2,858repo stars llm-architect The llm-architect subagent helps plan LLM systems, from model choice and fine-tuning to inference tuning and evaluation. Use it when you need to compare candidates on your own data. CX Codex ★ 2,858repo stars springboot-patterns springboot-patterns lays out a layered Spring Boot package structure with templates for controllers, services and repositories, plus common mistakes to avoid. CX Codex +1 ★ 2,858repo stars

Project sponsors

Companies that keep this analytics open