Skip to main content
skill CX Codex

doc-to-vector-dataset-generator — Claude Code skill

doc-to-vector-dataset-generator turns source documents into cleaned, chunked JSONL for embeddings and vector search. Use it when building document-processing or training datasets.

MIT 227 installs ★ 64 updated 2026-01-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add patricio0312rev/skills --skill doc-to-vector-dataset-generator -a claude-code
Open on GitHub

What doc-to-vector-dataset-generator does

The skill extracts text from PDF, DOCX, and HTML files, then removes noise such as extra whitespace, page markers, and URLs. It splits content at paragraph and sentence boundaries and attaches metadata for the source, chunk position, size, timestamp, code, tables, and detected language.

Before writing JSONL, it checks chunk length and content, filters out low-quality entries, and removes near-duplicates with TF-IDF and cosine similarity. Each remaining chunk becomes one JSON object per line. The quality gate accepts English-language chunks only, so non-English content will be filtered out during processing.

What it helps with

  • extract text from PDF, DOCX, and HTML
  • clean page noise and unwanted URLs
  • split documents at semantic boundaries
  • attach searchable metadata to chunks
  • deduplicate entries and export JSONL

Example prompts for the agent

  • Turn the documents in docs into a vector-search dataset and save it as dataset.jsonl.
  • Process manual.pdf, clean its text, split it into chunks, and attach metadata.
  • Filter the chunks, remove near-duplicates, and write the accepted records as JSONL.

How to install doc-to-vector-dataset-generator

1Skills CLI (Vercel)

npx skills add patricio0312rev/skills --skill doc-to-vector-dataset-generator -a claude-code

2Manually: put the files in place

~/.claude/skills/doc-to-vector-dataset-generator/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/doc-to-vector-dataset-generator/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about doc-to-vector-dataset-generator

How do I install doc-to-vector-dataset-generator in Claude Code?

Run npx skills add patricio0312rev/skills --skill doc-to-vector-dataset-generator -a claude-code, then check ~/.claude/skills/doc-to-vector-dataset-generator/SKILL.md.

What does doc-to-vector-dataset-generator do?

It extracts and cleans document text, creates semantic chunks, adds metadata, filters near-duplicates, and exports one JSON object per line for embeddings and vector search.

Why are some chunks rejected?

The quality check rejects entries that are too short or too long, have too few alphabetic characters, or are detected as a language other than en.

Source text

SKILL.md · MIT
---
name: doc-to-vector-dataset-generator
description: Converts documents into clean, chunked datasets suitable for embeddings and vector search. Produces chunked JSONL files with metadata, deduplication logic, and quality checks. Use when preparing "training data", "vector datasets", "document processing", or "embedding data".
---

# Doc-to-Vector Dataset Generator

Transform documents into high-quality vector search datasets.

## Pipeline Steps

1. **Extract text** from various formats (PDF, DOCX, HTML)
2. **Clean text** (remove noise, normalize)
3. **Chunk strategically** (semantic boundaries)
4. **Add metadata** (source, timestamps, classification)
5. **Deduplicate** (near-duplicate detection)
6. **Quality check** (length, content validation)
7. **Export JSONL** (one chunk per line)

## Text Extraction

```python
# PDF extraction
import pymupdf

def extract_pdf(filepath: str) -> str:
    doc = pymupdf.open(filepath)
    text = ""
    for page in doc:
        text += page.get_text()
    return text

# Markdown extraction
def extract_markdown(filepath: str) -> str:
    with open(filepath) as f:
        return f.read()
```

## Text Cleaning

```python
import re

def clean_text(text: str) -> str:
    # Remove extra whitespace
    text = re.sub(r'\s+', ' ', text)

    # Remove page numbers
    text = re.sub(r'Page \d+', '', text)

    # Remove URLs (optional)
    text = re.sub(r'http\S+', '', text)

    # Normalize unicode
    text = text.encode('utf-8', 'ignore').decode('utf-8')

    return text.strip()
```

## Semantic Chunking

```python
def semantic_chunk(text: str, max_chunk_size: int = 1000) -> List[str]:
    """Chunk at semantic boundaries (paragraphs, sentences)"""
    # Split by paragraphs first
    paragraphs = text.split('\n\n')

    chunks = []
    current_chunk = ""

    for para in paragraphs:
        if len(current_chunk) + len(para) <= max_chunk_size:
            current_chunk += para + "\n\n"
        else:
            if current_chunk:
                chunks.append(current_chunk.strip())
            current_chunk = para + "\n\n"

    if current_chunk:
        chunks.append(current_chunk.strip())

    return chunks
```

## Metadata Extraction

````python
def extract_metadata(filepath: str, chunk: str, chunk_idx: int) -> dict:
    return {
        "source": filepath,
        "chunk_id": f"{hash(filepath)}_{chunk_idx}",
        "chunk_index": chunk_idx,
        "char_count": len(chunk),
        "word_count": len(chunk.split()),
        "created_at": datetime.now().isoformat(),

        # Content classification
        "has_code": bool(re.search(r'```|def |class |function', chunk)),
        "has_table": bool(re.search(r'\|.*\|', chunk)),
        "language": detect_language(chunk),
    }
````

## Deduplication

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

def deduplicate_chunks(chunks: List[dict], threshold: float = 0.95) -> List[dict]:
    """Remove near-duplicate chunks"""
    texts = [c["text"] for c in chunks]

    # Compute TF-IDF vectors
    vectorizer = TfidfVectorizer()
    vectors = vectorizer.fit_transform(texts)

    # Compute pairwise similarity
    similarity_matrix = cosine_similarity(vectors)

    # Find duplicates
    to_remove = set()
    for i in range(len(chunks)):
        if i in to_remove:
            continue
        for j in range(i+1, len(chunks)):
            if similarity_matrix[i][j] > threshold:
                to_remove.add(j)

    # Return unique chunks
    return [c for i, c in enumerate(chunks) if i not in to_remove]
```

## Quality Checks

```python
def quality_check(chunk: dict) -> bool:
    """Validate chunk quality"""
    text = chunk["text"]

    # Min length check
    if len(text) < 50:
        return False

    # Max length check
    if len(text) > 5000:
        return False

    # Content check (not just numbers/symbols)
    alpha_ratio = sum(c.isalpha() for c in text) / len(text)
    if alpha_ratio < 0.5:
        return False

    # Language check (English only)
    if chunk["metadata"]["language"] != "en":
        return False

    return True
```

## JSONL Export

```python
import json

def export_jsonl(chunks: List[dict], output_path: str):
    """Export chunks as JSONL (one JSON object per line)"""
    with open(output_path, 'w') as f:
        for chunk in chunks:
            f.write(json.dumps(chunk) + '\n')

# Example output format
{
  "text": "Chunk text content here...",
  "metadata": {
    "source": "docs/auth.md",
    "chunk_id": "abc123_0",
    "chunk_index": 0,
    "char_count": 542,
    "word_count": 89,
    "has_code": true
  }
}
```

## Complete Pipeline

```python
def process_documents(input_dir: str, output_path: str):
    all_chunks = []

    # Process each document
    for filepath in glob(f"{input_dir}/**/*.md"):
        # Extract and clean
        text = extract_markdown(filepath)
        text = clean_text(text)

        # Chunk
        chunks = semantic_chunk(text)

        # Add metadata
        for i, chunk in enumerate(chunks):
            chunk_obj = {
                "text": chunk,
                "metadata": extract_metadata(filepath, chunk, i)
            }

            # Quality check
            if quality_check(chunk_obj):
                all_chunks.append(chunk_obj)

    # Deduplicate
    unique_chunks = deduplicate_chunks(all_chunks)

    # Export
    export_jsonl(unique_chunks, output_path)

    print(f"Processed {len(unique_chunks)} chunks")
```

## Best Practices

- Chunk at semantic boundaries
- Rich metadata for filtering
- Deduplicate aggressively
- Quality checks prevent garbage
- JSONL format for streaming
- Version your datasets

## Output Checklist

- [ ] Text extraction from all formats
- [ ] Cleaning pipeline implemented
- [ ] Semantic chunking strategy
- [ ] Metadata schema defined
- [ ] Deduplication logic
- [ ] Quality validation checks
- [ ] JSONL export format
- [ ] Dataset statistics logged

© patricio0312rev · MIT · patricio0312rev/skills

Similar skills

All skills for Data and databases →

From the same set148

The whole set →
rag-pipeline-builder rag-pipeline-builder designs a retrieval-augmented pipeline for a document assistant: chunking, metadata, hybrid search, reranking and a way to measure retrieval quality. CX Codex +1 251installs preview-environments-builder preview-environments-builder sets up ephemeral preview environments for pull requests with automated deployments, unique URLs, and automatic cleanup once a PR closes. CX Codex +1 250installs load-test-scenario-builder load-test-scenario-builder generates load testing plans, k6 automation scripts, traffic models, and threshold criteria to check service capacity. CX Codex +1 236installs graphql-schema-designer graphql-schema-designer is a Claude Code skill for shaping GraphQL APIs with types, queries, mutations, subscriptions, and resolvers. It activates for requests about schema design, GraphQL setup, or Apollo Server. CX Codex +1 230installs test-data-factory-builder The test-data-factory-builder Claude Code skill builds factories and builders for test data. It activates for requests about fixture factories, test data builders, or generating related records. CX Codex +1 229installs terraform-module-builder terraform-module-builder generates structured, reusable Terraform modules with input variable validation, outputs, and example configurations. CX Codex +1 229installs snapshot-test-refactorer snapshot-test-refactorer refactors fragile DOM and API snapshots into targeted assertions. Helps replace noisy snapshot diffs with tests focused on behavior and critical data. CX Codex +1 228installs performance-budget-setter The performance-budget-setter skill turns bundle, API, database, and render performance into measurable limits. Use it when CI checks and ongoing monitoring need clear targets. CX Codex +1 228installs

Project sponsors

Companies that keep this analytics open