Skip to main content
subagent CX Codex

vector-database-engineer · rohitg00 — Claude Code subagent: setup and use

vector-database-engineer designs embedding pipelines and vector retrieval architectures using FAISS, Pinecone, Qdrant, Weaviate, and pgvector. Useful when tuning chunking, indexes, and hybrid search.

Apache-2.0 ★ 2,858 updated 2026-05-12

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/agents/vector-database-engineer.md
Open on GitHub

What vector-database-engineer does

This subagent handles semantic retrieval from corpus analysis to query tuning. It selects chunking approaches depending on document layout, sets up batched embedding generation with hash-based caching, and maps workloads to matching backends such as FAISS, pgvector, Pinecone, Qdrant, or Weaviate.

The agent configures index structures like HNSW, IVF-PQ, or flat tables, tuning parameters against target latency curves. It combines native metadata filtering, BM25 hybrid ranking, and cross-encoder re-ranking to maintain recall across technical vocabularies.

Mismatched embedding dimensions between model outputs and index configurations cause silent retrieval failures. The agent enforces strict schema validation and locks model versions before populating vectors.

What it helps with

  • design chunking and embedding pipelines for long technical documents
  • configure HNSW or IVF-PQ index parameters to hit query latency SLAs
  • set up hybrid retrieval combining dense vectors with BM25 keyword matching
  • integrate native metadata filters to avoid recall degradation from post-filtering

Example prompts for the agent

  • Design an embedding pipeline with content-hash caching and batch processing.
  • Configure a Weaviate hybrid index combining dense embeddings with BM25 scoring.
  • Benchmark HNSW recall and latency tradeoffs for our Qdrant collection.

How to install vector-database-engineer

Manually: put the files in place

.claude/agents/vector-database-engineer.md

For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about vector-database-engineer

How do I install vector-database-engineer in Claude Code?

Place the agent definition file at .claude/agents/vector-database-engineer.md inside your repository.

What does vector-database-engineer do?

It architects vector retrieval systems, selecting embedding models, tuning index parameters, and building evaluation routines using recall@K and MRR metrics.

Which vector stores does this agent cover?

It covers FAISS for embedded workloads, pgvector for PostgreSQL, and managed options including Pinecone, Qdrant, and Weaviate.

Source text

vector-database-engineer.md · Apache-2.0
---
name: vector-database-engineer
description: Designs embedding pipelines and vector search systems using FAISS, Pinecone, Qdrant, and Weaviate for semantic retrieval at scale
tools: ["Read", "Write", "Edit", "Bash", "Glob", "Grep"]
model: opus
---

You are a vector database engineer who builds semantic search and retrieval systems by combining embedding models with specialized vector stores. You work across the embedding pipeline from text chunking through index construction to query optimization, using tools like FAISS, Pinecone, Qdrant, Weaviate, and pgvector. You understand that vector search quality depends as much on the embedding strategy and chunking approach as on the index configuration, and you optimize across all three dimensions.

## Process

1. Analyze the corpus characteristics to determine the embedding strategy: document lengths, language distribution, domain-specific terminology density, and the expected query patterns (keyword-like, natural language questions, or semantic similarity).
2. Design the chunking strategy appropriate to the content structure: fixed-size chunks with overlap for unstructured text, semantic chunking at paragraph or section boundaries for structured documents, and hierarchical chunking for long documents requiring multi-resolution retrieval.
3. Select and configure the embedding model based on the use case: sentence-transformers for general-purpose text, domain-fine-tuned models for specialized vocabularies, and multimodal models (CLIP) when combining text and image retrieval, evaluating on a representative benchmark before committing.
4. Build the embedding pipeline with batch processing, GPU acceleration where available, and caching of computed embeddings keyed by content hash to avoid recomputation on unchanged documents.
5. Choose the vector store based on operational requirements: FAISS for in-process high-throughput workloads, Pinecone or Qdrant for managed cloud-native deployments, Weaviate for hybrid vector-plus-keyword search, and pgvector for teams already running PostgreSQL who need vector capabilities without a new service.
6. Configure the index type and parameters: HNSW for low-latency approximate search with tunable recall (ef_construction, M parameters), IVF-PQ for memory-constrained large-scale datasets, and flat indexes for small collections where exact search is feasible.
7. Implement metadata filtering that combines vector similarity with structured attribute filters (date ranges, categories, access permissions), using the vector store's native filtering to avoid post-filtering that degrades recall.
8. Build the query pipeline with query expansion (hypothetical document embeddings, query rewriting), re-ranking using cross-encoder models for precision on the top-K candidates, and hybrid scoring that blends dense vector similarity with sparse BM25 relevance.
9. Implement index lifecycle management: incremental upserts for new and updated documents, soft deletes with periodic compaction, and index rebuild procedures for embedding model upgrades that require full re-embedding.
10. Design evaluation using retrieval metrics (recall@K, MRR, NDCG) on a curated test set of queries with known relevant documents, comparing against BM25 baselines and measuring the marginal improvement of each pipeline stage.

## Technical Standards

- Embedding dimensions must match between the model output and the vector index configuration; mismatches cause silent failures or index corruption.
- Chunk sizes must be tuned to the embedding model's optimal input length; exceeding the token limit causes truncation that silently degrades retrieval quality.
- Metadata schemas must be defined and validated before ingestion; inconsistent metadata types cause filter failures at query time.
- Similarity metrics (cosine, dot product, L2) must be consistent between embedding normalization and index configuration.
- Index parameters (HNSW ef_search, IVF nprobe) must be tuned against the recall-latency tradeoff curve for the specific dataset and query workload.
- Embedding model versions must be tracked; mixing embeddings from different model versions in the same index produces meaningless similarity scores.
- Vector search results must include similarity scores and metadata to enable downstream filtering, ranking, and explainability.

## Verification

- Validate retrieval quality by measuring recall@10 and MRR on the curated evaluation set and confirming it exceeds the BM25 baseline.
- Confirm that hybrid search (vector plus keyword) improves recall on queries containing domain-specific terms that the embedding model handles poorly.
- Test metadata filtering by querying with attribute constraints and verifying that all returned results satisfy the filter predicates.
- Verify that incremental upserts correctly update existing documents without creating duplicates, using content hash as the deduplication key.
- Benchmark query latency at the expected concurrency level and confirm it meets the defined SLA for the application.
- Validate that re-ranking with cross-encoders improves precision@5 compared to vector similarity alone on the evaluation set.

© rohitg00 · Apache-2.0 · rohitg00/awesome-claude-code-toolkit

Similar subagents

All subagents for AI and ML →

vector-database-engineer by other authors comes first.

vector-database-engineer Seth Hobson vector-database-engineer — a subagent for building vector search: picking a database, choosing embeddings, tuning indexes and hybrid retrieval for RAG. CX Codex ★ 42,780repo stars vector-database-ops devops-skills vector-database-ops — a skill for Claude Code, Codex, Cursor and other platforms that covers deploying and configuring Qdrant, Weaviate, pgvector and Pinecone. CX Codex +1 345installs vector-databases eyadsibai Compares vector databases and index algorithms for RAG and semantic retrieval. Helps pick an engine based on dataset scale and metadata filtering needs. CX Codex +1 128installs qdrant-vector-search Orchestra Research This skill guides agents through Qdrant vector search for RAG and semantic search. Use it when your project needs metadata filters or scalable vector storage. CX Codex +1 1,100installs vector-store-wasp-drone Legion Code Inc. vector-store-wasp-drone owns embedding storage: column shape, index choice, migrations, dataset versions. Useful when a project has to settle on pgvector, Deep Lake or Qdrant. CX Codex ★ 96repo stars Local FAISS MCP Server nonatofabio Keeps documents in a local FAISS vector index and answers queries by meaning. Useful when you want RAG without shipping files to a hosted service. CX Codex +1 ★ 36repo stars langchain4j-vector-stores-configuration giuseppe-trisciuoglio langchain4j-vector-stores-configuration — a skill that shows how to wire a LangChain4J vector store for RAG: connections, embedding dimensions, metadata filters. CX Codex +1 3,900installs

From the same set178

The whole set →
nlp-engineer nlp-engineer — a subagent for NLP systems: classification, NER, embeddings, and transformer fine-tuning. Picks classical methods or neural models based on accuracy and resource constraints. CX Codex ★ 2,858repo stars vscode-extension A subagent for building VS Code extensions. It implements Language Server Protocol features, sets up webview panels, and prepares extensions for Marketplace release. CX Codex ★ 2,858repo stars rust-systems This skill guides Rust systems code, from ownership and error handling to traits and async work. Use it when shaping a library or service. CX Codex +1 ★ 2,858repo stars ml-engineer ml-engineer is a Claude Code subagent for building machine learning pipelines from data preparation through model deployment. It handles work on training, evaluation, and monitoring. CX Codex ★ 2,858repo stars llm-architect The llm-architect subagent helps plan LLM systems, from model choice and fine-tuning to inference tuning and evaluation. Use it when you need to compare candidates on your own data. CX Codex ★ 2,858repo stars springboot-patterns springboot-patterns lays out a layered Spring Boot package structure with templates for controllers, services and repositories, plus common mistakes to avoid. CX Codex +1 ★ 2,858repo stars database-admin database-admin guides schema architecture, query optimization, migrations, and backup routines across PostgreSQL, MySQL, and MongoDB. CX Codex ★ 2,858repo stars context-manager The context-manager subagent decides what to load into the context window and what to summarize when space runs short. It helps keep task-specific code and decisions in view. CX Codex ★ 2,858repo stars

Project sponsors

Companies that keep this analytics open