Skip to main content
skill CX Codex

clip-aware-embeddings — Claude Code skill: setup and features

clip-aware-embeddings guides AI coding agents to pick CLIP or dedicated computer vision models. It stops agents from misusing CLIP for counting or spatial reasoning.

MIT 188 installs ★ 274 updated 2026-10-30

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add curiositech/some_claude_skills --skill clip-aware-embeddings -a claude-code
Open on GitHub

What clip-aware-embeddings does

This skill guides the coding agent during image-text matching workflows. It retains standard CLIP for broad zero-shot classification and semantic photo search, while blocking its use whenever a task requires localized visual coordinates.

CLIP cannot count items. For numeric queries, this skill redirects the agent to detection models like DETR. Directional queries using terms like left or above go to models like SWIG or GQA instead of relying on a single collapsed vector. Without these guardrails, language models default to plain CLIP and fail.

Local scripts depend on Python libraries: transformers, pillow, torch, and sentence-transformers. You can test query suitability with python scripts/validate_clip_usage.py before executing model inference.

What it helps with

  • route visual search queries to standard CLIP embeddings
  • switch object counting from CLIP to detection models like DETR
  • prevent spatial relation failures in vision prompts
  • select dedicated classifiers for fine-grained object categories

Example prompts for the agent

  • Find the right vision model to search our photo archive for beach scenes
  • How do I count cars in this image without using standard CLIP?
  • Check whether my visual query needs a compositional model like PC-CLIP

How to install clip-aware-embeddings

1Skills CLI (Vercel)

npx skills add curiositech/some_claude_skills --skill clip-aware-embeddings -a claude-code

2Manually: put the files in place

~/.claude/skills/clip-aware-embeddings/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/clip-aware-embeddings/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about clip-aware-embeddings

How to install clip-aware-embeddings in Claude Code?

Run npx skills add curiositech/some_claude_skills --skill clip-aware-embeddings -a claude-code in your terminal. The skill file is stored at ~/.claude/skills/clip-aware-embeddings/SKILL.md.

What does clip-aware-embeddings do?

It prevents the agent from treating CLIP as a universal tool for computer vision. The skill routes counting, spatial logic, and attribute binding to specialized alternatives.

What dependencies are required to run the scripts?

You need Python with transformers, pillow, torch, and sentence-transformers installed. You can check your setup with python scripts/validate_setup.py.

Source text

SKILL.md · MIT
---
name: clip-aware-embeddings
description: Semantic image-text matching with CLIP and alternatives. Use for image search, zero-shot classification, similarity matching. NOT for counting objects, fine-grained classification (celebrities,
  car models), spatial reasoning, or compositional queries. Activate on "CLIP", "embeddings", "image similarity", "semantic search", "zero-shot classification", "image-text matching".
allowed-tools: Read,Write,Edit,Bash
metadata:
  category: AI & Machine Learning
  pairs-with:
  - skill: photo-content-recognition-curation-expert
    reason: Content-aware photo processing
  - skill: collage-layout-expert
    reason: Semantic image matching for layouts
  tags:
  - clip
  - embeddings
  - vision
  - similarity
  - zero-shot
---

# CLIP-Aware Image Embeddings

Smart image-text matching that knows when CLIP works and when to use alternatives.

## MCP Integrations

| MCP | Purpose |
|-----|---------|
| **Firecrawl** | Research latest CLIP alternatives and benchmarks |
| **Hugging Face** (if configured) | Access model cards and documentation |

## Quick Decision Tree

```
Your task:
├─ Semantic search ("find beach images") → CLIP ✓
├─ Zero-shot classification (broad categories) → CLIP ✓
├─ Counting objects → DETR, Faster R-CNN ✗
├─ Fine-grained ID (celebrities, car models) → Specialized model ✗
├─ Spatial relations ("cat left of dog") → GQA, SWIG ✗
└─ Compositional ("red car AND blue truck") → DCSMs, PC-CLIP ✗
```

## When to Use This Skill

✅ **Use for**:
- Semantic image search
- Broad category classification
- Image similarity matching
- Zero-shot tasks on new categories

❌ **Do NOT use for**:
- Counting objects in images
- Fine-grained classification
- Spatial understanding
- Attribute binding
- Negation handling

## Installation

```bash
pip install transformers pillow torch sentence-transformers --break-system-packages
```

**Validation**: Run `python scripts/validate_setup.py`

## Basic Usage

### Image Search

```python
from transformers import CLIPProcessor, CLIPModel
from PIL import Image

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

# Embed images
images = [Image.open(f"img{i}.jpg") for i in range(10)]
inputs = processor(images=images, return_tensors="pt")
image_features = model.get_image_features(**inputs)

# Search with text
text_inputs = processor(text=["a beach at sunset"], return_tensors="pt")
text_features = model.get_text_features(**text_inputs)

# Compute similarity
similarity = (image_features @ text_features.T).softmax(dim=0)
```

## Common Anti-Patterns

### Anti-Pattern 1: "CLIP for Everything"

**❌ Wrong**:
```python
# Using CLIP to count cars in an image
prompt = "How many cars are in this image?"
# CLIP cannot count - it will give nonsense results
```

**Why wrong**: CLIP's architecture collapses spatial information into a single vector. It literally cannot count.

**✓ Right**:
```python
from transformers import DetrImageProcessor, DetrForObjectDetection

processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")

# Detect objects
results = model(**processor(images=image, return_tensors="pt"))
# Filter for cars and count
car_detections = [d for d in results if d['label'] == 'car']
count = len(car_detections)
```

**How to detect**: If query contains "how many", "count", or numeric questions → Use object detection

---

### Anti-Pattern 2: Fine-Grained Classification

**❌ Wrong**:
```python
# Trying to identify specific celebrities with CLIP
prompts = ["Tom Hanks", "Brad Pitt", "Morgan Freeman"]
# CLIP will perform poorly - not trained for fine-grained face ID
```

**Why wrong**: CLIP trained on coarse categories. Fine-grained faces, car models, flower species require specialized models.

**✓ Right**:
```python
# Use a fine-tuned face recognition model
from transformers import AutoFeatureExtractor, AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained(
    "microsoft/resnet-50"  # Then fine-tune on celebrity dataset
)
# Or use dedicated face recognition: ArcFace, CosFace
```

**How to detect**: If query asks to distinguish between similar items in same category → Use specialized model

---

### Anti-Pattern 3: Spatial Understanding

**❌ Wrong**:
```python
# CLIP cannot understand spatial relationships
prompts = [
    "cat to the left of dog",
    "cat to the right of dog"
]
# Will give nearly identical scores
```

**Why wrong**: CLIP embeddings lose spatial topology. "Left" and "right" are treated as bag-of-words.

**✓ Right**:
```python
# Use a spatial reasoning model
# Examples: GQA models, Visual Genome models, SWIG
from swig_model import SpatialRelationModel

model = SpatialRelationModel()
result = model.predict_relation(image, "cat", "dog")
# Returns: "left", "right", "above", "below", etc.
```

**How to detect**: If query contains directional words (left, right, above, under, next to) → Use spatial model

---

### Anti-Pattern 4: Attribute Binding

**❌ Wrong**:
```python
prompts = [
    "red car and blue truck",
    "blue car and red truck"
]
# CLIP often gives similar scores for both
```

**Why wrong**: CLIP cannot bind attributes to objects. It sees "red, blue, car, truck" as a bag of concepts.

**✓ Right - Use PC-CLIP or DCSMs**:
```python
# PC-CLIP: Fine-tuned for pairwise comparisons
from pc_clip import PCCLIPModel

model = PCCLIPModel.from_pretrained("pc-clip-vit-l")
# Or use DCSMs (Dense Cosine Similarity Maps)
```

**How to detect**: If query has multiple objects with different attributes → Use compositional model

---

## Evolution Timeline

### 2021: CLIP Released
- Revolutionary: zero-shot, 400M image-text pairs
- Widely adopted for everything
- Limitations not yet understood

### 2022-2023: Limitations Discovered
- Cannot count objects
- Poor at fine-grained classification
- Fails spatial reasoning
- Can't bind attributes

### 2024: Alternatives Emerge
- **DCSMs**: Preserve patch/token topology
- **PC-CLIP**: Trained on pairwise comparisons
- **SpLiCE**: Sparse interpretable embeddings

### 2025: Current Best Practices
- Use CLIP for what it's good at
- Task-specific models for limitations
- Compositional models for complex queries

**LLM Mistake**: LLMs trained on 2021-2023 data will suggest CLIP for everything because limitations weren't widely known. This skill corrects that.

---

## Validation Script

Before using CLIP, check if it's appropriate:

```bash
python scripts/validate_clip_usage.py \
    --query "your query here" \
    --check-all
```

Returns:
- ✅ CLIP is appropriate
- ❌ Use alternative (with suggestion)

## Task-Specific Guidance

### Image Search (CLIP ✓)
```python
# Good use of CLIP
queries = ["beach", "mountain", "city skyline"]
# Works well for broad semantic concepts
```

### Zero-Shot Classification (CLIP ✓)
```python
# Good: Broad categories
categories = ["indoor", "outdoor", "nature", "urban"]
# CLIP excels at this
```

### Object Counting (CLIP ✗)
```python
# Use object detection instead
from transformers import DetrImageProcessor, DetrForObjectDetection
# See /references/object_detection.md
```

### Fine-Grained Classification (CLIP ✗)
```python
# Use specialized models
# See /references/fine_grained_models.md
```

### Spatial Reasoning (CLIP ✗)
```python
# Use spatial relation models
# See /references/spatial_models.md
```

---

## Troubleshooting

### Issue: CLIP gives unexpected results

**Check**:
1. Is this a counting task? → Use object detection
2. Fine-grained classification? → Use specialized model
3. Spatial query? → Use spatial model
4. Multiple objects with attributes? → Use compositional model

**Validation**:
```bash
python scripts/diagnose_clip_issue.py --image path/to/image --query "your query"
```

### Issue: Low similarity scores

**Possible causes**:
1. Query too specific (CLIP works better with broad concepts)
2. Fine-grained task (not CLIP's strength)
3. Need to adjust threshold

**Solution**: Try broader query or use alternative model

---

## Model Selection Guide

| Model | Best For | Avoid For |
|-------|----------|-----------|
| CLIP ViT-L/14 | Semantic search, broad categories | Counting, fine-grained, spatial |
| DETR | Object detection, counting | Semantic similarity |
| DINOv2 | Fine-grained features | Text-image matching |
| PC-CLIP | Attribute binding, comparisons | General embedding |
| DCSMs | Compositional reasoning | Simple similarity |

## Performance Notes

**CLIP models**:
- ViT-B/32: Fast, lower quality
- ViT-L/14: Balanced (recommended)
- ViT-g-14: Highest quality, slower

**Inference time** (single image, CPU):
- ViT-B/32: ~100ms
- ViT-L/14: ~300ms
- ViT-g-14: ~1000ms

## Further Reading

- `/references/clip_limitations.md` - Detailed analysis of CLIP's failures
- `/references/alternatives.md` - When to use what model
- `/references/compositional_reasoning.md` - DCSMs and PC-CLIP deep dive
- `/scripts/validate_clip_usage.py` - Pre-flight validation tool
- `/scripts/diagnose_clip_issue.py` - Debug unexpected results

---

*See CHANGELOG.md for version history.*

© curiositech · MIT · curiositech/some_claude_skills

Similar skills

All skills for AI and ML →

From the same set180

The whole set →
postgresql-optimization postgresql-optimization helps identify PostgreSQL bottlenecks: interprets EXPLAIN ANALYZE plans, selects indexing strategies, and tunes parameters like shared_buffers and work_mem. CX Codex +1 190installs wedding-immortalist The wedding-immortalist skill outlines a workflow for turning wedding photos and video into 3D scenes, a guest roster, and selected highlights. Use it to plan an interactive wedding archive. CX Codex +1 190installs 2000s-visualization-expert 2000s-visualization-expert — a skill for rebuilding 2000s music visualizers — Milkdrop, AVS, Geiss — with WebGL and Web Audio. Use it when a player needs an audio-reactive backdrop or generative art driven by music. CX Codex +1 190installs cloudflare-worker-dev The cloudflare-worker-dev skill guides Cloudflare Workers work for edge APIs, KV caching, request limits, and real-time coordination. CX Codex +1 189installs code-necromancer The code-necromancer skill maps an inherited or dormant codebase, helps get it running, and lays out a modernization plan. It starts by examining the system before changing it. CX Codex +1 187installs vercel-deployment vercel-deployment — a skill that handles Vercel deploys for Next.js and React apps: env vars, edge functions, build fixes, preview deployments. Not for AWS, GCP or Azure. CX Codex +1 187installs speech-pathology-ai speech-pathology-ai guides coding assistants through speech therapy tools, acoustic phoneme inspection, and AAC systems. Helps build clinical practice software. CX Codex +1 186installs code-review-checklist code-review-checklist is a Claude Code skill that builds a review checklist from a PR diff, using the project’s language and conventions. It highlights what reviewers should examine. CX Codex +1 185installs

Project sponsors

Companies that keep this analytics open