Skip to main content
skill CX Codex

agent-traffic — Claude Code skill for traffic classification

This Claude Code skill labels HTTP traffic as agents, crawlers, other bots, or people. It comes into play when adding detection to request handlers, analytics, logs, or SQL queries.

MIT ★ 88 updated 2026-11-10

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add mintlify/agent-traffic --skill agent-traffic -a claude-code
Open on GitHub

What agent-traffic does

The skill guides HTTP request classification in handlers, middleware, and log analysis. It uses the agent-traffic dataset to distinguish agents, AI crawlers, search crawlers, bots, and people. A user agent is required for each result; the Accept header and the client's real IP provide further signals.

For JavaScript or TypeScript request handling, it uses the @mintlify/agent-traffic package and its classify function. For another language, the guidance is to port src/classify.ts while preserving the check order. For SQL, one option classifies each distinct user agent and joins the results to the logs table. Another generates a CASE expression from the dataset's patterns. Long-running processes should refresh IP ranges about daily.

The IP header must come from the user's own proxy: the client controls the leftmost X-Forwarded-For value, so it is not a trustworthy source. If no reliable IP is available, omit ip; identities then remain self-declared. Reports should keep agent and ai-crawler as separate categories.

What it helps with

  • label HTTP requests by agent or bot type
  • measure agent traffic in stored logs
  • join user-agent classifications to a SQL table
  • separate verified identities from self-declared ones

Example prompts for the agent

  • Add agent and bot detection to my HTTP request handler.
  • Analyze these logs and calculate the share of requests from agents.
  • Write a SQL approach to classify user agents in this log table.

How to install agent-traffic

1Skills CLI (Vercel)

npx skills add mintlify/agent-traffic --skill agent-traffic -a claude-code

2Manually: put the files in place

~/.claude/skills/agent-traffic/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/agent-traffic/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about agent-traffic

How do I install agent-traffic in Claude Code?

Run npx skills add mintlify/agent-traffic --skill agent-traffic -a claude-code. The skill file is available at ~/.claude/skills/agent-traffic/SKILL.md.

What does agent-traffic do?

It guides classification of HTTP requests as agents, crawlers, other bots, or people. The skill covers request-time code, ports to other languages, and SQL analysis of stored logs.

What data is needed to classify traffic?

Each result requires a user agent. The Accept header and the client's real IP add signals; identities remain self-declared when no trusted IP is available.

Source text

SKILL.md · MIT
---
name: agent-traffic
description: Classify HTTP traffic as agents, AI crawlers, search crawlers, bots or humans using the agent-traffic dataset. Use when adding agent or bot detection to request handlers, middleware, analytics, server logs or SQL queries, or when someone asks how much of their traffic comes from agents.
---

# Classify agent traffic

[agent-traffic](https://github.com/mintlify/agent-traffic) classifies HTTP requests as agents, AI crawlers, search crawlers, bots or humans. The dataset and rules are already written. Use them for whatever the user asks, whether that's classifying requests, analyzing logs or answering questions about their traffic, without changing what the rules mean.

## Sources

Read these before writing anything:

- [`GUIDELINES.md`](https://github.com/mintlify/agent-traffic/blob/main/GUIDELINES.md) has the categories, decision order and verification rules.
- [`data/agents.json`](https://github.com/mintlify/agent-traffic/blob/main/data/agents.json) lists known agents and crawlers: user agent patterns, `Accept` and `Signature-Agent` fingerprints, verification sources and real example user agents.
- [`data/http-clients.json`](https://github.com/mintlify/agent-traffic/blob/main/data/http-clients.json) lists generic HTTP libraries and headless browsers.

If the package is installed, the same data is at `node_modules/@mintlify/agent-traffic/dist/data/`. Every pattern is a case-insensitive regular expression.

## 1. Find out what's available

Before choosing an approach, look at the user's code or schema and confirm:

- **Where classification can run:** in request-handling code, in an ingest pipeline, or only over stored logs.
- **Which fields exist:** the user agent, the `Accept` header, other request headers, and the client IP. Also check whether that IP is the real client or a proxy or CDN address.
- **What they want out:** a single label, per-agent breakdowns, or verified-only numbers.

Every result needs at least the user agent. The `Accept` header and the real client IP make it much more accurate. If the user can add fields to their logs, recommend logging `Accept` and the real client IP.

## 2. Pick the approach

### JavaScript or TypeScript at request time

Install the package and call `classify` where requests are handled. Don't reimplement the rules.

```
npm i @mintlify/agent-traffic
```

```js
import { classify, loadIpRanges } from '@mintlify/agent-traffic'

let ipRanges

async function classifyRequest(request) {
  ipRanges ??= loadIpRanges()
  return classify({ headers: request.headers, ip: request.headers.get('cf-connecting-ip') }, { ipRanges: await ipRanges })
}
```

- **Use the real client IP.** Take it from the header the user's own proxy sets, such as `CF-Connecting-IP` on Cloudflare or the platform's equivalent. Never use the leftmost `X-Forwarded-For` value, which the client controls. If there's no trustworthy IP, omit `ip`.
- **Only pass `headersComplete: true` when browser headers arrive intact.** That means HTTPS, with no proxy stripping `sec-fetch-*`.
- **Refresh IP ranges about daily** in long-running processes.
- **Store the result** if it feeds analytics: `category`, `kind`, `id` and `verified`.

### Another language at request time

Port `classify` from [`src/classify.ts`](https://github.com/mintlify/agent-traffic/blob/main/src/classify.ts), reading `agents.json` and `http-clients.json` at startup. Keep the check order from GUIDELINES.md exactly:

1. agent patterns in file order;
2. then exact `Accept` fingerprints;
3. then `Signature-Agent` on signed requests only;
4. then markdown-first `Accept`;
5. then HTTP clients;
6. then browser checks;
7. then the fallbacks.

### SQL over stored logs

There are two options. Prefer the first.

**A. Lookup table.** Classify each distinct user agent once with the real package, load the results as a table, and join on it. The output is identical to request-time classification, and there are no regex dialect issues.

```js
import { classify } from '@mintlify/agent-traffic'
const rows = distinctUserAgents.map((ua) => ({ user_agent: ua, ...classify({ userAgent: ua }) }))
```

Write `user_agent, category, kind, id, name, operator` to a table and `JOIN ON user_agent`. Re-run it on a schedule so new user agents and dataset updates get picked up.

**B. Generated `CASE` expression.** Use this when a lookup table isn't practical. Build one `CASE` from the data, with a `WHEN` per pattern in this order: every `agents.json` entry in file order, then every `http-clients.json` entry. Map each match to that entry's `id`, `category` and `kind`. Generate it with a script rather than by hand, and regenerate it when the dataset changes.

Translate patterns for the engine:

| Engine | Match expression | Translation needed |
|---|---|---|
| ClickHouse | `match(ua, '(?i)<pattern>')` | none |
| DuckDB | `regexp_matches(ua, '<pattern>', 'i')` | replace named groups `(?<name>` with `(?:` |
| Postgres | `ua ~* '<pattern>'` | replace `(?<name>` with `(?:`, and `\b` with `\y` (in Postgres, `\b` means backspace) |
| BigQuery, Snowflake, others | check the engine's regex docs | test every pattern; Snowflake has no `\b` or lookarounds |

The first three rows were verified against every example in the dataset. For any other engine, run the test in step 3 before trusting the output.

Results from logs use only the user agent unless the logs also kept `Accept`. Agents that send a browser user agent and ask for markdown will show as `human`; say so in any report you produce. To apply the markdown rule in SQL, classify the request as an `agent` when the `Accept` column ranks `text/markdown` or `text/x-markdown` ahead of `text/html`.

## 3. Test it

Every entry in `agents.json` has real `examples`. Run all of them through the implementation you built. Each must come back with that entry's `id` and `category`. Also check these:

| User agent | Expected |
|---|---|
| `curl/8.7.1` | `bot` / `http-client` |
| `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36` | `human`, or no match in SQL |
| `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Code/1.109.3 Chrome/142.0.0.0 Electron/39.2.3 Safari/537.36` | not GitHub Copilot |
| `Claude-Code-Docs-Fetcher/3.0` | not Claude Code |

Don't report the work as done until these pass.

## 4. Report the numbers correctly

- **Keep `agent` and `ai-crawler` separate.** Crawlers are scheduled and usually far larger, and combining them hides real agent usage.
- **A markdown fetch by curl, axios or a script is a `bot`, not an agent,** unless it identifies itself or asks for markdown in `Accept`.
- **Don't call unverified identities verified.** With no real client IP, every identity is self-declared.
- **Re-label carefully when replacing an older classification.** If the user already shows an "AI traffic" number that a stricter definition would shrink, offer two numbers instead of shrinking the headline:
  - a broad one: agents, plus AI crawlers, plus automated reads of `.md` and `llms.txt`;
  - a narrow one: just `agent`.

  Explain the difference to the user.

## Don't

- Match bare product words like `Cursor`, `Copilot` or `Code`. Use the anchored patterns from the data.
- IP-check entries marked `"runsOn": "user"`. Those agents run on the user's machine.
- Treat a range-file mismatch as spoofing when the file is older than 180 days, or when the IP isn't a single valid address.
- Copy a pattern list into code by hand. Read it from the dataset so updates flow through.

© mintlify · MIT · mintlify/agent-traffic

Similar skills

All skills for Backend and APIs →

Project sponsors

Companies that keep this analytics open