Skip to main content
skill CX Codex

site-audit — Claude Code skill: what it does, how to install

The site-audit skill for Claude Code performs a technical SEO check of a website. It identifies indexation issues, crawler accessibility problems, and broken links.

MIT ★ 778 updated 2026-11-16

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add boringmarketer/boring-marketing-skills
/plugin install boring-marketing@boring-marketing-skills
Open on GitHub

What site-audit does

This tool crawls websites like Google and AI bots to detect noindex errors, canonical mistakes, and blocked content in robots.txt or CDNs. It generates a ranked list of fixes based on potential business impact and checks Core Web Vitals via the PageSpeed API. The report focuses on actionable insights to resolve technical barriers preventing search visibility.

The skill incorporates project context from local marketing files to prioritize landing and service pages. It offers full audit, quick scan, and re-audit modes to verify fixes. Optional integration with PageSpeed Insights, Crux, or Firecrawl APIs is supported to provide deeper performance metrics and scrape content rendered via JavaScript.

What it helps with

  • find why pages are missing from search results
  • check if AI crawlers can access the site
  • detect technical SEO errors
  • analyze performance of key templates
  • compare site health after applying fixes

Example prompts for the agent

  • /plugin:site-audit https://example.com
  • site-audit https://example.com quick
  • site-audit https://example.com re-audit

How to install site-audit

1Plugin marketplace

/plugin marketplace add boringmarketer/boring-marketing-skills
/plugin install boring-marketing@boring-marketing-skills

This installs the whole boring-marketing plugin with all its skills

2Skills CLI (Vercel)

npx skills add boringmarketer/boring-marketing-skills --skill site-audit -a claude-code

3Manually: put the files in place

~/.claude/skills/site-audit/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/site-audit/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about site-audit

How to install site-audit in Claude Code?

Run /plugin marketplace add boringmarketer/boring-marketing-skills followed by /plugin install boring-marketing@boring-marketing-skills.

What does site-audit do?

It crawls your site to find technical SEO issues and AI crawler blocks, providing a prioritized list of fixes based on their impact.

What is needed for site-audit to work?

The skill works standalone, but you can optionally provide PAGESPEED_API_KEY, CRUX_API_KEY, or FIRECRAWL_API_KEY for enhanced analysis.

Source text

SKILL.md · MIT
---
name: site-audit
description: Crawls a website the way Google and the AI crawlers see it and returns a short, ranked fix list in plain words. It covers pages blocked from Google, noindex and canonical mistakes, AI crawlers blocked in robots.txt or at the CDN, sitemap health, broken links, duplicate and thin pages, Core Web Vitals from Google's free PageSpeed API, and the content signals that make a page worth citing. Use for a technical SEO audit, when pages aren't showing up on Google, to check whether AI crawlers can reach the site, or to re-check a site after fixes and see what changed.
when_to_use: technical seo audit, site audit, crawl my site, check my robots.txt, is google indexing my pages, why aren't my pages indexed, are AI crawlers blocked, is GPTBot blocked, broken links on my site, core web vitals, page speed check, check my sitemap, noindex check, duplicate pages on my site, technical seo health check, re-audit my site, what changed since the last audit
argument-hint: "[site URL] [quick | re-audit]"
metadata:
  boring:
    version: 4.1.1
    layer: execution
    reads: [brand-manifest, keyword-plan, stack]
    writes: [assets]
    optional-keys: [PAGESPEED_API_KEY, CRUX_API_KEY, FIRECRAWL_API_KEY]
    enhances_with:
      - toolset: web-scrape
        gains: "re-reads pages that build their text with JavaScript, so content checks run on what visitors see instead of an empty shell"
      - toolset: search-console
        gains: "confirms which pages Google actually indexed and why it skipped the rest"
---

# Site Audit

Most sites that don't get found have a short list of reasons, and a 75-row audit table
hides them. This skill crawls the site politely, the way Google and the AI crawlers meet
it, and hands back the five to ten fixes that matter most for getting found, each backed
by what the crawl saw. Done means an owner or their developer can act on the list today,
and anyone can re-check every claim on it. The checks and the ranking below are a floor.
When the business in front of you argues for a different order, change it and say why in
one line.

## Context

Only if `./brand/`, `./brands/` or `MARKETING.md` exists (nothing to load otherwise), run:
`python3 ${CLAUDE_SKILL_DIR}/../start-here/scripts/brand_context.py --for site-audit`
(`${CLAUDE_SKILL_DIR}` is this skill's folder; outside Claude Code use the folder this
file was read from, e.g. `.agents/skills/site-audit`. If the script or python3 is
unavailable, read the brand folder directly: `./brands/<slug>/` when `./brands/active`
names one, else `./brand/`; then `MARKETING.md`, `preferences.md` and the files in
`reads:`. If none exist, proceed standalone and ask only for the URL.)

The business facts in `brand.yaml` (services, towns, category) and the keyword plan tell
you which pages make money. Use them to judge severity. Without them, infer money pages
from the menu and the URL paths (services, locations, products, pricing, booking, quote).

Auditing a prospect's site rather than the user's own? Don't load or write brand files
for it. The report lands in `./campaigns/` and nothing else changes.

No URL, or only a placeholder (example.com, yourdomain.com)? Ask for the URL in one line.
Never crawl a placeholder.

## What the evidence says

Load `references/crawler-facts.md` before describing any crawler, CDN or PageSpeed
behavior (check its header; refresh it if stale). The judgment that doesn't expire:

- **Being crawlable and indexable comes first.** One Disallow or noindex on a money page
  outweighs every other item on the list. Most real audits turn up two or three of these,
  and a pile of cosmetic warnings.
- **Classic organic rank is upstream of AI citation.** On Google's AI answers most cited
  pages already rank, and other engines lean on their own search indexes. So AI readiness
  mostly means letting the AI search crawlers in, putting the text in the HTML, and
  getting the same fundamentals right. Never promise that a technical fix produces
  citations.
- **Optional hygiene stays optional.** llms.txt and generic structured data are cheap and
  unproven. They go in "also worth doing", never at the top. Schema earns its place only
  where the page content supports it, and a business markup block that names the business
  consistently is the one piece worth adding everywhere (`references/schema-and-entity.md`).
- **Content signals with evidence behind them** are reported for articles: statistics,
  quotations and cited sources (the one controlled study of generative answers found these
  lift visibility), visible publish and updated dates, a named author, and answer capsules
  (a question heading followed by a short, self-contained, link-light answer). Signals,
  not rules. A missing date on a blog post is never a top-five fix.
- **Severity is business impact**, not how many pages a warning touches. 300 images
  without alt text matter less than one service page marked noindex.

## Modes

| The user says | Mode | What runs |
|---|---|---|
| a URL, "audit my site's SEO", "technical SEO audit" | **Full** (default) | Up to 200 pages (`--max-pages`, hard cap 1,000), the AI crawler checks, PageSpeed on the homepage and three key templates |
| "quick", "is anything blocking Google", "are AI crawlers blocked" | **Quick** | Homepage plus about 20 pages, robots and AI crawler access, PageSpeed on the homepage |
| "re-audit", "what changed", "check again after the fixes" | **Re-audit** | Full or quick, compared with the last findings file |

A narrow question ("is GPTBot blocked?") gets Quick mode and a few lines of answer, with
the findings still saved. `start-here` owns the marketing review ("review my site",
"audit my site"); this skill is the technical half.

## Phase 1: Set up the run

Settle three things before crawling, from the brand files or the request. Ask only what
you can't infer, in one message.

1. **The site and the money pages.** The homepage URL, and the pages that bring customers
   (service, location, product, pricing, booking). These set severity later.
2. **Does the business want AI visibility?** Default yes for any business that wants
   customers. If robots.txt blocks AI search crawlers on purpose (a publisher protecting
   its content, say), ask once before calling it a problem. Blocking AI training crawlers
   is a business choice. Report it once, never as a fix.
3. **Time and politeness.** A full crawl of 200 pages takes about 2 to 6 minutes, plus 1
   to 3 minutes of PageSpeed calls. Quick takes about a minute. The crawler names itself
   in its user agent, honors robots.txt for its own crawling, keeps about 4 requests in
   flight with a gap between request starts, and slows to one request at a time when
   robots.txt sets a crawl delay. The CDN check sends one homepage request per AI crawler
   name, which is a light load even for a prospect's site.

**PageSpeed key.** Google's PageSpeed Insights API works without a key, but the keyless
quota is shared and often used up; the audit then says Core Web Vitals weren't checked.
A free key fixes it: in the Google Cloud console, enable the PageSpeed Insights API,
create an API key, and set `PAGESPEED_API_KEY`. Enabling the Chrome UX Report API on the
same project lets that key fetch real-visitor data too (or set `CRUX_API_KEY`).

## Phase 2: Run the crawl

```bash
python3 ${CLAUDE_SKILL_DIR}/scripts/crawl_audit.py <url>                  # full
python3 ${CLAUDE_SKILL_DIR}/scripts/crawl_audit.py <url> --mode quick     # quick
python3 ${CLAUDE_SKILL_DIR}/scripts/crawl_audit.py <url> --compare auto   # re-audit against the last run for this site
```

The script (Python 3 standard library, no installs) reads robots.txt and every sitemap it
lists, fetches the homepage, crawls same-site links breadth first with a sample of
sitemap URLs mixed in, checks the internal link targets it didn't crawl, compares how the
homepage answers a browser and each AI crawler, probes a made-up URL for soft 404s, looks
for llms.txt and ai.txt, and runs PageSpeed. It writes `findings.json` and `pages.jsonl`
to `./campaigns/site-audit-<domain>-<YYYY-MM-DD>/` (a second run that day gets `-2`, so
earlier runs are never overwritten) and prints a summary. Useful flags: `--max-pages N`,
`--no-psi`, `--no-bot-check`, `--psi-templates N`, `--max-minutes N`. Run `--help` for
the rest.

No python3? Do a sample audit by hand: fetch robots.txt, the sitemap, the homepage and 5
to 10 money pages (WebFetch or curl), run the same checks on what you read, and say in
the Sources and assumptions note that it was a sample of N pages.

**Sites that build their text with JavaScript.** When most pages send almost no text in
their HTML, the crawl reports `js_rendered_content` and leaves those pages out of the
content checks rather than calling them thin. Say so plainly in the report. Google
renders JavaScript, later than HTML; AI crawlers may read only the HTML they're sent
(the AI search facts in `/ai-search` list server-rendered text as a prerequisite for
ChatGPT). To audit what visitors actually see, re-run with rendered HTML. We use
Firecrawl for this (free tier, see TOOLS.md): with `FIRECRAWL_API_KEY` set, add
`--render firecrawl` (up to 25 pages, each counted against your Firecrawl credits). With
the Firecrawl MCP instead, scrape each page with
the `rawHtml` format, save the results as JSON lines (`{"url": ..., "html": ...}`), and
pass `--rendered <file>`. Without either, list the skipped checks and stop there.

## Phase 3: Read the findings

`findings.json` holds `findings` (each with `id`, `severity`, `count`,
`key_pages_affected`, `examples`, `urls`, `observation`, `fix`, `effort`, `confidence`),
`checked_and_fine`, `not_checked`, a `site` block (robots, AI crawler policy per bot, the
CDN comparison, sitemaps, AI files, host variants, soft 404, content-signal summary) and
`cwv`. Load `references/checks.md` for what each finding means, when to raise or lower
it, its edge cases, and the fix by platform.

Before anything goes in the top list:

- **Look at one example yourself.** Fetch the example URL and find the line that proves
  it (the meta tag, the Disallow rule, the 404). The script's heuristics have edge cases,
  and an owner who catches one wrong item stops trusting the rest. Note what you verified
  in `notes.md`.
- **Keep "couldn't check" apart from "fine".** `not_checked` items go in the Sources and
  assumptions note as unknowns. Only `checked_and_fine` items may be called fine.
- **Respect the scope.** When `crawl.truncated` is true, counts mean "at least N among
  the pages crawled", and orphan pages are weak evidence.
- **The CDN check uses stand-in requests.** It sends the crawler's name from this machine,
  not from the vendor's IP ranges. A refusal is strong evidence that the CDN or firewall
  blocks that name, and the fix gets confirmed in the CDN's bot settings or server logs.
  Say exactly that.
- **Field beats lab.** PageSpeed returns real-visitor data (Chrome UX Report, 28 days)
  only for pages with enough traffic. When only lab data exists, say "a simulated phone
  load", never "your visitors wait".

## Phase 4: Rank the fixes

Rank by how much each fix helps the business get found, then by effort. The default
order, which you may change with a stated reason:

1. **Anything that keeps Google out of the site or a money page.** The homepage down or
   failing HTTPS, robots.txt erroring or disallowing `/`, noindex on money pages (every
   page noindex is usually one setting, such as WordPress's "Discourage search engines"
   box), money pages canonicalized to another page or another domain.
2. **AI crawlers blocked when the business wants AI visibility.** The search and
   assistant crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User,
   Claude-User, Perplexity-User), in robots.txt or at the CDN.
3. **Broken paths to money pages.** Broken links in the menu or from the homepage,
   redirect loops, server errors, a sitemap that lists noindex, broken or blocked URLs.
4. **Content search engines can't see or tell apart.** Text built by JavaScript,
   duplicate and near-copy pages (town pages with the city swapped), thin money pages.
5. **Real visitors waiting on money pages.** Poor field Core Web Vitals. Lab-only
   slowness sits lower.
6. **Basics on money pages.** Missing titles, no H1, no mobile viewport.
7. **Everything else** (description length, alt text, share tags, HSTS, URL style,
   llms.txt, generic schema, content signals on old posts) goes to "also worth doing" or
   the one-line fine list.

Judgment for the close calls:

- **Weight by page value.** One service page outranks fifty tag archives. A ten-minute
  fix that unblocks indexing goes ahead of a redesign.
- **Merge findings with one cause.** A page that is noindex and also in the sitemap is
  one item. Twenty broken links from one menu entry are one fix.
- **Pick five to ten.** A healthy site may have three; say so and stop. Padding the list
  hides the items that matter.
- **When rank is the real problem**, technical fixes won't produce AI citations or
  traffic. Say so and route to `/keyword-research` or `/local-seo-plan`.

Hard rules: never call something broken without the observation behind it; never present
"couldn't check" as fine; never invent traffic, revenue or ranking impact; never promise
citations from AI-readiness fixes.

## Phase 5: Write the report

Write for the owner, in plain words. No script paths, tool names used as process, check
ids or severity labels in the report; those stay in `notes.md`. Length: the plan budget
(about 1,200 to 2,500 words) is a ceiling, and most audits need 600 to 1,200. **Fix
first** carries item 1 in full; the numbered list holds the rest in order, so nothing is
said twice. Each item names what's wrong, why it matters for this business, the affected
pages (count plus two or three examples), the fix, and rough effort.

A sample, for the sample business Hartline Plumbing, Dallas TX (illustrative, not a real
site or real numbers):

```markdown
## SITE AUDIT: hartlineplumbing.com (sample)
Nov 16, 2026 · 184 pages crawled, for getting found on Google and in AI answers

**Fix first:** the water heater pages tell Google not to index them. All 6 pages under
/water-heaters/ carry `noindex` (for example /water-heaters/tankless-installation/), and 4
of them are in the sitemap. They are the services you advertise most, so they can't rank
or be cited until this changes. Fix: untick "hide from search engines" on those pages in
the SEO plugin, then ask Google to recrawl them in Search Console. About 15 minutes.

**Then, in this order:**

1. **ChatGPT's and Perplexity's search crawlers are blocked.** robots.txt has
   `User-agent: OAI-SearchBot` / `Disallow: /` and the same for PerplexityBot, so the site
   can't be cited in their answers. Pages: the whole site. Fix: delete those two groups;
   keep GPTBot blocked if you don't want AI training. 5 minutes.
2. **The "Book online" link in the menu is broken.** It points at /book-now/, which
   returns 404, on all 184 pages. Fix: point the menu at /schedule/. 5 minutes.
3. **Twelve town pages are near copies.** /plumber-plano/ and /plumber-allen/ differ only
   in the town name (12 pages, e.g. /plumber-frisco/). Fix: add jobs done there, local
   details and photos, or merge them into one service-area page. An hour per page.
4. **Phones wait on the homepage.** Real visitors on phones see the main image after
   about 4.6 seconds (Google's good mark is 2.5). Fix: compress and resize the hero photo
   and stop loading the chat widget before the page. Half a day for a developer.

**Also worth doing:** add a self-referencing canonical to the 40 pages without one; add
alt text to the gallery photos; add business markup (name, phone, address) to the
homepage.

**Checked and fine:** HTTPS and redirects, robots.txt for Google, sitemap format, no
server errors, titles present and unique, mobile viewport on every page.

**Sources and assumptions:**
- Crawled 184 of ~210 URLs on Nov 16, 2026 (the crawl stops at 200 pages), plus 150
  internal link checks; PageSpeed real-visitor data for the homepage, lab data for two
  service pages.
- The CDN check sends the crawlers' names from this machine, not from their own networks;
  it found no blocks.
- Not checked: which pages Google has indexed (needs Search Console access).
```

For a re-audit, open with **What changed since <date>** (resolved, new, still open,
from `changes` in the findings file), then the current fix list. Each run crawls within
its own page limit, so a page seen in only one run may just have been outside the other
run's sample. Say so instead of calling it new or gone.

## Output

Write `site-audit.md` (the report) into the run's folder,
`./campaigns/site-audit-<domain>-<YYYY-MM-DD>/`, beside the script's `findings.json` and
`pages.jsonl`, and `notes.md` for the
working record: the examples you verified by hand, findings merged or left out and why,
raw Core Web Vitals, and anything the user should forward to a developer. Append one line
to `./brand/assets.md` when a brand folder exists (never for a prospect). Then the chat
summary: header, the fix list, **Files saved**, **What's next**.

## When the user edits your work

Apply the edit, then make it stick: save a gold case to `./brand/gold/<this skill>/` and
one to three yes/no checks to `./brand/checks.md`, and confirm in one line. Before handing
over any work, answer the checks the context block loaded, silently, and fix every no.
How: `${CLAUDE_SKILL_DIR}/../start-here/references/taste-loop.md`.

## What's next

- `/ai-search`: now that crawlers can reach the site, check whether ChatGPT, Perplexity
  and Google's AI actually name and cite the business (~20 min)
- `/keyword-research`: when the pages are sound but nothing ranks, find what the site
  should be ranking for (~15 min)
- `/local-seo-plan`: for a local business, decide which service and town pages to build,
  merge or drop (~30 min)
- `/first-screen-check`: see the money pages the way a customer does on a phone (~10 min)

© James Dickerson (@boringmarketer) · MIT · boringmarketer/boring-marketing-skills

Similar skills

All skills for SEO and GEO →
seo-site-health-audit rampstackco The seo-site-health-audit skill reranks crawler results (Ahrefs, similar tools) by actual impact on rankings instead of issue count. It helps you decide which technical fixes to tackle first when dev time is limited. CX Codex +1 291installs seo-audit sickn33 seo-audit audits your site for indexation problems, performance issues, mobile readiness, and content gaps. Identifies what blocks organic visibility and helps prioritize fixes. CX Codex +1 1,000installs website-audit-tools Legion Code Inc. website-audit-tools — a plugin for comprehensive public-site audits: crawl, SEO, accessibility, conversion, security. Outputs a scorecard with findings and reports. CX Codex ★ 96repo stars SEO Web Analysis MCP — Site Audit & Tech Stack nexgendata-apify SEO Web Analysis MCP — Site Audit & Tech Stack is an MCP server for Claude Code that reveals a site's tech stack, DNS records, SSL status, and WHOIS data. It supports competitive research and security audits. CX Codex +1 9,357installs multipliers.ai Site Auditor multipliers The multipliers.ai Site Auditor MCP server connects to Claude Code and checks whether a website is ready to be cited by LLM-powered search engines, returning a score and findings in about 20 seconds. CX Codex +1 109installs technical-seo-stinger Legion Code Inc. technical-seo-stinger is a Claude Code skill that audits technical SEO across 100 pages, checking crawlability, indexability, sitemaps, robots.txt, canonicalization, and crawl-budget use to assess an external site's technical health. CX Codex +1 ★ 96repo stars

From the same set35

The whole set →
reviews reviews is a Claude Code skill that audits Google reviews against the local map pack top 3. It spots stalls in recent feedback, drafts owner responses and sets post-job ask scripts. CX Codex +1 ★ 778repo stars find-local-clients find-local-clients is a Claude Code skill that finds local businesses to pitch by trade and town. It activates for requests to build a prospect list or identify businesses that need marketing help. CX Codex +1 ★ 778repo stars first-screen-check first-screen-check is a Claude Code skill that audits mobile landing screens. It spots missing call buttons, review badges, and unkept ad promises. CX Codex +1 ★ 778repo stars creative creative is a Claude Code skill that makes images, videos, and ad assets for a brand. It joins requests to create or edit visuals and prepare campaign materials. CX Codex +1 ★ 778repo stars audience-research audience-research is a Claude Code skill that finds where people discuss a product and captures their wording. Run it before positioning or copy work, or when entering a new market. CX Codex +1 ★ 778repo stars programmatic-seo programmatic-seo is a Claude Code skill that checks and builds SEO page libraries from a template and dataset. It activates for keyword patterns repeated across modifiers. CX Codex +1 ★ 778repo stars content-atomizer content-atomizer is a Claude Code skill that turns finished articles, newsletters, podcasts, or videos into social posts. It can also draft comment replies and build a posting calendar. CX Codex +1 ★ 778repo stars lead-magnet The lead-magnet skill for Claude Code analyzes competitors and builds interactive lead-capture tools. It runs when you need to generate freebie concepts to grow your audience. CX Codex +1 ★ 778repo stars

Project sponsors

Companies that keep this analytics open