Skip to main content
skill CX Codex

chatbot-analytics — Claude Code skill: what it does, how to install

chatbot-analytics adds conversation monitoring and token usage tracking for AI chats while respecting HIPAA constraints.

MIT 221 installs ★ 274 updated 2026-10-30

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add curiositech/some_claude_skills --skill chatbot-analytics -a claude-code
Open on GitHub

What chatbot-analytics does

The skill provides TypeScript implementation patterns for 13 conversation metrics. It records session counts, response latency, fallback rates, and token consumption. Aggregation queries also estimate expenses using Claude pricing ($3.00 per million input tokens and $15.00 per million output tokens).

Raw message text cannot be logged under HIPAA. The skill tracks token counts and operational flags instead. Categories come from metadata booleans rather than text inspection. It relies on file modification tools alongside npm and npx.

What it helps with

  • log conversation metrics without storing protected health information
  • monitor token usage and estimate API costs across 30 days
  • categorize chat topics from AI response flags instead of text parsing
  • calculate session completion rates and response latency for dashboards

Example prompts for the agent

  • Set up HIPAA-safe conversation analytics for the AI chat
  • Add token tracking and response time metrics to the message handler
  • Write dashboard aggregation queries for 30-day session stats

How to install chatbot-analytics

1Skills CLI (Vercel)

npx skills add curiositech/some_claude_skills --skill chatbot-analytics -a claude-code

2Manually: put the files in place

~/.claude/skills/chatbot-analytics/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/chatbot-analytics/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about chatbot-analytics

How to install chatbot-analytics in Claude Code?

Run npx skills add curiositech/some_claude_skills --skill chatbot-analytics -a claude-code in your terminal.

What does chatbot-analytics track?

It tracks session duration, token volume, error rates, and response latency. It collects these dimensions without logging raw message contents.

How does it handle sensitive health information under HIPAA?

User queries and assistant replies are never stored in the analytics schema. Topic categorization relies solely on boolean flags returned by the model.

Source text

SKILL.md · MIT
---
name: chatbot-analytics
description: Implement AI chatbot analytics and conversation monitoring. Use when adding conversation metrics, tracking AI usage, measuring user engagement with chat, or building conversation dashboards.
  Activates for AI analytics, token tracking, conversation categorization, and chat performance.
allowed-tools: Read,Write,Edit,Bash(npm:*,npx:*)
metadata:
  category: Data & Analytics
  tags:
  - analytics
  - chatbot
  - ai-metrics
  pairs-with:
  - skill: llm-streaming-response-handler
    reason: Streaming response metrics (TTFT, tokens/sec) are key chatbot analytics dimensions
  - skill: data-pipeline-engineer
    reason: Conversation data pipelines feed analytics dashboards and reporting systems
  - skill: prompt-engineer
    reason: Analytics on prompt effectiveness drive iterative prompt optimization
---

# AI Chatbot Analytics

This skill helps you implement analytics for the AI coaching chat feature while maintaining HIPAA compliance.

## Core Metrics to Track

Based on [industry best practices](https://hiverhq.com/blog/chatbot-analytics), track these 13 key metrics:

| Metric | Description | HIPAA Safe? |
|--------|-------------|-------------|
| Total Sessions | Number of chat sessions | Yes |
| Avg Messages/Session | Messages per conversation | Yes |
| Avg Session Duration | Time spent in chat | Yes |
| Engagement Rate | % users who use chat | Yes |
| Completion Rate | Sessions ended naturally | Yes |
| Abandonment Rate | Sessions ended early | Yes |
| Response Time | AI response latency | Yes |
| Token Usage | Total/avg tokens consumed | Yes |
| Error Rate | Failed responses | Yes |
| Fallback Rate | "I don't understand" responses | Yes |
| Topic Categories | What users discuss | Metadata only |
| Sentiment Trend | Emotional direction | Derived only |
| Crisis Triggers | Emergency detection | Metadata only |

## HIPAA-Compliant Analytics

### What to Track

```typescript
// Conversation metadata (SAFE)
interface ConversationAnalytics {
  id: string;
  conversationId: string;
  userId: string;  // For aggregation, not individual tracking
  startedAt: Date;
  endedAt: Date | null;
  messageCount: number;
  userMessageCount: number;
  aiMessageCount: number;
  totalTokens: number;
  inputTokens: number;
  outputTokens: number;
  category: string;  // Derived from metadata flags
  outcome: 'completed' | 'abandoned' | 'error' | 'crisis_escalated';
  avgResponseTime: number;
  hadFallback: boolean;
}
```

### What NOT to Track

```typescript
// NEVER store these in analytics
interface PROHIBITED {
  messageContent: string;      // PHI
  userQuery: string;           // PHI
  aiResponse: string;          // PHI
  specificTopics: string[];    // Could reveal health info
  exactSentiment: 'sad';       // Could reveal mental state
}
```

## Implementation Pattern

### Tracking Conversation Start

```typescript
// src/lib/ai/analytics.ts
export async function trackConversationStart(
  conversationId: string,
  userId: string
): Promise<void> {
  await db.insert(conversationAnalytics).values({
    id: generateId(),
    conversationId,
    userId,
    startedAt: new Date(),
    messageCount: 0,
    totalTokens: 0,
    category: 'unknown',
    outcome: 'in_progress'
  });
}
```

### Tracking Message Exchange

```typescript
export async function trackMessageExchange(
  conversationId: string,
  tokens: { input: number; output: number },
  responseTimeMs: number,
  flags: { hadFallback: boolean; hasCrisisIndicator: boolean }
): Promise<void> {
  await db
    .update(conversationAnalytics)
    .set({
      messageCount: sql`message_count + 1`,
      totalTokens: sql`total_tokens + ${tokens.input + tokens.output}`,
      inputTokens: sql`input_tokens + ${tokens.input}`,
      outputTokens: sql`output_tokens + ${tokens.output}`,
      avgResponseTime: sql`(avg_response_time * (message_count - 1) + ${responseTimeMs}) / message_count`,
      hadFallback: flags.hadFallback,
      ...(flags.hasCrisisIndicator && { outcome: 'crisis_escalated' })
    })
    .where(eq(conversationAnalytics.conversationId, conversationId));
}
```

### Tracking Conversation End

```typescript
export async function trackConversationEnd(
  conversationId: string,
  outcome: 'completed' | 'abandoned' | 'error'
): Promise<void> {
  await db
    .update(conversationAnalytics)
    .set({
      endedAt: new Date(),
      outcome
    })
    .where(eq(conversationAnalytics.conversationId, conversationId));
}
```

## Category Detection (Metadata-Based)

Detect conversation categories WITHOUT reading content:

```typescript
// Categories based on metadata flags from AI response
interface AIResponseMetadata {
  usedCopingStrategies: boolean;
  usedCrisisProtocol: boolean;
  usedCheckInSupport: boolean;
  usedGeneralChat: boolean;
  requestedClarification: boolean;
}

function deriveCategory(metadata: AIResponseMetadata): string {
  if (metadata.usedCrisisProtocol) return 'crisis_support';
  if (metadata.usedCopingStrategies) return 'coping_strategies';
  if (metadata.usedCheckInSupport) return 'checkin_support';
  if (metadata.requestedClarification) return 'clarification';
  return 'general_chat';
}
```

## Dashboard Aggregations

### Session Metrics

```typescript
// Get aggregated session stats (HIPAA safe - no individual data)
async function getSessionStats(days: number = 30) {
  const since = subDays(new Date(), days);

  return db
    .select({
      totalSessions: count(),
      avgMessages: avg(conversationAnalytics.messageCount),
      avgDuration: avg(
        sql`JULIANDAY(ended_at) - JULIANDAY(started_at)) * 24 * 60`
      ),
      completionRate: sql`
        CAST(SUM(CASE WHEN outcome = 'completed' THEN 1 ELSE 0 END) AS FLOAT) /
        CAST(COUNT(*) AS FLOAT)
      `,
      crisisEscalations: sql`
        SUM(CASE WHEN outcome = 'crisis_escalated' THEN 1 ELSE 0 END)
      `
    })
    .from(conversationAnalytics)
    .where(gte(conversationAnalytics.startedAt, since));
}
```

### Token Usage for Cost Tracking

```typescript
async function getTokenUsage(days: number = 30) {
  const since = subDays(new Date(), days);

  const result = await db
    .select({
      totalTokens: sum(conversationAnalytics.totalTokens),
      inputTokens: sum(conversationAnalytics.inputTokens),
      outputTokens: sum(conversationAnalytics.outputTokens),
      avgTokensPerSession: avg(conversationAnalytics.totalTokens)
    })
    .from(conversationAnalytics)
    .where(gte(conversationAnalytics.startedAt, since));

  // Estimate cost (Claude pricing)
  const inputCost = (result.inputTokens / 1_000_000) * 3.00;  // $3/M input
  const outputCost = (result.outputTokens / 1_000_000) * 15.00; // $15/M output

  return {
    ...result,
    estimatedCost: inputCost + outputCost
  };
}
```

### Category Breakdown

```typescript
async function getCategoryBreakdown(days: number = 30) {
  const since = subDays(new Date(), days);

  return db
    .select({
      category: conversationAnalytics.category,
      count: count(),
      percentage: sql`
        CAST(COUNT(*) AS FLOAT) * 100.0 /
        (SELECT COUNT(*) FROM conversation_analytics WHERE started_at >= ${since})
      `
    })
    .from(conversationAnalytics)
    .where(gte(conversationAnalytics.startedAt, since))
    .groupBy(conversationAnalytics.category)
    .orderBy(desc(count()));
}
```

## Alert Configuration

Set up alerts for concerning patterns:

```typescript
interface AnalyticsAlert {
  type: 'crisis_spike' | 'error_spike' | 'abandonment_spike';
  threshold: number;
  windowHours: number;
  action: 'log' | 'email' | 'slack';
}

const alerts: AnalyticsAlert[] = [
  {
    type: 'crisis_spike',
    threshold: 5,  // 5+ crisis escalations
    windowHours: 24,
    action: 'email'
  },
  {
    type: 'error_spike',
    threshold: 10, // 10+ errors
    windowHours: 1,
    action: 'slack'
  },
  {
    type: 'abandonment_spike',
    threshold: 0.5, // 50%+ abandonment rate
    windowHours: 24,
    action: 'log'
  }
];
```

## Database Schema

```sql
CREATE TABLE conversation_analytics (
  id TEXT PRIMARY KEY,
  conversation_id TEXT NOT NULL,
  user_id TEXT NOT NULL,
  started_at TEXT NOT NULL,
  ended_at TEXT,
  message_count INTEGER DEFAULT 0,
  user_message_count INTEGER DEFAULT 0,
  ai_message_count INTEGER DEFAULT 0,
  total_tokens INTEGER DEFAULT 0,
  input_tokens INTEGER DEFAULT 0,
  output_tokens INTEGER DEFAULT 0,
  category TEXT DEFAULT 'unknown',
  outcome TEXT DEFAULT 'in_progress',
  avg_response_time REAL DEFAULT 0,
  had_fallback INTEGER DEFAULT 0,

  FOREIGN KEY (conversation_id) REFERENCES conversations(id),
  FOREIGN KEY (user_id) REFERENCES users(id)
);

CREATE INDEX idx_conv_analytics_started ON conversation_analytics(started_at);
CREATE INDEX idx_conv_analytics_user ON conversation_analytics(user_id);
CREATE INDEX idx_conv_analytics_outcome ON conversation_analytics(outcome);
```

## Testing Analytics

```typescript
describe('Conversation Analytics', () => {
  it('tracks session without PHI', async () => {
    const analytics = await trackConversationStart('conv-123', 'user-456');

    // Verify no PHI is stored
    expect(analytics).not.toHaveProperty('messageContent');
    expect(analytics).not.toHaveProperty('userQuery');

    // Verify metadata is stored
    expect(analytics.conversationId).toBe('conv-123');
    expect(analytics.messageCount).toBe(0);
  });

  it('calculates aggregates correctly', async () => {
    const stats = await getSessionStats(30);

    expect(stats.totalSessions).toBeGreaterThanOrEqual(0);
    expect(stats.completionRate).toBeBetween(0, 1);
  });
});
```

## Resources

- [Chatbot Analytics Guide](https://hiverhq.com/blog/chatbot-analytics)
- [Botpress Analytics](https://botpress.com/blog/chatbot-analytics)
- [13 Core Metrics](https://www.tidio.com/blog/chatbot-analytics/)
- Admin Suite Design: `docs/ADMIN-DEVELOPER-SUITE.md`

© curiositech · MIT · curiositech/some_claude_skills

Similar skills

All skills for AI and ML →

From the same set180

The whole set →
drone-inspection-specialist drone-inspection-specialist — a skill for drone footage analysis: fire detection, roof and hail inspection, thermal checks, and 3D Gaussian Splatting reconstruction. CX Codex +1 227installs drizzle-migrations drizzle-migrations — a skill for schema work with Drizzle ORM on SQLite: tables, columns, indexes, migration generation and applying them. Fits projects where schema changes need to stay repeatable. CX Codex +1 226installs seo-visibility-expert seo-visibility-expert configures technical SEO, llms.txt files for AI scrapers, and schema markup. Useful when preparing web projects for launch. CX Codex +1 224installs mcp-creator mcp-creator is a Claude Code skill for building MCP servers that connect agents to external services. It activates on requests to create or develop an MCP server. CX Codex +1 222installs technical-writer technical-writer structures developer docs, API references, ADRs, and runbooks using the Diátaxis framework with working code examples. CX Codex +1 221installs reactive-dashboard-performance reactive-dashboard-performance sets up React dashboard performance patterns and speeds up async test suites. Helps replace spinners with skeletons and resolve test runner timeouts. CX Codex +1 220installs devops-automator devops-automator — a skill that turns your AI coder into a DevOps engineer: designs pipelines, writes Terraform and Kubernetes configs, sets up GitOps. Use it when building infrastructure or automating deployments. CX Codex +1 219installs bot-developer The bot-developer skill guides bot development for Discord, Telegram, and Slack, covering conversation state, API limits, and moderation. Use it when a bot needs multi-step interactions or automated rules. CX Codex +1 218installs

Project sponsors

Companies that keep this analytics open