future-agi-platform — Claude Code skill: trace and evaluate LLM agents

What future-agi-platform does

Future AGI is an open-source platform for observing and scoring LLM and agent applications. It covers OpenTelemetry-native tracing, more than 50 evaluation metrics, multi-turn simulations, datasets, an OpenAI-compatible gateway, and guardrails. The skill hands your agent working recipes for each part instead of leaving it to read the docs.

Instrumentation ships as separate packages. Python users install fi-instrumentation plus a framework instrumentor such as traceai-openai, traceai-langchain, traceai-llamaindex, or traceai-crewai. On the TypeScript side there are @traceai/fi-core and @traceai/openai. In code you call register(project_name=...), attach the instrumentor, and later calls get traced without extra work. Evaluation lives in fi.evals: Hallucination, Groundedness, ResponseRelevance, ToneCheck, PIICheck, ToolCallAccuracy. You can run one metric, several at once, or a batch over a dataset.

Self-hosting means cloning the repo, copying futureagi/.env.example to futureagi/.env, filling in keys, and running docker compose up -d; the UI comes up at http://localhost:3031. The SDK registers nothing without FI_API_KEY and FI_BASE_URL in the environment, so set those before the first run. A local deployment also expects Postgres, ClickHouse, Redis, and RabbitMQ connection strings in the same .env file.