simulation-calibrator · davepoon — Claude Code command: setup and guide
simulation-calibrator calibrates simulation accuracy through back-testing and structured validation loops. Useful when tuning business, behavioral, or technical system models.
MIT ★ 3,785 updated 2026-10-25
/plugin marketplace add davepoon/buildwithclaude
/plugin install all-commands@buildwithclaudeWhat simulation-calibrator does
The command begins by evaluating simulation scope and precision targets. Mission-critical tasks demand 95%+ precision, while exploratory work allows a 50–70% threshold. Missing context stops the procedure immediately. The assistant prompts for domain details first: business projections, architecture scaling, or user behavior.
Performance tracking relies on historical back-testing. Four core dimensions each carry a 25% weight: input reliability, model structure, methodology rigor, and output utility. Beyond raw error margins, it tracks directional trends and timing offsets.
A dedicated module targets structural skew. It flags survivorship bias, recency skew, linear assumptions, and cognitive anchoring. Corrections include red-teaming and Bayesian parameter updates.
What it helps with
- evaluate simulation accuracy against historical outcomes via back-testing
- detect cognitive, data, and structural modeling biases
- calculate a weighted simulation quality score across four 25% categories
- establish a four-level validation cycle from internal logic to live predictions
Example prompts for the agent
- /simulation-calibrator calibrate our quarterly revenue model against last year results
- /simulation-calibrator check backend capacity simulation for linear and selection biases
- /simulation-calibrator build a validation loop for user retention forecasts at 85% accuracy
How to install simulation-calibrator
1Plugin marketplace
/plugin marketplace add davepoon/buildwithclaude/plugin install all-commands@buildwithclaude2Manually: put the files in place
.claude/commands/simulation-calibrator.mdCall it with /simulation-calibrator. For all projects use ~/.claude/commands/
Manually: put the files in place
~/.codex/prompts/simulation-calibrator.mdCall it with /prompts:simulation-calibrator. Codex marks this format deprecated in favour of skills
Manually: put the files in place
.github/prompts/simulation-calibrator.prompt.mdManually: put the files in place
.gemini/commands/simulation-calibrator.tomlGemini stores commands as TOML: the prompt text goes into the prompt field
Questions about simulation-calibrator
How to install simulation-calibrator in Claude Code?
First add the marketplace using /plugin marketplace add davepoon/buildwithclaude, then run /plugin install all-commands@buildwithclaude. The command will be located at .claude/commands/simulation-calibrator.md.
What does simulation-calibrator do?
It guides step-by-step calibration of predictive models. It measures accuracy baselines, spots analytical biases, and structures validation workflows.
How does the command assess simulation quality?
It computes a weighted score across four areas at 25% each: input data completeness, model sophistication, process discipline, and actionable output quality.
Source text
simulation-calibrator.md · MIT
--- description: Test and refine simulation accuracy with validation loops, bias detection, and continuous improvement frameworks. category: simulation-modeling argument-hint: "Specify calibration parameters" --- # Simulation Calibrator Test and refine simulation accuracy with validation loops, bias detection, and continuous improvement frameworks. ## Instructions You are tasked with systematically calibrating simulations to ensure accuracy, reliability, and actionable insights. Follow this approach: **$ARGUMENTS** ### 1. Prerequisites Assessment **Critical Calibration Context Validation:** - **Simulation Type**: What kind of simulation are you calibrating? - **Accuracy Requirements**: How precise does the simulation need to be? - **Validation Data**: What real-world data can test simulation accuracy? - **Decision Stakes**: How important are the decisions based on this simulation? - **Update Frequency**: How often should calibration be performed? **If context is unclear, guide systematically:** ``` Missing Simulation Context: "What type of simulation needs calibration? - Business Simulations: Market response, financial projections, strategic scenarios - Technical Simulations: System performance, architecture behavior, scaling predictions - Process Simulations: Operational workflows, resource allocation, timeline predictions - Behavioral Simulations: Customer behavior, team dynamics, adoption patterns Each simulation type requires different calibration approaches and validation methods." Missing Accuracy Requirements: "How accurate does your simulation need to be for effective decision-making? - Mission Critical (95%+ accuracy): Safety, financial, or legal decisions - Strategic Planning (80-95% accuracy): Investment, expansion, or major initiative decisions - Operational Optimization (70-80% accuracy): Process improvement and resource allocation - Exploratory Analysis (50-70% accuracy): Option generation and conceptual understanding" ``` ### 2. Baseline Accuracy Assessment **Establish current simulation performance levels:** #### Historical Validation Framework ``` Simulation Accuracy Baseline: Back-Testing Analysis: - Compare simulation predictions to known historical outcomes - Measure prediction accuracy across different time horizons - Identify systematic biases and error patterns - Assess prediction confidence calibration Accuracy Metrics: - Overall Prediction Accuracy: [percentage of correct predictions] - Directional Accuracy: [percentage of correct trend predictions] - Magnitude Accuracy: [percentage of predictions within acceptable error range] - Timing Accuracy: [percentage of events predicted within correct timeframe] - Confidence Calibration: [alignment between prediction confidence and actual accuracy] Error Pattern Analysis: - Systematic Biases: [consistent over/under-estimation patterns] - Context Dependencies: [accuracy variations by scenario type or conditions] - Time Horizon Effects: [accuracy changes over different prediction periods] - Complexity Correlation: [accuracy relationship to scenario complexity] ``` #### Simulation Quality Scoring ``` Quality Assessment Framework: Input Quality (25% weight): - Data completeness and accuracy - Assumption validation and documentation - Expert input quality and consensus - Historical precedent availability Model Quality (25% weight): - Algorithm sophistication and appropriateness - Relationship modeling accuracy and completeness - Constraint modeling and boundary definition - Uncertainty quantification and propagation Process Quality (25% weight): - Systematic methodology application - Bias detection and mitigation - Stakeholder validation and feedback integration - Documentation and reproducibility Output Quality (25% weight): - Prediction accuracy and reliability - Insight actionability and clarity - Decision support effectiveness - Communication and presentation quality Overall Simulation Quality Score = Sum of weighted component scores ``` ### 3. Systematic Bias Detection **Identify and correct simulation biases:** #### Bias Identification Framework ``` Common Simulation Biases: Cognitive Biases: - Confirmation Bias: Seeking information that supports expected outcomes - Anchoring Bias: Over-relying on first estimates or reference points - Availability Bias: Overweighting easily recalled or recent examples - Optimism Bias: Systematic overestimation of positive outcomes - Planning Fallacy: Underestimating time and resource requirements Data Biases: - Selection Bias: Non-representative data samples - Survivorship Bias: Only analyzing successful cases - Recency Bias: Overweighting recent data points - Historical Bias: Assuming past patterns will continue unchanged - Measurement Bias: Systematic errors in data collection Model Biases: - Complexity Bias: Over-simplifying or over-complicating models - Linear Bias: Assuming linear relationships where non-linear exist - Static Bias: Not accounting for dynamic system changes - Independence Bias: Ignoring correlation and interaction effects - Boundary Bias: Incorrect system boundary definition ``` #### Bias Mitigation Strategies ``` Systematic Bias Correction: Process-Based Mitigation: - Multiple perspective integration and diverse expert consultation - Red team analysis and devil's advocate approaches - Assumption challenging and alternative hypothesis testing - Structured decision-making and bias-aware processes Data-Based Mitigation: - Multiple data source integration and cross-validation - Out-of-sample testing and validation dataset use - Temporal validation across different time periods - Segment validation across different contexts and conditions Model-Based Mitigation: - Ensemble modeling and multiple algorithm approaches - Sensitivity analysis and robust parameter testing - Cross-validation and bootstrap sampling - Bayesian updating and continuous learning integration ``` ### 4. Validation Loop Design **Create systematic accuracy improvement processes:** #### Multi-Level Validation Framework ``` Comprehensive Validation Approach: Level 1: Internal Consistency Validation - Logical consistency checking and constraint satisfaction - Mathematical relationship verification and balance testing - Scenario coherence and narrative consistency - Assumption compatibility and interaction validation Level 2: Expert Validation - Domain expert review and credibility assessment - Stakeholder feedback and perspective integration - Peer review and professional validation - External advisor consultation and critique Level 3: Empirical Validation - Historical data comparison and pattern matching - Market research validation and customer feedback - Pilot testing and proof-of-concept validation - Real-world experiment and A/B testing Level 4: Predictive Validation - Forward-looking accuracy testing and prediction tracking - Real-time outcome monitoring and comparison - Continuous feedback integration and model updating - Long-term performance assessment and trend analysis ``` #### Feedback Integration Mechanisms - Automated accuracy tracking and alert systems - Stakeholder feedback collection and analysis - Expert consultation and validation scheduling - Real-world outcome monitoring and comparison ### 5. Real-Time Calibration Systems **Establish ongoing accuracy monitoring and adjustment:** #### Continuous Monitoring Framework ``` Real-Time Calibration Dashboard: Accuracy Tracking Metrics: - Current Prediction Accuracy: [real-time accuracy percentage] - Accuracy Trend: [improving, stable, or declining accuracy] - Bias Detection: [systematic error patterns identified] - Confidence Calibration: [prediction confidence vs. actual accuracy alignment] Early Warning Indicators: - Prediction Deviation Alerts: [when predictions diverge significantly from reality] - Model Drift Detection: [when model performance degrades over time] - Assumption Violation Warnings: [when key assumptions prove incorrect] - Data Quality Alerts: [when input data quality degrades] Automated Adjustments: - Parameter Recalibration: [automatic model parameter updates] - Weight Rebalancing: [factor importance adjustments based on performance] - Threshold Updates: [decision threshold modifications based on accuracy] - Alert Sensitivity: [notification threshold adjustments] ``` #### Adaptive Learning Integration - Machine learning model updates based on new data - Bayesian updating for probability and parameter estimation - Expert feedback integration and model refinement - Context-aware calibration for different scenario types ### 6. Calibration Quality Assurance **Ensure systematic improvement and reliability:** #### Calibration Validation Framework ``` Meta-Calibration Assessment: Calibration Process Quality: - Validation methodology appropriateness and rigor - Feedback integration effectiveness and speed - Bias detection and mitigation success - Continuous improvement demonstration Calibration Outcome Quality: - Accuracy improvement measurement and tracking - Prediction reliability enhancement - Decision support effectiveness improvement - Stakeholder confidence and satisfaction growth Calibration Sustainability: - Process scalability and resource efficiency - Knowledge capture and institutional learning - Methodology transferability to other simulations - Long-term performance maintenance and enhancement ``` #### Quality Control Mechanisms - Independent calibration validation and audit - Cross-functional calibration team and review processes - External benchmark comparison and best practice integration - Documentation and knowledge management systems ### 7. Simulation Improvement Roadmap **Generate systematic enhancement strategies:** #### Calibration-Based Improvement Plan ``` Simulation Enhancement Framework: ## Simulation Calibration Analysis: [Simulation Name] ### Current Performance Assessment - Baseline Accuracy: [current accuracy percentages] - Key Biases Identified: [systematic errors found] - Validation Coverage: [validation methods applied] - Stakeholder Confidence: [user trust and satisfaction levels] ### Calibration Findings #### Accuracy Analysis: - Strong Performance Areas: [where simulation excels] - Accuracy Gaps: [where improvements are needed] - Bias Patterns: [systematic errors identified] - Validation Results: [validation testing outcomes] #### Improvement Opportunities: - Quick Wins: [immediate accuracy improvements available] - Strategic Enhancements: [longer-term improvement possibilities] - Data Quality Improvements: [input enhancement opportunities] - Model Sophistication: [algorithm and methodology upgrades] ### Improvement Roadmap #### Phase 1: Immediate Fixes (30 days) - Critical bias corrections and parameter adjustments - Data quality improvements and source validation - Process enhancement and workflow optimization - Stakeholder feedback integration and communication #### Phase 2: Systematic Enhancement (90 days) - Model sophistication and algorithm upgrades - Validation framework expansion and automation - Feedback loop optimization and real-time calibration - Training and capability building for users #### Phase 3: Advanced Optimization (180+ days) - Machine learning integration and automated improvement - Cross-simulation learning and best practice sharing - Innovation and methodology advancement - Strategic capability building and competitive advantage ### Success Metrics and Monitoring - Accuracy Improvement Targets: [specific goals and timelines] - Bias Reduction Objectives: [systematic error elimination goals] - Validation Coverage Goals: [comprehensive validation targets] - User Satisfaction Improvements: [stakeholder confidence goals] ``` ### 8. Knowledge Capture and Transfer **Establish institutional learning from calibration:** #### Learning Documentation - Calibration methodology documentation and best practices - Bias detection and mitigation technique libraries - Validation approach templates and reusable frameworks - Success pattern identification and replication guides #### Cross-Simulation Learning - Calibration insight sharing across different simulations - Best practice identification and standardization - Common pitfall documentation and avoidance strategies - Expertise development and capability building programs ## Usage Examples ```bash # Business simulation calibration /simulation:simulation-calibrator Calibrate customer acquisition cost simulation using 12 months of actual campaign data # Technical simulation validation /simulation:simulation-calibrator Validate system performance simulation against production monitoring data and user experience metrics # Market response calibration /simulation:simulation-calibrator Calibrate market response model using A/B testing results and customer behavior analytics # Strategic scenario validation /simulation:simulation-calibrator Test business scenario accuracy using post-decision outcome analysis and market development tracking ``` ## Quality Indicators - **Green**: Systematic validation, documented biases, automated monitoring, continuous improvement - **Yellow**: Regular validation, some bias detection, manual monitoring, periodic improvement - **Red**: Ad-hoc validation, undetected biases, no monitoring, no improvement tracking ## Common Pitfalls to Avoid - Validation theater: Going through validation motions without learning - Bias blindness: Not recognizing systematic errors and prejudices - Static calibration: Not updating models based on new information - Perfection paralysis: Waiting for perfect accuracy before using insights - Context ignorance: Not adapting calibration to different scenarios - Learning isolation: Not sharing insights across teams and simulations Transform simulation accuracy from guesswork into systematic, reliable decision support through comprehensive calibration and continuous improvement.
© BuildWithClaude Community · MIT · davepoon/buildwithclaude
Similar slash commands
All slash commands for AI and ML →simulation-calibrator by other authors comes first.