image-to-data — Claude Code skill: extract data from construction photos and drawings
The image-to-data skill analyzes construction site photos, scanned documents, and drawings, extracting text, tables, and detected objects using computer vision and OCR.
MIT 125 installs ★ 434 updated 2026-10-19
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill image-to-data -a claude-codeWhat image-to-data does
image-to-data handles multiple image types: site photos, scanned documents, floor plans, elevations, and detail drawings. It recognizes text (project names, drawing numbers, scales), detects objects (equipment, materials, workers), parses tables, and measures work progress.
The skill includes an OCR engine for text extraction, an object detector for equipment and materials, and a table parser. It supports multiple languages for text recognition. Results arrive structured with bounding boxes, confidence scores, and metadata.
Requires Python 3 and OPENAI_API_KEY set in environment variables.
What it helps with
- recognize text from scanned construction documents or drawings
- detect equipment and objects on site photos
- extract tabular data from images
- track construction progress by analyzing photos
- classify image type — site photo, plan, elevation, or drawing
Example prompts for the agent
- Extract text from a construction drawing and return the project number and scale
- Analyze a site photo and identify all equipment with coordinates and confidence scores
- Pull a table from a scanned document into structured rows and columns
How to install image-to-data
1Skills CLI (Vercel)
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill image-to-data -a claude-code2Manually: put the files in place
~/.claude/skills/image-to-data/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/image-to-data/
1Skills CLI (Vercel)
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill image-to-data -a codex2Manually: put the files in place
.agents/skills/image-to-data/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill image-to-data -a cursor2Manually: put the files in place
.cursor/skills/image-to-data/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction 2_DDC_Book/2.4-PDF-CAD-to-Data/image-to-data2Manually: put the files in place
.github/skills/image-to-data/SKILL.md1Skills CLI (Vercel)
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill image-to-data -a gemini-cli2Platform CLI
gemini skills install https://github.com/datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --path 2_DDC_Book/2.4-PDF-CAD-to-Data/image-to-dataManually: put the files in place
.devin/skills/image-to-data/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill image-to-data -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about image-to-data
How do I install image-to-data in Claude Code?
Run npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill image-to-data -a claude-code, then open ~/.claude/skills/image-to-data/SKILL.md.
What does image-to-data do?
The skill analyzes construction images: performs OCR for text, detects equipment and objects, extracts tables, and measures work progress. It processes site photos, scanned documents, floor plans, and technical drawings.
What do I need to use this skill?
You need Python 3 installed and the OPENAI_API_KEY environment variable set with your OpenAI API key.
Source text
SKILL.md · MIT
---
name: "image-to-data"
description: "Extract data from construction images using AI Vision. Analyze site photos, scanned documents, drawings."
homepage: "https://datadrivenconstruction.io"
metadata: {"openclaw":{"emoji":"📸","os":["darwin","linux","win32"],"homepage":"https://datadrivenconstruction.io","requires":{"bins":["python3"],"env":["OPENAI_API_KEY"]},"primaryEnv":"OPENAI_API_KEY"}}
---
# Image To Data
## Overview
Based on DDC methodology (Chapter 2.4), this skill extracts structured data from construction images using computer vision, OCR, and AI models to analyze site photos, scanned documents, and drawings.
**Book Reference:** "Преобразование данных в структурированную форму" / "Data Transformation to Structured Form"
## Quick Start
```python
from dataclasses import dataclass, field
from enum import Enum
from typing import List, Dict, Optional, Any, Tuple
from datetime import datetime
import json
import base64
class ImageType(Enum):
"""Types of construction images"""
SITE_PHOTO = "site_photo"
SCANNED_DOCUMENT = "scanned_document"
FLOOR_PLAN = "floor_plan"
ELEVATION = "elevation"
DETAIL_DRAWING = "detail_drawing"
PROGRESS_PHOTO = "progress_photo"
SAFETY_PHOTO = "safety_photo"
DEFECT_PHOTO = "defect_photo"
MATERIAL_PHOTO = "material_photo"
EQUIPMENT_PHOTO = "equipment_photo"
class ExtractionType(Enum):
"""Types of data extraction"""
OCR_TEXT = "ocr_text"
TABLE = "table"
OBJECT_DETECTION = "object_detection"
MEASUREMENT = "measurement"
CLASSIFICATION = "classification"
PROGRESS = "progress"
@dataclass
class BoundingBox:
"""Bounding box for detected region"""
x: int
y: int
width: int
height: int
confidence: float = 1.0
@dataclass
class TextRegion:
"""Extracted text region from image"""
text: str
bbox: BoundingBox
confidence: float
language: str = "en"
@dataclass
class DetectedObject:
"""Detected object in image"""
label: str
bbox: BoundingBox
confidence: float
attributes: Dict[str, Any] = field(default_factory=dict)
@dataclass
class ExtractedTable:
"""Extracted table from image"""
headers: List[str]
rows: List[List[str]]
bbox: BoundingBox
confidence: float
@dataclass
class ProgressMeasurement:
"""Progress measurement from image"""
element_type: str
total_count: int
completed_count: int
percent_complete: float
area_sqft: Optional[float] = None
volume_cuft: Optional[float] = None
@dataclass
class ImageAnalysisResult:
"""Complete image analysis result"""
image_id: str
image_type: ImageType
text_regions: List[TextRegion]
detected_objects: List[DetectedObject]
tables: List[ExtractedTable]
progress: Optional[ProgressMeasurement] = None
metadata: Dict[str, Any] = field(default_factory=dict)
processing_time: float = 0.0
class OCREngine:
"""OCR engine for text extraction"""
def __init__(self, engine: str = "tesseract"):
self.engine = engine
self.supported_languages = ["en", "ru", "de", "fr", "es"]
def extract_text(
self,
image_data: bytes,
language: str = "en"
) -> List[TextRegion]:
"""Extract text from image"""
# Simulated OCR extraction (use actual OCR library in production)
# In production: pytesseract, EasyOCR, or cloud OCR services
regions = []
# Simulate detecting title block in drawing
regions.append(TextRegion(
text="PROJECT: OFFICE BUILDING",
bbox=BoundingBox(x=100, y=50, width=300, height=30, confidence=0.95),
confidence=0.95,
language=language
))
regions.append(TextRegion(
text="DRAWING: A-101",
bbox=BoundingBox(x=100, y=90, width=200, height=25, confidence=0.92),
confidence=0.92,
language=language
))
regions.append(TextRegion(
text="SCALE: 1:100",
bbox=BoundingBox(x=100, y=120, width=150, height=20, confidence=0.88),
confidence=0.88,
language=language
))
return regions
def extract_structured_text(
self,
image_data: bytes,
template: Optional[Dict] = None
) -> Dict[str, str]:
"""Extract structured text using template matching"""
# Extract text regions
regions = self.extract_text(image_data)
# Match to template fields
structured = {}
if template:
for field_name, field_config in template.items():
# Find matching region
for region in regions:
if field_config.get("keyword") in region.text.lower():
structured[field_name] = region.text
break
else:
# Default extraction
for region in regions:
if "PROJECT:" in region.text:
structured["project_name"] = region.text.split(":")[-1].strip()
elif "DRAWING:" in region.text:
structured["drawing_number"] = region.text.split(":")[-1].strip()
elif "SCALE:" in region.text:
structured["scale"] = region.text.split(":")[-1].strip()
return structured
class ObjectDetector:
"""Object detection for construction images"""
def __init__(self, model: str = "yolov8"):
self.model = model
self.construction_classes = self._load_construction_classes()
def _load_construction_classes(self) -> Dict[str, Dict]:
"""Load construction-specific object classes"""
return {
# Equipment
"excavator": {"category": "equipment", "safety_zone": 20},
"crane": {"category": "equipment", "safety_zone": 30},
"forklift": {"category": "equipment", "safety_zone": 10},
"concrete_mixer": {"category": "equipment", "safety_zone": 5},
"scaffolding": {"category": "equipment", "safety_zone": 5},
# Safety
"hard_hat": {"category": "ppe", "required": True},
"safety_vest": {"category": "ppe", "required": True},
"safety_glasses": {"category": "ppe", "required": False},
"harness": {"category": "ppe", "required": False},
# Materials
"rebar_bundle": {"category": "material", "unit": "bundle"},
"concrete_block": {"category": "material", "unit": "pallet"},
"lumber_stack": {"category": "material", "unit": "bundle"},
"pipe_stack": {"category": "material", "unit": "bundle"},
# Workers
"worker": {"category": "person", "track": True},
# Building elements
"column": {"category": "structure"},
"beam": {"category": "structure"},
"slab": {"category": "structure"},
"wall": {"category": "structure"},
}
def detect(
self,
image_data: bytes,
confidence_threshold: float = 0.5
) -> List[DetectedObject]:
"""Detect objects in image"""
# Simulated detection (use actual model in production)
# In production: YOLO, Faster R-CNN, etc.
detected = []
# Simulate detected objects
sample_detections = [
("worker", 0.92, BoundingBox(200, 300, 80, 180, 0.92)),
("hard_hat", 0.88, BoundingBox(210, 300, 30, 25, 0.88)),
("safety_vest", 0.85, BoundingBox(210, 340, 60, 80, 0.85)),
("scaffolding", 0.78, BoundingBox(400, 100, 200, 400, 0.78)),
("concrete_block", 0.72, BoundingBox(50, 450, 100, 50, 0.72)),
]
for label, conf, bbox in sample_detections:
if conf >= confidence_threshold:
class_info = self.construction_classes.get(label, {})
detected.append(DetectedObject(
label=label,
bbox=bbox,
confidence=conf,
attributes=class_info
))
return detected
def detect_safety_compliance(
self,
image_data: bytes
) -> Dict:
"""Detect safety compliance in image"""
objects = self.detect(image_data)
workers = [o for o in objects if o.label == "worker"]
hard_hats = [o for o in objects if o.label == "hard_hat"]
vests = [o for o in objects if o.label == "safety_vest"]
compliance = {
"workers_detected": len(workers),
"hard_hats_detected": len(hard_hats),
"vests_detected": len(vests),
"hard_hat_compliance": len(hard_hats) / len(workers) if workers else 1.0,
"vest_compliance": len(vests) / len(workers) if workers else 1.0,
"overall_compliance": "compliant" if len(hard_hats) >= len(workers) else "non-compliant",
"violations": []
}
if len(hard_hats) < len(workers):
compliance["violations"].append({
"type": "missing_hard_hat",
"count": len(workers) - len(hard_hats)
})
return compliance
class TableExtractor:
"""Extract tables from images"""
def extract_tables(
self,
image_data: bytes,
detect_headers: bool = True
) -> List[ExtractedTable]:
"""Extract tables from image"""
# Simulated table extraction
# In production: Camelot, Tabula, or custom CNN
tables = []
# Simulate a schedule table
tables.append(ExtractedTable(
headers=["Activity", "Start", "End", "Duration"],
rows=[
["Foundation", "2024-01-01", "2024-01-15", "14 days"],
["Framing", "2024-01-16", "2024-02-28", "44 days"],
["MEP Rough-in", "2024-03-01", "2024-03-31", "31 days"]
],
bbox=BoundingBox(50, 200, 500, 200, 0.85),
confidence=0.85
))
return tables
def table_to_dataframe(self, table: ExtractedTable) -> Dict:
"""Convert table to dictionary (DataFrame-like)"""
return {
"columns": table.headers,
"data": table.rows,
"records": [
dict(zip(table.headers, row))
for row in table.rows
]
}
class ProgressAnalyzer:
"""Analyze construction progress from images"""
def __init__(self):
self.reference_models = {}
def analyze_progress(
self,
current_image: bytes,
reference_image: Optional[bytes] = None,
element_type: str = "general"
) -> ProgressMeasurement:
"""Analyze progress by comparing images"""
# Simulated progress analysis
# In production: Use semantic segmentation + comparison
# Simulate progress detection
return ProgressMeasurement(
element_type=element_type,
total_count=100,
completed_count=65,
percent_complete=65.0,
area_sqft=15000.0,
volume_cuft=None
)
def compare_with_plan(
self,
site_photo: bytes,
plan_image: bytes
) -> Dict:
"""Compare site photo with plan"""
return {
"match_score": 0.78,
"deviations": [],
"completion_estimate": 65.0,
"areas_of_concern": []
}
class ConstructionImageAnalyzer:
"""
Main class for construction image analysis.
Based on DDC methodology Chapter 2.4.
"""
def __init__(self):
self.ocr = OCREngine()
self.detector = ObjectDetector()
self.table_extractor = TableExtractor()
self.progress_analyzer = ProgressAnalyzer()
def analyze_image(
self,
image_data: bytes,
image_type: ImageType,
image_id: str = "img_001",
extract_types: Optional[List[ExtractionType]] = None
) -> ImageAnalysisResult:
"""
Analyze a construction image.
Args:
image_data: Image data as bytes
image_type: Type of image
image_id: Unique image identifier
extract_types: Types of extraction to perform
Returns:
Complete analysis result
"""
start_time = datetime.now()
if extract_types is None:
extract_types = [ExtractionType.OCR_TEXT, ExtractionType.OBJECT_DETECTION]
text_regions = []
detected_objects = []
tables = []
progress = None
# OCR extraction
if ExtractionType.OCR_TEXT in extract_types:
text_regions = self.ocr.extract_text(image_data)
# Object detection
if ExtractionType.OBJECT_DETECTION in extract_types:
detected_objects = self.detector.detect(image_data)
# Table extraction
if ExtractionType.TABLE in extract_types:
tables = self.table_extractor.extract_tables(image_data)
# Progress analysis
if ExtractionType.PROGRESS in extract_types:
progress = self.progress_analyzer.analyze_progress(image_data)
processing_time = (datetime.now() - start_time).total_seconds()
return ImageAnalysisResult(
image_id=image_id,
image_type=image_type,
text_regions=text_regions,
detected_objects=detected_objects,
tables=tables,
progress=progress,
metadata={"extraction_types": [e.value for e in extract_types]},
processing_time=processing_time
)
def analyze_site_photo(
self,
image_data: bytes,
image_id: str = "site_001"
) -> Dict:
"""Analyze site photo for progress and safety"""
result = self.analyze_image(
image_data,
ImageType.SITE_PHOTO,
image_id,
[ExtractionType.OBJECT_DETECTION, ExtractionType.PROGRESS]
)
safety = self.detector.detect_safety_compliance(image_data)
return {
"image_id": result.image_id,
"objects_detected": len(result.detected_objects),
"progress": result.progress,
"safety_compliance": safety,
"equipment": [o.label for o in result.detected_objects if o.attributes.get("category") == "equipment"],
"materials": [o.label for o in result.detected_objects if o.attributes.get("category") == "material"]
}
def extract_drawing_data(
self,
image_data: bytes,
image_id: str = "dwg_001"
) -> Dict:
"""Extract data from scanned drawing"""
result = self.analyze_image(
image_data,
ImageType.FLOOR_PLAN,
image_id,
[ExtractionType.OCR_TEXT, ExtractionType.TABLE]
)
# Extract title block info
title_block = self.ocr.extract_structured_text(image_data)
return {
"image_id": result.image_id,
"title_block": title_block,
"text_regions": len(result.text_regions),
"tables": [
self.table_extractor.table_to_dataframe(t)
for t in result.tables
],
"all_text": [r.text for r in result.text_regions]
}
def batch_analyze(
self,
images: List[Tuple[bytes, ImageType, str]]
) -> List[ImageAnalysisResult]:
"""Analyze multiple images"""
results = []
for image_data, image_type, image_id in images:
result = self.analyze_image(image_data, image_type, image_id)
results.append(result)
return results
def export_results(
self,
result: ImageAnalysisResult,
format: str = "json"
) -> str:
"""Export analysis results"""
data = {
"image_id": result.image_id,
"image_type": result.image_type.value,
"text_count": len(result.text_regions),
"object_count": len(result.detected_objects),
"table_count": len(result.tables),
"texts": [
{"text": r.text, "confidence": r.confidence}
for r in result.text_regions
],
"objects": [
{"label": o.label, "confidence": o.confidence}
for o in result.detected_objects
],
"processing_time": result.processing_time
}
if format == "json":
return json.dumps(data, indent=2)
else:
raise ValueError(f"Unsupported format: {format}")
```
## Common Use Cases
### Analyze Site Photo
```python
analyzer = ConstructionImageAnalyzer()
# Load image (in production, read from file)
with open("site_photo.jpg", "rb") as f:
image_data = f.read()
result = analyzer.analyze_site_photo(image_data)
print(f"Objects detected: {result['objects_detected']}")
print(f"Safety compliance: {result['safety_compliance']['overall_compliance']}")
print(f"Progress: {result['progress'].percent_complete}%")
```
### Extract Drawing Data
```python
with open("floor_plan.png", "rb") as f:
drawing_data = f.read()
data = analyzer.extract_drawing_data(drawing_data)
print(f"Drawing: {data['title_block'].get('drawing_number')}")
print(f"Project: {data['title_block'].get('project_name')}")
for table in data['tables']:
print(f"Table with {len(table['records'])} rows")
```
### Detect Safety Violations
```python
detector = ObjectDetector()
with open("site_photo.jpg", "rb") as f:
image_data = f.read()
safety = detector.detect_safety_compliance(image_data)
if safety['overall_compliance'] == 'non-compliant':
for violation in safety['violations']:
print(f"Violation: {violation['type']} - Count: {violation['count']}")
```
## Quick Reference
| Component | Purpose |
|-----------|---------|
| `ConstructionImageAnalyzer` | Main analysis engine |
| `OCREngine` | Text extraction |
| `ObjectDetector` | Object detection |
| `TableExtractor` | Table extraction |
| `ProgressAnalyzer` | Progress analysis |
| `ImageAnalysisResult` | Complete analysis result |
## Resources
- **Book**: "Data-Driven Construction" by Artem Boiko, Chapter 2.4
- **Website**: https://datadrivenconstruction.io
## Next Steps
- Use [cad-to-data](../cad-to-data/SKILL.md) for CAD/BIM extraction
- Use [defect-detection-ai](../../../DDC_Innovative/defect-detection-ai/SKILL.md) for defects
- Use [safety-compliance-checker](../../../DDC_Innovative/safety-compliance-checker/SKILL.md) for safety
© datadrivenconstruction · MIT · datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction