Skip to main content
skill CX Codex

scene-memory — Claude Code skill for tracking scenes

The scene-memory Claude Code skill tracks object identities, reference states, and event order. It activates when an agent’s next action depends on an earlier observation, a demonstration, or another actor’s move.

MIT ★ 488 updated 2026-11-15

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add Open-RUA/uni-robot-agent --skill scene-memory -a claude-code
Open on GitHub

What scene-memory does

The skill helps an agent keep track of objects through movement and occlusion without confusing their starting positions with their current locations. It separates the demonstrator’s progress from the agent’s own actions, recording transitions when they are observed and retaining uncertainty when one is missed.

Before acting after a demonstration, the agent must establish that the handoff is complete; a still image alone does not prove it. Continuously running environments call for fresh, time-associated observations, since the scene can change while the agent waits. The skill also preserves the original pose and calls for refreshed contact coordinates after an object moves.

What it helps with

  • track similar objects after they move or become occluded
  • keep an object’s initial reference separate from its live position
  • record observed event order while preserving uncertainty about missed transitions
  • confirm a demonstration handoff before the agent acts
  • refresh contact coordinates after an object moves

Example prompts for the agent

  • Track which objects moved in this demonstration and preserve their starting order.
  • Determine whether the demonstrator has finished and handed control to the agent.
  • Keep the object’s original pose and refresh its contact coordinates after it moves.

How to install scene-memory

1Skills CLI (Vercel)

npx skills add Open-RUA/uni-robot-agent --skill scene-memory -a claude-code

2Manually: put the files in place

~/.claude/skills/scene-memory/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/scene-memory/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about scene-memory

How do I install scene-memory in Claude Code?

Run npx skills add Open-RUA/uni-robot-agent --skill scene-memory -a claude-code. The skill file is at ~/.claude/skills/scene-memory/SKILL.md.

What does scene-memory do?

It helps an agent track object identity, reference states, and event order during scene manipulation. It distinguishes observed transitions from planned actions and requires evidence of a completed handoff after a demonstration.

How does scene-memory determine that a demonstration is complete?

It looks for a stable final placement and the demonstrator fully withdrawing to rest across successive physical steps. Still frames alone do not establish completion.

Source text

SKILL.md · MIT
---
name: scene-memory
description: Preserve identities, reference images and event order across occlusion, demonstrations and changing arrangements. Use when a past observation, demonstration handoff or external event determines the next action.
---

# Keep references, events and physical progress separate

Use `general/gripper/state-verification/SKILL.md` for relationship evidence and
completion. This reference preserves identities, initial references and events
needed to choose the next operation.

## Retain identity through changes

Assign distinct identities to similar instances before moving or covering them.
Record any initial positions, orientations or order that the goal refers to;
do not overwrite those references with the objects' later locations. Update
identity-to-location links through observed transfers, not planned endpoints.

A picture or demonstration can establish identity or order without giving a
current executable world position. Match visible distinguishing features and
localize the actual object separately. Preserve identity through observed
transfers; an intended destination cannot identify an unseen face. Derive an
ordered or symbolic plan from the instruction and visible instances, keeping
that plan separate from the confirmed physical relationships.

## Observe a relevant event, then act

Keep a demonstration or another actor's progress separate from your own. Record
an event when a transition is observed, not once per frame showing the same
state. Preserve ambiguous or missed transitions as uncertain.

Give waiting a specific observable purpose and leave the waiting phase when
that event is established. If a later action confirms an earlier operation,
verify that operation before submitting the confirmation; blind replay can
advance the sequence twice.

## Separate sequence recognition from the execution handoff

The hold/step examples below assume a step-driven environment. In a continuously
running environment, use fresh time-associated observations and account for sensor
latency; waiting can change the scene even without a submitted robot action.

When the instruction requires observing a demonstration before acting, track
"sequence identified" separately from "demonstration completed; my phase started."
Inferring the last item or seeing its release does not establish the handoff.
Confirm the final demonstrated action and an explicit start signal or completed
withdrawal to a clear resting posture. "Withdrawing" is not "withdrawn"; a hand
still over the work area or a temporary pause is not sufficient evidence.

Without an explicit start signal, keep your arms in their waiting posture and
use short hold intervals to advance the demonstration. Inspect fresh observations
from successive physical steps: the final placement should remain stable and
the demonstrator should have completed withdrawal and remained at rest across
those observations. Judge the actor and relevant objects, not pixel equality of
the whole image. Static frames alone do not establish completion.

Check observation step identities. Re-reading the same state or sleeping after
queued motion ends advances no physics; apparent stillness then is not evidence
that the demonstration has finished. Wait for hold completion before comparing
its final observation with the previous one. If handoff evidence is ambiguous,
continue bounded waiting and observation rather than starting an exploratory move.

Reorientation and reachability tests are physical actions too, even without
object contact. Keep the waiting posture until the handoff is established.
This applies to observe-then-execute tasks, not explicitly concurrent interaction.

Keep observation checkpoints around placements and the expected phase transition;
do not batch long waits across unresolved events. Use no fixed waiting duration
or step count as a substitute for completion evidence. If an event was missed,
retain uncertainty and inspect recorded observations before relying on its order.

## Keep historical references separate from live state

Associate relevant observations with their capture step or time. Keep the initial
goal reference, latest observed geometry and last confirmed relation separate.
A historical view requested for recognition does not reset the scene or establish
the current object position. After compaction, retain pending job IDs and resolve
them before planning from a fresh completed observation; a queued endpoint is
not an observed state.

A segmentation match on a reference picture can identify the depicted object
without locating its physical instance. Verify scene membership and visible
material before using that geometry for contact. If the object moves, preserve
its identity but refresh contact coordinates using
`general/gripper/moving-target-control/SKILL.md`.

Preserve the original goal reference through compaction as well as movement.
A convenient canonical orientation must not replace a requested historical pose.
For asymmetric objects, retain directed distinguishing features, not only an
undirected axis or bounding box. If pose or order references need a compact
record, read `general/gripper/scene-memory/references/pose-and-order-memory.md`.

© Open-RUA · MIT · Open-RUA/uni-robot-agent

Similar skills

All skills for Agent setup: memory, context, orchestration →

From the same set41

The whole set →
return-to-start return-to-start — a skill for Claude Code and Codex guides robotic arms back to their measured starting poses. Verifies position and orientation tolerances before ending the movement. CX Codex +1 ★ 488repo stars held-object-geometry The held-object-geometry Claude Code skill locates a held object's relevant feature relative to the hand. It applies to clearance checks, placement, and tool contact during robot motion. CX Codex +1 ★ 488repo stars reachability-and-recovery The reachability-and-recovery Claude Code skill selects an arm, approach, and permitted relay for a complete gripper route. It activates when the destination is hard to reach or an existing route needs reassignment. CX Codex +1 ★ 488repo stars handover The handover Claude Code skill transfers an object held by one robot hand to another. It is triggered for hand-to-hand transfers, not ordinary placement with a single arm. CX Codex +1 ★ 488repo stars robodojo-motion-control The robodojo-motion-control Claude Code skill explains RoboDojo motion planning, IK recovery, and diagnostic counters. It helps interpret why a robot did not reach its requested pose. CX Codex +1 ★ 488repo stars supported-contact The supported-contact Claude Code skill plans object contact and the assisting hand’s motion. It activates when a held load needs a receiver adjusted or restrained at the same time. CX Codex +1 ★ 488repo stars pose-targeting The pose-targeting Claude Code skill builds robot motion targets from a position and orientation. It activates for tool changes, reorientation, and aligning a feature on a held object. CX Codex +1 ★ 488repo stars contact-effect The contact-effect Claude Code skill checks contact by observing how the target responds and guides adjustments. It activates for new or uncertain contact, or after a change to grip, tool geometry, or support. CX Codex +1 ★ 488repo stars

Project sponsors

Companies that keep this analytics open