Token Lab検索

← フィジカルAI
PHYSICAL AI / 8 OF 8

08 · Robot RL, sim-to-real and deployment evidence

Design rewards and evaluations that reveal failure before making claims about real robots.

Prerequisites: MDPs, PPO, control constraints and statistical evaluation.

Technical posts currently have English and Chinese editions. This locale displays the English article with localized navigation.

Understand the mechanism

RL optimizes a reward in the training environment; it does not know whether the reward matches the intended task. Domain randomization exposes a policy or perception system to variation, but variation must cover meaningful real-world uncertainty. The original domain-randomization paper studies simulated visual variation; randomizing dynamics is a broader experiment, not the same claim. Separate simulation success, hardware transfer, recovery and safety evidence. Evaluate held-out parameter combinations and record every reset and human intervention.

Text / 原始公式
J(θ) = E[Σₜ γᵗ rₜ]
rₜ = progressₜ − λ·effortₜ − μ·violationₜ
SE(p̂) ≈ √[p̂(1−p̂)/n]  [rough large-sample diagnostic]
zero events: one-sided 95% upper bound = 1 − 0.05^(1/n)
Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →
  • Understand: state assumptions, shapes/units and a counterexample.
  • Implement: reproduce a numerical example and inspect boundaries.
  • Investigate: compare one alternative under a matched budget.

Mechanism map & worked derivation

Task + reward + constraints → Train with declared randomization → Held-out stress tests → Gated deployment + incident log
Original schematic; read the steps from top to bottom. SVG ↓
Symbols and assumptions
SymbolMeaning
γ, λ, μDiscount and explicit reward weights
n, p̂Independent trials and observed success fraction
  1. Suppose a policy earns +1 for approaching an object and −0.01 for effort but has no completion term. It may hover nearby instead of grasping. Add task checks, not merely larger weights.
  2. 18 successes in 20 trials give 90% observed success but a rough SE of 6.7 percentage points. Small-sample interval methods and task stratification are preferable for reporting.
  3. With zero failures in 20 independent identical trials, the one-sided 95% upper bound on failure probability is about 13.9%. Zero observed failures is weak safety evidence at this sample size.
Experiment checkpoint

Create an evaluation matrix crossing objects, lighting, mass, friction and delay. Hold out combinations, not only seeds. Report success, intervention rate, p95 latency and violation counts with denominators. Hardware work needs a separate validated control/safety setup.

Original interview exercises & analysis

Answer aloud before revealing the analysis. Add assumptions, a counterexample and a test you would run.

Why can more randomization reduce transfer?

Unrealistic ranges can overwhelm useful training signals or force overly conservative behavior. Measure sensitivity, calibrate ranges and test unseen combinations.

What evidence belongs in an interview project?

A versioned dataset and configuration, baselines, ablations, failure videos or traces, latency distributions and a candid limits section. Clearly distinguish simulated, offline and hardware results.

Read the primary sources

Domain Randomization ↗ · arXiv PDF ↗

Underactuated Robotics — Russ Tedrake ↗

Original Token Lab explanations and synthetic calculations; not measured robot performance. Source links checked 2026-10-25. Unversioned arXiv links may change; record a version for reproduction.

Continue with RL →
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

Zero failures does not imply zero risk. The one-sided 95% upper bound 1−0.05^(1/n) assumes independent Bernoulli trials; deployment drift and correlated resets violate that model.

Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.

Before running: predict a boundary case

Zero failures in 20 IID trials still permits about 13.9% failure probability at the one-sided 95% bound.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node rl-sim2real-evaluation.mjs. No third-party runtime dependencies.

const solve=()=>{
const trials=20,alpha=.05;return 1-alpha**(1/trials);
};
const actual=solve();
const expected=0.13910834066826516;
if(JSON.stringify(actual)===JSON.stringify(expected)){console.log('PASS',actual)}else{
const a=[actual].flat(), b=[expected].flat();
if(a.length!==b.length||a.some((v,i)=>!Number.isFinite(v)||Math.abs(v-b[i])>1e-10))throw Error('Numerical mismatch');
console.log('PASS within 1e-10',actual);
}
Download this example
Reference output — reveal after predicting
0.13910834066826516

Test and transfer

Zero failures in 20 IID trials still permits about 13.9% failure probability at the one-sided 95% bound.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope