How It Works: The Architecture and the Plan
Part 2 of the build log — how it actually works: the architecture and the phased plan. (Part 1: the vision · Part 3: current state →)
The big "how"
The architecture is three layers, with a hard rule about who is allowed to be expensive.
Decompose once, expensively. Execute and verify continuously, cheaply. Escalate only on disagreement.
flowchart TB
M(["Mission — in natural language"])
subgraph CLOUD["Cloud · expensive · rarely"]
direction TB
LLM["High-capability model<br/>mission + scene to typed plan"]
PLAN["Typed plan<br/>bounded behaviors · success criteria<br/>pre-declared escalation conditions"]
LLM --> PLAN
end
subgraph DEVICE["On-device · cheap · constantly"]
direction LR
EXEC["Execute and verify<br/>cheap perception checks"]
NAV["Navigation stack<br/>pose · detector · timeout"]
BOT["Crawler<br/>moves through the world"]
EXEC --> NAV --> BOT
end
subgraph HW["Hardware · safe by default"]
SW["Transmitter channel<br/>selects RC input vs. autonomy"]
end
M --> LLM
PLAN -->|handed down| EXEC
EXEC -.->|escalate on disagreement| LLM
BOT -.-> SW
SW -.->|switch returns pure RC| BOT
classDef cloud fill:#EEF2FF,stroke:#6366F1,color:#312E81;
classDef device fill:#ECFDF5,stroke:#10B981,color:#065F46;
classDef hw fill:#FFFBEB,stroke:#F59E0B,color:#92400E;
class LLM,PLAN cloud;
class EXEC,NAV,BOT device;
class SW hw;
- An expensive, high-capability model (cloud-side) takes a mission and the initial scene and turns it into a typed plan — a structured, validatable sequence of behaviors, each with success criteria and pre-declared conditions for when to ask for help. This happens rarely.
- A cheap, on-device layer executes that plan, drives the navigation stack, and continuously checks whether reality matches the plan. This happens constantly.
- Control escalates back to the expensive model only when the on-device layer detects a disagreement it can't resolve.
The dependent variable — the thing the whole project is really measuring — is that escalation policy. How much grounding accuracy does smart escalation buy you per unit of compute, battery, and latency? That's the curve I want to produce.
Safe by default
Underneath all of it sits a non-negotiable design choice: manual control is a hardware-level fallback, not a software mode. A transmitter channel selects between human RC input and the robot's own commands at the hardware level. If the autonomy stack crashes, locks up, or does something dumb, a switch on the transmitter returns the machine to pure RC control. Autonomy lives on top of a system that is safe by default.
stateDiagram-v2
direction LR
[*] --> ManualRC
ManualRC: Manual RC control
Autonomy: Autonomy in command
ManualRC --> Autonomy: transmitter channel selects autonomy
Autonomy --> ManualRC: switch flipped, stack crash, or fault
note right of ManualRC
Selection happens at the hardware level.
Safe by default — autonomy lives on top.
end note
The plan, in phases
I've structured the project so that every phase ends in something real — a working demo, a dataset, a result, a plot — that has value even if the next phase never happens. This is deliberate. Ambitious systems projects die in the gap between "impressive architecture diagram" and "thing that works," and the cure is to make each step independently shippable.
flowchart TB
classDef current fill:#EDE9FE,stroke:#8B5CF6,color:#5B21B6;
classDef core fill:#D1FAE5,stroke:#10B981,color:#065F46;
classDef deferred fill:#FAFAFA,stroke:#A1A1AA,color:#71717A,stroke-dasharray:5 4;
P1["Phase −1 · CURRENT — RC-only bring-up<br/><i>driving chassis · manual override · first drive logs</i>"]:::current
P0["Phase 0 — The contract<br/><i>plan schema + mission→plan fixtures</i>"]
P2["Phase 1 — Cloud decomposition<br/><i>decomposition accuracy + failure catalogue</i>"]
P3["Phase 2 — On-device execution<br/><i>robot runs pre-authored plans</i>"]
P4["Phase 3 · CORE RESULT — Closing the loop<br/><i>accuracy-vs-cost curve across 3 policies</i>"]:::core
P5["Phase 4 · deferred — Richer verification<br/><i>a second, comparable curve</i>"]:::deferred
P1 --> P0 --> P2 --> P3 --> P4 --> P5
Phase −1 — RC-only bring-up (current)
Get the bare chassis driving reliably under transmitter control. Wire and test the hardware fallback that lets a human take over instantly. Have the motor controller timestamp and log every manual input as I drive.
Deliverable: a video of the crawler moving under its own power, a proven manual-override path, and the first logged drive traces.
Phase 0 — The contract
Lock the plan schema before writing any robot autonomy code: what a "mission" is, what a "plan" looks like, what counts as success, and what triggers an escalation. Define a small fixed set of mission templates and one controlled test scene with known ground-truth positions.
Deliverable: a system-design document and a set of hand-authored mission → plan examples that double as test fixtures.
Phase 1 — Cloud decomposition
Prove the expensive model can turn a mission into a valid, executable plan — with no robot involved yet. Score its output against the hand-authored plans and document where it gets the decomposition wrong.
Deliverable: a decomposition-accuracy result and a catalogue of failure cases.
Phase 2 — On-device execution
Run a given plan on the actual crawler using cheap perception checks only — did it reach the pose, is the target detector firing, did it time out. Escalation conditions are logged but not yet acted on.
Deliverable: the robot autonomously executing pre-authored plans, plus a log of every escalation that would have fired. The first real autonomy demo.
Phase 3 — Closing the loop
Implement and compare three escalation policies: never escalate, always escalate, and escalate-on-disagreement. Instrument grounding accuracy, time-to-detection, battery, latency, and number of expensive calls.
Deliverable: the core result — a curve of grounding accuracy against cost across the three policies.
Phase 4 — Richer verification (deferred)
Replace the cheap perception checks with genuine on-device visual re-grounding and re-run the curve. Does richer verification move the cheap-and-smart hybrid closer to the expensive ceiling?
Deliverable: a second, directly comparable curve — future work, delivered.
Where this is going
The end state is a small, rugged robot you can talk to in plain language, that plans intelligently when it has to, executes cheaply when it can, knows when it's wrong, and stays safe by default — running offline, sipping power, built from parts you can replace. The crawler is how I get there one shippable phase at a time.
Part 3 is where the plan meets reality — the actual bill of materials, a tightened scope, and why I'm starting with software while the hardware is still in the mail: Tightening the scope →.
More as it crawls.