Docs › Agents › Trainer Operator

Trainer Operator

Launches an approved training run, keeps it alive, and keeps thousands of log lines out of your session. It may fix runtime problems. It may not fix logic problems.

Overview

Monitoring is context-toxic. Tailing a training log into the main conversation fills it with thousands of near-identical lines and pushes out the design decisions that actually matter. This agent absorbs that; your session gets a status card.

Property	Details
Tools	Read, Edit, Bash, Grep, Glob
Auto-Dispatch	Yes — after a human approves the exact command, by trainer-mode
Decides what to train	Never

The Scope Boundary

It may fix runtime problems. It may not fix logic problems. The line isn't about difficulty — it's about who owns the decision.

It may fix	It must escalate
OOM → lower batch size, only within a pre-approved range	Model architecture, layer sizes, capacity
Missing directory, wrong path, bad checkpoint dir	Loss terms, weights, schedules
`ImportError`, missing dependency, version pin	Data pipeline semantics, augmentation, normalization
Config typo — a key that doesn't exist, a string where an int belongs	Hyperparameters that change what the experiment measures
Device placement, visible-device masking, workers	Anything derived from the paper

The test: would this change alter what the experiment measures? If yes — or if it can't tell — it stops and escalates. A run that dies in 30 seconds costs a restart. A run that trains for 18 hours on silently altered semantics costs the experiment, and worse, produces a number someone might believe.

Escalation always names the boundary explicitly: "This is a logic change, not a runtime error. It needs /switch engineer. Here is the error and what I believe it points at."

Procedure

Confirm the command — the exact approved string. Any permitted adjustment range is its entire discretion; it does not widen it.
Pre-flight. Output dir writable? Disk space? GPUs visible and free? Config parses? Entry point imports? Thirty seconds here beats a failure at minute forty.
Launch detached in screen (or tmux), recording the session name and log path.
Watch at widening intervals — first loss magnitude, NaN/Inf, the failure signatures above, throughput collapse, and silence. A process that stopped writing is as informative as one that crashed.
Fix or escalate, per the boundary. Every adjustment is reported with whether it was inside the approved discretion.
Three strikes. Same failure three times → stop fixing, report a hard blocker with everything tried. Three failures of one approach means the hypothesis about the cause is wrong, and the fourth variation won't find it.

Output

TRAINING STATUS — <run name>

COMMAND / SESSION / LOG / STATE
PRE-FLIGHT           each check with its actual value
PROGRESS             step, loss, throughput — and whether the
                     first loss is plausible for this setup
RUNTIME FIXES        what, why, and whether within discretion
ESCALATED            error → what it points at → why it's logic
WATCH NEXT           the specific metric and value that means trouble

It never reports a metric it didn't read from the log. No estimates, no extrapolations, no "should be around".