Steering Embodied Behavior from Any Discrete Codes

A discrete vocabulary compresses continuous motion into a few named actions, which is what makes planning and annotation over behavior tractable. Such vocabularies are plentiful, from unsupervised segmentation to hand annotation — but they only describe motion. Executing one on a body still requires a hand-engineered controller, or a separately trained policy, per symbol.

Code2Act turns a discrete vocabulary into motor commands for a 38-actuator biomechanical rodent. One recurrent controller reads only the symbol stream and proprioception, is regressed onto the frozen intention of a pretrained imitation model, and decodes through that model's inverse-dynamics module — covering an entire vocabulary with no per-symbol reward or training. The vocabulary is fixed externally and never adapted to the controller, so the front end is swappable: the same controller is driven by an unsupervised AR-HMM segmenter (Keypoint-MoSeq), a learned kinematics-VAE codebook, and a hand-specified label set.

The symbols then generate rather than merely describe. Their motion is distributionally close to the expert imitation policy and far from a random stream over the same vocabulary; an independent kinematic labeler re-reads the generated behavior as the commanded symbol in 0.71 of cases; and a commanded symbol survives perturbation, reached from 100 distinct initial states and restored after an external velocity impulse. The interface also reports which symbols the body can actually realize, and composes symbols into transitions the corpus never contains while staying within the distribution of rodent motion. A description of behavior can therefore be made to produce it.

The hierarchical motor control pathway: prefrontal/premotor cortex generates competing plans, basal ganglia selects discrete initial conditions, and motor cortex (M1) evolves continuous trajectories through the brainstem and spinal cord to muscles.

This project builds on earlier work on dynamic modeling for biomechanical planning. See MimicDyn for the previous iteration that motivated this direction. This work is advised by professor Scott W Linderman at Stanford University and Talmo D Pereira at the Salk Institute for Biological Studies.