Behavior has names. Bodies need commands.
Behavioral neuroscience and embodied AI both describe behavior with discrete vocabularies. Keypoint-MoSeq splits motion into syllables, pose classifiers assign discriminative classes, annotators write ethograms, and planners emit symbols. Over such a vocabulary, planning and annotation become tractable. But none of these symbols can be executed: they describe recorded motion, and making a body perform one usually takes a hand-engineered controller or a policy trained, with its own reward, for that symbol alone.
Controllers that do act through a discrete bottleneck learn their own codebook, optimized to reconstruct motion, so any planner, language model or annotation scheme that issues commands has to adopt the controller’s vocabulary instead of its own. Code2Act inverts this for the 38-actuator biomechanical rat of MIMIC-MJX. The vocabulary is defined outside the controller and fixed before training; the controller never shapes it. One recurrent controller reads only that symbol stream and the body’s proprioception, maps each symbol into the latent space of a frozen, pretrained imitation model, and decodes it to torques. A single training run covers an entire vocabulary, and the vocabulary can be swapped for any other labeling without changing the controller’s design.
On this page we walk through what the controller does, with the rollouts behind the paper’s figures rendered as video and several of the analyses made interactive.
A note on the examples. The video clips on this page are curated for the best visualization: from each set of rollouts we show an example chosen by a fixed rule, usually the clearest or best-recovering one, and each caption states that rule. The numbers are not curated. Tables, charts, label-recovery bars and success rates use all rollouts, as in the paper.
How Code2Act works
Code2Act is built in three stages, each run as its own process. (1) A pretrained imitation model supplies both a privileged teacher (its frozen encoder) and an inverse-dynamics decoder. (2) An external vocabulary assigns every reference frame a discrete symbol; it can be any labeling, and the controller never changes it. (3) A recurrent controller learns, by reinforcement learning on its own rollouts, to reproduce the motor intention the imitation model would have produced from the full reference, while seeing only the symbols. Stage 3 is the contribution: Stage 1 is the model it decodes through, initialized from the imitator and fine-tuned with the controller, and Stage 2 is its input.
An external vocabulary, defined outside Code2Act and frozen, turns the reference motion into a stream of discrete symbols. A recurrent controller reads only that stream and the body's proprioception, predicts the motor intention a pretrained imitation model would have produced, and decodes it to torques through that model's inverse-dynamics decoder.
The goal trajectory only reaches a frozen encoder that supplies a KL target during training. It never enters the action path.
Pick a tab to light up the modules and data paths active in each stage. The symbol box cycles through real bout sequences of each vocabulary. In the controller the goal trajectory reaches only the frozen encoder, which supplies a stop-gradient KL target and never the action path.
The key property comes from the pretrained latent. The imitation model’s intention encodes the change the body must apply to reach the next pose from its current state, so no single intention can serve a symbol across every body state. One symbol therefore maps to many intention trajectories, each selected by the body’s state at that moment. That is why a held symbol can reach the same motion from very different starting points, as the experiments below show.
Three vocabularies, one controller
Behavioral neuroscience already maintains many such vocabularies, built independently of any controller. We train the same controller design on three of them (one training run each) that differ in where they come from and how large they are:
- KPMS: unsupervised Keypoint-MoSeq syllables, K = 100. Usage is heavy-tailed; the 25 most-used syllables cover about 97% of frames.
- KVAE: latents of a kinematics VAE on short pose windows, clustered with K-means into K = 25 codes.
- MLB: nine manually labeled behaviors, Walk, Rear and Turn crossed with Left, Straight and Right, plus Immobile. MLB is written by hand rather than fit to the corpus, as a human annotation scheme would supply it. Because the three differ in origin, in size, and in whether they were fit to data at all, any property that holds for all three belongs to Code2Act rather than to a particular vocabulary.
Below, each code is held constant on one body.
One body holds each code for six seconds, drawn as in the paper’s codebook figures: the live body, a fading trail of its recent past, and its root path, colored by behavior (hue) and direction (shade). For each code we show the body whose motion the labeler reads as the code’s most common behavior, and among those the one with the median path length. Click a clip to enlarge it. Several directional variants share a base motion, and Turn L is executed as a tightly curving left walk.
Executed on the body, not every symbol is distinguishable. The nine MLB symbols steer to about five distinct behaviors: there are only about 100 labeled Walk R frames in the corpus, and a rear is the same movement whether it is labeled left or right. Keypoint-MoSeq concentrates its usage on about 7 of its 100 syllables, while the KVAE codebook stays near its full size (an effective 24 of 25 codes). The check therefore discriminates between vocabularies rather than always finding collapse: running a vocabulary on a body shows which of its symbols are distinct actions, which are redundant, and which distinctions the data cannot support.
Steer it yourself
The controller is not limited to holding one symbol. Chain symbols into a command stream and it moves between motor programs.
Try it live. Compose your own symbol sequence, or switch symbols on the fly, and watch the paper’s MLB controller execute it. Physics and controller run in your browser, with no server.
Open the live demo ↗Faint clouds: GRU states while each symbol is held. The bright trace is this rollout, colored by the commanded symbol.
Each preset chains MLB symbols into a command stream for the paper’s MLB controller, running decoder-only from a test-set start pose. The command lane is the symbol stream; the readout lane is what the independent kinematic labeler reads from the motion. The right panel follows the controller’s GRU state on the first two principal components of its states while each symbol is held (faint clouds), so you can watch the state leave one symbol’s region and settle into the next. Drag the timeline to scrub. For each preset we rolled out 48 bodies from different start poses and show the one whose readout best matches the command; the agreement of the shown body and the median over all 48 are printed under the viewer.
Symbols steer realistic behavior
Checking individual symbols is only meaningful if the controller’s motion looks like a rat. We quantify realism with the kernel inception distance (KID) between controller rollouts and the expert MIMIC-MJX policy, computed in the feature space of a small VAE trained on expert motion. Each controller is driven by five code streams: its original codes, samples from the empirical transition matrix, two learned autoregressive priors (GPT and Mamba), and uniformly random codes.
| Steering strategy | Original | Transition | Mamba | GPT | Random |
|---|---|---|---|---|---|
| KPMS (K = 100) | 0.251 ± 0.007 | 0.378 ± 0.012 | 0.298 ± 0.014 | 0.268 ± 0.014 | 0.664 ± 0.024 |
| KVAE (K = 25) | 0.247 ± 0.031 | 0.266 ± 0.022 | 0.292 ± 0.026 | 0.244 ± 0.015 | 0.998 ± 0.099 |
| MLB (K = 9) | 0.162 ± 0.027 | 0.198 ± 0.036 | 0.246 ± 0.020 | 0.205 ± 0.037 | 0.729 ± 0.073 |
Every structured stream reaches a KID 1.8 to 4.5 times lower than the random stream of the same vocabulary, with the corpus sequences lowest for two of the three vocabularies and the learned priors close behind: realism tracks the temporal structure of the symbol stream, for all three vocabularies. This is a property of the motion distribution, so we report it as a table rather than as a single clip.
Inside the controller
The body and the controller organize the same symbols differently. We compare per-symbol dynamics with dynamical similarity analysis (DSA), once on the body state and once on the GRU state. On the body, symbols split into held postures and locomotion: the three rears group with Immobile, and the walks and turns form the other block. Inside the controller the split moves: Walk L and Walk S separate from everything else, and the turns group with Walk R rather than with the other walks. This mirrors the data: in the corpus a turn is mostly a curving walk, and Walk R is the rarest symbol.


Pairwise DSA angular distance between per-symbol operators (the paper’s Fig. 3a,b), each matrix ordered by average-linkage clustering of its own distances (dendrogram above) and drawn on its own color scale. Outlined blocks are the clusters at the paper’s fixed cut (dashed line). Behavior labels play no part in the ordering.
The matrices summarize dynamics; the trajectories themselves are below. Each held symbol starts from a random pose, and the GRU state travels into a region of its own.
GRU-state trajectories of the MLB controller holding each symbol, projected on the first three principal components. Press play, or drag the slider, to grow every trajectory from its start pose; drag the plot to rotate it. Click a symbol to hide it, or double-click to show it alone.
The same symbol, from anywhere
A symbolic layer needs each symbol to mean the same behavior whenever it is issued, from whatever state the body is in. A symbol that merely replayed a stored trajectory would work from one start pose. Because each symbol is defined outside the controller, re-labeling the motion it produces scores the controller against a criterion it was never trained on. We hold each symbol from 60 start poses and let the vocabulary’s own labeler read the motion back, a test we call label recovery.
Four bodies, four random start poses, one held symbol: Walk L. Each tag is the readout of the same kinematic labeler that scores label recovery, computed on that body's settled motion.
60 start poses per symbol, each strategy read back by its own labeler. Chance is 11% (symbol) and 25% (behavior).
Left: four of the 60 start poses for the selected MLB symbol (a seeded random draw), with the readout label recovery assigns to each. Right: each commanded symbol’s readouts split into the symbol itself, the same behavior in another direction, and a different behavior; the most common wrong destination is printed for large error segments. Switch vocabularies to compare KPMS and KVAE, which are read back by their own labelers.
For MLB the commanded behavior is recovered in 69% of instantiations and the exact symbol in 48%, against chance levels of 25% and 11%; KVAE reaches 83% and 80%, and KPMS 51% and 48%. Most of the shortfall is the directional collapse seen above: 8 of the 9 symbols place their most common readout inside the correct behavior. The exception, Turn L, reads as Walk L in 98% of instantiations, which makes it the most consistent symbol in the codebook, executed as a curving walk.
Kick it, and it comes back
Label recovery only varies the start pose. We also perturb the body during execution: after it settles under a held symbol, a single-step velocity impulse of scale a hits it, and the controller, still reading only the symbol, has to bring it back.
Both panels are the same body from the same start pose, holding Rear R. On the right a single-step velocity impulse hits at 2 s (the body flashes red), and the controller, still reading only the symbol, tries to bring it back to the motion of its unperturbed twin. For each impulse size we show the best-recovering of the 48 bodies kicked per symbol; the curves on the right average all 48, including those that do not recover.
The MLB controller. Left: one body and its unperturbed twin from the same start pose, holding Rear R or Walk L, kicked at the impulse size you pick. The kicked body flashes red at the impulse and fades back as it recovers. For each impulse size we show the best-recovering of 48 bodies: for the rear, among bodies that took at least a median-sized shove, the one whose posture ends closest to its twin’s; for the walk, among bodies walking steadily through their last three seconds, the one closest to its twin’s pace. Right: the share of the impulse-induced displacement that has decayed five seconds later, and the pose deviation from unperturbed rollouts against the unperturbed floor (dotted) and the distance between different symbols (dashed), averaged by behavior over all 48 bodies per symbol.
Five seconds after the impulse, 85% (MLB), 92% (KPMS) and 74% (KVAE) of the induced displacement has decayed on average, with little dependence on impulse size. Across a 6.7-fold range of impulse, MLB’s peak displacement grows 7.9-fold but its final displacement only 2.0-fold. The deviation from unperturbed motion grows with the impulse yet stays below the distance between distinct symbols for every behavior at a = 0.6. For MLB, rears cross that line at a = 1.2 and turns at a = 4.0, while walks and immobility never do; a rear sits near the limit of static stability. Recovery is consistent across every tested symbol and vocabulary but depends on posture: rear, the posture nearest the limit of static stability, is the least stable.
Transitions the corpus never contains
Can the controller execute symbol pairs it has never seen? For MLB, 30 ordered pairs have zero bout-level transitions anywhere in the reference corpus. We command each as two phases of 250 control steps, holding A and then B, from 60 initial states.
Rear L → Walk R never occurs in the training corpus. Across 50 start poses the labeler reads the held rear in 74% of bodies and the commanded walk after the switch in 98%.
The matrix lists every ordered pair of MLB symbols. Colored cells are the 30 pairs that never occur in the corpus; click one to play its rollout. For each pair we show, among the 60 start poses, a body the labeler reads as the held symbol and then the commanded one with no unrequested behavior in between, choosing the one whose rear height, walking speed, turning or stillness is clearest. Grey cells occur in the corpus, with their bout counts. Below the video, the labeler’s reading of the held symbol and of the commanded one after the switch, across 50 diverse start poses.
| Code sequence | KID ↓ |
|---|---|
| Corpus sequences | 0.204 ± 0.023 |
| Unobserved pairs | 0.418 ± 0.045 |
| Random codes | 0.735 ± 0.083 |
The resulting motion is less realistic than corpus-derived sequences but stays far from the random-code level, and the commanded behaviors remain recognizable pair by pair. Sequences with no support in the data still fall within the distribution of rat motion.
Discussion
Code2Act executes externally defined behavioral vocabularies on a biomechanical rat without the controller ever shaping the vocabulary. Across three vocabularies of different origin, commanded symbols produce realistic motion, keep recognizable behaviors across diverse initial states, recover from physical perturbation, and compose into sequences the data never contained. For neuro-symbolic embodied agents, this lets a higher-level reasoning system keep its own action vocabulary while its symbols are grounded in physical control. For neuroethology, it lets a behavioral vocabulary be evaluated by executing it rather than only describing it: execution reveals which symbols correspond to physically distinguishable behaviors, and different labeling methods can be compared on the same body.
Several limitations remain. Code2Act needs a pretrained imitation model for the body of interest. Fine behavioral distinctions, such as direction, can be lost when the training data holds too few examples of them, so richer or more balanced data should improve symbol-level fidelity. Longer composed sequences may drive the body outside the training distribution and degrade control; integrating planners or language models with mechanisms that correct execution drift is a natural next step. Finally, symbol-level accuracy is measured with the same behavioral labelers that define and recover each vocabulary.
Citation
@inproceedings{bian2026code2act, title = {Discrete Actions for Naturalistic Behavior in Embodied Biomechanical Animal Models}, author = {Bian, Kaiwen and Zhang, Charles Y. and Yang, Yuanjia and Sirbu, Aidan and Leonardis, Eric J. and Richards, Blake A. and {\"O}lveczky, Bence P. and Pereira, Talmo D.}, booktitle = {NeurIPS 2026 Workshop on Neuro-Symbolic Embodied Intelligence (NEmo)}, year = {2026}, url = {https://openreview.net/forum?id=FBEAYXac4j}}Code2Act builds on MIMIC-MJX and track-mjx for the biomechanical rat and imitation pipeline, Keypoint-MoSeq for behavioral syllables, and MuJoCo and Brax for simulation and training.