C-T2M: Ablations

Is the headline comparison fair? Neutral encoder evaluation

The main results table on the home page uses Kimodo's own retrieval encoder (TMR-SOMA-RP), which is what Kimodo's authors report numbers with. That encoder was trained on Kimodo's training distribution, so motions that look like Kimodo's training data score higher on retrieval, regardless of whether they are actually closer to the target caption. This gives Kimodo a built-in home-field advantage that no other system can match. We initially planned to report only those numbers because our proposal targeted Kimodo's benchmark, but the bias means they overstate the gap.

To remove this bias, we re-evaluated both systems with a neutral encoder: TMR-petrovich, the standard text-to-motion encoder used in the broader T2M literature, trained on HumanML3D, a different dataset. Now neither system is at home: any remaining differences reflect actual text-to-motion retrievability, not which model the encoder was trained against. The retrieval protocol is identical for both systems.

The result, shown below: the ~30-point R@3 gap on Kimodo's encoder (58 vs. 88) collapses to a ~2-point gap on the neutral one (9.5 vs. 11.6). Both systems also track the BONES ground truth closely. Most of the apparent quality gap in the headline table was encoder bias, not a real difference in motion quality.

And why does Kimodo edge slightly above the ground truth (R@3 11.6 vs. 11.1)? R@k does not measure motion quality, it measures how easily an encoder can map a motion back to its caption. Real captured motion is messy: a person "walking forward" also breathes, shifts weight, glances around. The neutral encoder, trained on a different dataset (HumanML3D), treats those idiosyncrasies as noise that pushes the motion away from the canonical "walking forward" cluster in its embedding space. Kimodo's generated motion, by contrast, is smoother and closer to the Platonic version of the caption, which is friendlier to retrieval even though it carries less information than the real thing. In other words, Kimodo has learned to produce motion that is more retrieval-friendly than the BONES ground truth itself, an interesting side-effect of training a generative model on caption-aligned data. The takeaway is not that generation beats reality, but that all three systems sit in the same retrievability regime on a neutral encoder, with our system within ~2 R@3 points of the ground truth.

R@k: text-to-motion R-Precision at top-k, the fraction of queries whose correct motion is retrieved within the top k (higher is better). MedR: median rank of the correct motion (lower is better), shown in the chart tooltips. The in-distribution HumanML3D test ceiling (R@1 15.3 / R@10 57.1) is omitted from the chart axis so the three close-to-GT systems stay legible.

Could we just inject waypoints into the body GPT?

Before settling on the decoupled controller, we tried three end-to-end ways of feeding waypoints into the body GPT itself.

View table

How aggressively should the VQ-VAE compress time?

The body VQ-VAE acts as a temporal autoencoder: it compresses a motion clip into a shorter sequence of discrete tokens, and the autoregressive language model is then trained to predict those tokens. Higher temporal compression yields fewer tokens per clip, which lengthens the effective context the language model can attend to and reduces its computational cost, at the price of coarser temporal resolution at the decoder. We study four compression rates: 2x, 4x, 8x and 16x, where a rate of kx means that each token represents k frames of motion. All runs share the same training recipe (codebook of 2048 entries, batch size 512, learning rate 2e-4, window of 64 frames) and are compared at a fixed budget of 100k training iterations.

A rate of 4x gives the lowest reconstruction error and the highest per-clip code diversity, while remaining within the codebook utilisation range of 2x and 8x. We adopt 4x as the body VQ-VAE for the decoupled pipeline. 2x is retained for variants in which the language model benefits from a denser token stream. At 16x the model collapses on reconstruction: with only four tokens per 64-frame clip, the encoder runs out of latent slots before it can capture motion detail.

Is the VQ-VAE creating a bottleneck?

We compare two VQ-VAE variants, first on their own (encode a ground-truth clip, decode it back) and then plugged into the body GPT end-to-end. The first answer tells us the ceiling; the second tells us what actually makes it through to the generated motion.

Part 1 · The VQ-VAE on its own

We test two variants of the VQ-VAE in isolation: the single-codebook baseline (one quantizer, one index per frame) and a residual stack of four quantizers, each encoding the error left by the previous one. Highlighted in red are the tokens we are asking the gpt to predict per each motion token.

Reconstruction quality (VQ-VAE ceiling)

View table

The residual VQ-VAE wins on its own: ~9 points more R@3 on Overview, ~12 on Timeline-multi, half the FID, and ~6 cm/s less foot-skate. The skate gap to the ground truth (~20 vs. ~2) is what motivates the foot-contact re-projection step in our pipeline.

Part 2 · Paired with the body GPT (end-to-end)

Does that advantage carry over once we plug each VQ-VAE into the body GPT? Quadrupling the tokens per frame makes generation harder, and several training recipes for the residual setup did not converge cleanly. The RVQ-GPT we did manage to train underperforms VQ-GPT end-to-end on every metric. Making the residual pay off downstream likely needs a more elaborate design, possibly hierarchical.

Takeaway. A better VQ-VAE does not automatically give a better generative model. The residual stack wins as a reconstructor but loses as a token target. We keep the single VQ-VAE as our deployed body decoder.

Is the decoding strategy important?

We compare five token-sampling strategies on 25 held-out captions (5 samples each, 200-frame generations).

Multinomial dominates every axis and is the only decoder that never stands still. Sharpened samplers (top-p + repetition penalty, top-p, top-k) look smooth on locomotion captions but collapse to a held pose on roughly a third, a quarter and an eighth of the captions respectively.

View table

Qualitative comparison

Same skeleton, five decoders, five held-out captions. Multinomial (ours) is drawn in red with a star; the others turn grey at the frame they emit end-of-sequence.

Sliding-window decode trade-off

At each step the body GPT keeps only the last SW tokens of context. A larger SW lets the model produce longer motions but reaches end-of-sequence sooner; a smaller one keeps generating but starts forgetting its own context. We adopt SW = 25: roughly 290 frames (~10 s) before EOS while limb activity stays near its peak (~21 mm/frame). Multinomial is the only sampler whose limb activity is insensitive to window size; top-p and top-k both lose ~20 % when the window shrinks below 15.

Same metrics as above, swept along SW with the decoder fixed to Multinomial. Each row aggregates 200 generations (10 captions × 20 samples).

Small windows produce long but wandering motion (20-35 m in 6.7 s, much of it spurious as the model forgets its own context). The default SW = 50 is too tight: only ~80 frames before EOS. SW = 25 hits the sweet spot, with limb activity at its peak and the stuck count at its minimum.