Publication note (2026-08-03). This page previously contained only an internal planning summary rather than an article — an editorial error that went uncorrected for several months. The full article has now been written and published here, preserving the original publication date and URL. The Lab regrets the lapse.
Almost everything AI knows about how humans move, it learned from a small number of datasets, recorded from a small number of people, in a small number of laboratories. Human3.6M — one of the field's foundational collections — was built from eleven professional actors performing scripted everyday scenarios in a single motion capture studio. Models trained on such data learn what "human movement" is from that sample, and treat what falls outside it as unusual. This article explains how these datasets were assembled, uses one concrete movement tradition to show what gets missed, and argues that movement practitioners are the people best positioned to identify the gaps.
Every dataset is an act of selection
A motion capture dataset does not record human movement. It records some movement, by some people, doing some things, chosen by researchers under practical constraints. Those choices then become the implicit definition of normal.
Human3.6M (Ionescu et al., 2014) is among the most widely used. Its scale is what made it valuable — 3.6 million captured poses, hence the name — and its composition is what makes it partial: a small group of professional actors performing directed everyday scenarios in a laboratory, captured with a fixed studio rig.
AMASS (Mahmood et al., 2019) took a different approach, unifying many existing motion capture datasets into a single standardised format on a common body model. This was a substantial technical contribution and is now foundational infrastructure. But unification inherits: AMASS is broad because it aggregates many prior collections, and those collections were themselves assembled under the same laboratory constraints. Standardising them makes them interoperable, not more representative.
HumanAct12 (Guo et al., 2020) organises motion into a set of action categories for training action-conditioned generation — walking, sitting, jumping and similar. The category list itself is a selection: it defines what kinds of movement the resulting models will treat as nameable actions.
None of this is a criticism of the researchers involved. Motion capture is expensive, laboratory-bound and slow; recruiting participants who can perform reliably for hours is genuinely difficult. These constraints are real, and the datasets were built well within them.
The problem is what happens downstream. When a dataset becomes foundational infrastructure, its selections stop being visible as selections. They become the baseline against which everything else is measured — and movement outside that baseline registers not as unrepresented but as unusual.
This pattern is well documented elsewhere in AI. Buolamwini and Gebru (2018) showed that commercial facial analysis systems performed dramatically worse on darker-skinned female faces, and traced the failure to benchmark datasets composed overwhelmingly of lighter-skinned subjects. Noble (2018) documented how search systems reproduce and amplify existing social hierarchies through the data and design choices behind them. The mechanism in motion capture is the same, and it has received considerably less attention.
One concrete case: the ginga
Abstract arguments about representation are easy to nod along to. Here is a specific example.
Capoeira is an Afro-Brazilian practice combining martial art, dance and game, developed by enslaved Africans in Brazil and carrying a documented history of suppression and survival (Assunção, 2005). Its foundational movement is the ginga — a continuous, weight-shifting, triangular rocking pattern that a practitioner maintains as the baseline state from which all other movement emerges.
The ginga is technically specific and demanding. It is continuous rather than discrete: there is no moment where it starts or stops, no rest position between repetitions. It is deceptive by design, in that the visible pattern conceals what the practitioner is preparing to do. It is deeply asymmetric in timing while remaining spatially balanced. And it is relational — its rhythm and amplitude respond continuously to the other player and to the live music setting the tempo.
Now consider how such a movement fares in the standard pipeline. Action-labelled datasets are organised around discrete, nameable actions; the ginga is a sustained state, not an action. Datasets built from single-person laboratory capture cannot record a movement whose form is constituted by responsiveness to another person. And a model trained to recognise or generate "human movement" from these sources will treat a ginga as either an unclassifiable anomaly or the nearest labelled neighbour — likely "walking" or "dancing" — losing precisely what makes it what it is.
The ginga is not an obscure case. Capoeira is practised worldwide, well documented, and historically robust. It is simply not the kind of movement laboratory capture was set up to record.
Multiply that by every movement tradition with continuous rather than discrete structure, or whose form depends on relationship, or whose practitioners were never in a capture studio. The result is not a small gap at the margins. It is a systematic bias toward movement that is individual, discrete, nameable and Western-institutional — because that is what could be recorded.
Practitioners as expert witnesses
This is where movement practitioners have a role that is not decorative.
When a motion model handles a movement tradition badly, the failure is often invisible to the people who built the system. The generated output looks like plausible human movement. Error metrics report reasonable numbers. Nothing signals that something has been flattened, because the flattening is only detectable by someone who knows what the movement is supposed to be.
A capoeirista watching a generated ginga will see immediately that the weight is wrong, the timing is symmetrical when it should not be, the relational quality is absent. That perception is a form of expertise, and it functions as evidence about the system that no automatic metric currently produces.
Framing practitioners this way — as expert witnesses to what a system cannot represent — puts perceptual training in a specific relation to technical work. It is not consultation about user requirements, and it is not decorative cultural input. It is testimony about the limits of an instrument, given by people trained to perceive what the instrument does not capture. Sheets-Johnstone (2011) argues that movement is itself a mode of thinking and knowing rather than merely something the body executes; on that view, the practitioner's perception is not a supplementary opinion about the data but a distinct form of knowledge about the phenomenon.
Why this matters concretely
For clinicians. Movement AI is increasingly used to assess whether someone is moving correctly. If the underlying model's sense of normal comes from a narrow sample, then healthy variation — cultural movement habits, body types and histories outside that sample — can register as deviation. In a clinical context, that is not an abstract concern.
For dataset designers. The fix is not only more data but differently sourced data, with the composition documented. Whose movement, recorded where, under what constraints, and what could not be recorded. Datasets that state their limits let downstream users reason about them; datasets that do not, hide the selection inside the infrastructure.
For anyone using motion AI to evaluate movement. The practical question to ask of any such system is simple and rarely asked: whose movement did this learn from, and is the movement I am evaluating like theirs? If the answer is no, the system's confidence is not evidence.
The short version
Every dataset encodes a decision about whose movement counts as movement. Those decisions were made under real constraints and mostly without ill intent, but they are now buried in infrastructure that treats them as neutral. Making them visible again is ordinary maintenance work — and the people best equipped to say what is missing are the ones who have spent their lives learning to perceive it.
Continue through the archive
For connected context, read Weekly News Digest — Motion Representation & Generative Video (2026-04-13) and Community Scout Digest — 8–14 April 2026.
References
Assunção, M. R. (2005). Capoeira: The history of an Afro-Brazilian martial art. Routledge.
Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html
Guo, C., Zuo, X., Wang, S., Zou, S., Sun, Q., Deng, A., Gong, M., & Cheng, L. (2020). Action2Motion: Conditioned generation of 3D human motions. Proceedings of the 28th ACM International Conference on Multimedia, 2021–2029. https://doi.org/10.1145/3394171.3413635
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014). Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7), 1325–1339. https://doi.org/10.1109/TPAMI.2013.248
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., & Black, M. J. (2019). AMASS: Archive of motion capture as surface shapes. Proceedings of the IEEE/CVF International Conference on Computer Vision, 5442–5451. https://doi.org/10.1109/ICCV.2019.00554
Noble, S. U. (2018). Algorithms of oppression: How search engines reinforce racism. NYU Press.
Sheets-Johnstone, M. (2011). The primacy of movement (2nd ed.). John Benjamins. https://doi.org/10.1075/aicr.82