This guide is for creative technologists building a live body→signal→visual pipeline in Python or TouchDesigner. It provides a repeatable audit for a common failure: the system tracks a performer accurately but erases how the movement is organised. The workflow was checked against current documentation and research available in October 2026; it is an evaluation scaffold, not a validated clinical or biomechanical instrument.

What the audit measures

Recent humanoid-control work separates positional tracking from five expressive characteristics: amplitude, energy, body expansion, body lean and pelvis height (Bao et al., 2026). Those measures do not exhaust expression, but they are observable enough to help diagnose a live visual system. The question is not “does the avatar look artistic?” It is: which dynamic and postural relations survive sensing, smoothing, mapping and rendering?

Use a short reference phrase containing stillness, a level change, a reach, a directional lean and a quick accent. Capture at least three repetitions. Obtain informed consent, minimise retained video, and document the sensor, frame rate, coordinate convention and performer. A pipeline without provenance cannot support meaningful comparison.

Stage 1: body to timestamped signal

Start with the simplest sensor that answers the artistic question. A calibrated RGB camera may be enough for a single frontal performer. Depth or inertial sensing can improve robustness under occlusion, but every additional stream adds synchronisation and failure modes.

Every frame should become a record rather than a loose array:

Frame {
  capture_time_ns,
  receive_time_ns,
  body_id,
  confidence[joint],
  position[joint] in named coordinates,
  calibration_id
}

Keep capture time separate from arrival time. Network and inference delay can reorder frames, so arrival time is not movement time. Log dropped frames; do not silently repeat the previous pose. The archive's provenance guide explains why these fields matter.

For a first Python prototype, calculate features from a normalised skeleton while retaining the raw record:

scale = distance(left_shoulder, right_shoulder)
root = midpoint(left_hip, right_hip)
q = (joints - root) / max(scale, 1e-6)

amplitude = percentile(speed(q), 90)
energy_proxy = mean(square(speed(q)))
expansion = mean(distance(q, q[pelvis]))
lean = angle(torso_axis(q), vertical_axis)
height = q[pelvis].y

These are operational proxies, not direct measurements of effort, affect or intention. Name them accordingly in the interface.

Stage 2: signal to a stable feature stream

Run two branches in parallel. The reference branch records minimally processed features for later comparison. The live branch applies only the smoothing needed for the visual response.

In TouchDesigner, a compact graph is:

Sensor In → Timestamp/Confidence DAT → Coordinate Normalise CHOP
          → Select Joints CHOP → Feature Math CHOP
          → Filter CHOP → Null CHOP → Visual Mapping
                         ↘ Audit Recorder

Do not choose a filter only because the trace looks clean. Heavy low-pass filtering reduces peak amplitude and energy, delays accents and makes changes of direction appear rounded. Test at least three settings: no smoothing, current smoothing and a deliberately heavier setting. Record end-to-end delay for each.

Measure latency by introducing a visible event captured by both ends of the system—for example, an LED entering the camera frame and a corresponding display change—then inspect high-frame-rate video. Report median and 95th-percentile motion-to-photon delay. A low average can hide disruptive spikes. For tightly coupled performance, the perceptual effect of delay depends on task and expectation; there is no universal “real-time” threshold.

Stage 3: map features without collapsing them

One-to-one mappings are easiest to audit. Map pelvis height to vertical composition, expansion to visual spread, lean to field orientation, amplitude to spatial displacement and the energy proxy to change rate. Begin without cross-coupling. Only after each channel is legible should one feature influence multiple visual parameters.

Normalise with a performer-specific calibration range rather than a fixed universal body norm:

u = clamp((x - p05_calibration) / (p95_calibration - p05_calibration), 0, 1)

Keep the percentiles and calibration phrase in the session record. Recalibrate when clothing, camera placement, mobility condition or intended movement range changes. This is not a claim that every body needs to fit a standard; it is a way to preserve the range the performer actually offers.

In Chinese dance discourse, shenyun (身韵) refers to more than body shape: it points toward the patterned vitality, breath, rhythm and cultural organisation through which movement becomes legible. The five measures cannot capture 身韵. Their value is diagnostic: if even gross expansion or lean disappears, the pipeline is certainly not preserving subtler organisation.

Stage 4: run the shadow audit

Render two hidden recordings for each take:

  • Shadow A: the minimally processed reference drives a neutral diagnostic visual.
  • Shadow B: the production feature stream drives the same neutral visual.

Compare them before evaluating the final artwork. For each feature, calculate peak ratio, timing offset and shape similarity:

peak_ratio = peak(B) / peak(A)
timing_offset_ms = time_of_peak(B) - time_of_peak(A)
shape_error = mean(abs(normalise(B) - normalise(A)))

Then review the recordings with the performer. Ask where the response arrived too early or late, which contrast disappeared, and whether the visual encouraged an unintended change in performance. Quantitative measures locate a transformation; first-person review helps interpret whether it matters.

The review should distinguish three failures:

  1. Sensing failure: low confidence, occlusion or identity swap corrupts the body record.
  2. processing failure: smoothing or normalisation suppresses a dynamic distinction.
  3. mapping failure: the feature survives but the visual makes it illegible.

Fixing the wrong layer wastes time. A brighter shader cannot restore an accent already removed by filtering.

Safety, privacy and honest limitations

Body data can reveal identity, disability, health-related patterns or intimate behaviour. Store the minimum, separate identity from session data, set a deletion date and explain whether recordings will train a model. Do not infer emotion, diagnosis or personality from these five features.

Two-dimensional pose loses depth and is viewpoint-dependent. Skeleton scale normalisation can erase meaningful travel through space. Confidence scores are model-specific and do not guarantee correctness. The measures also encode a particular analytic tradition; they should sit beside the performer's account, not overrule it.

Finally, expressive fidelity is relational. A deliberately delayed visual may be artistically right even when the audit calls it a timing deviation. The audit supplies evidence for a decision; it does not make the decision.

Release checklist

  • Sensor, version, coordinates and calibration are recorded.
  • Capture and arrival timestamps are distinct.
  • Raw/reference and production branches can be compared.
  • Median and tail latency are measured.
  • Five proxy features are labelled as proxies.
  • The performer reviews the result.
  • Consent, retention and training-use terms are explicit.

This workflow complements the Lab's guide to latency-aware interaction and analysis of movement between frames.

References

  • Bao, L., Zhu, H., Pan, Y., Wang, J., Yu, J., Peng, T., Jia, Y., Kanoulas, D., & Zhou, C. (2026). Beyond kinematic tracking: Preserving expressive characteristics in humanoid whole-body control. arXiv. https://arxiv.org/abs/2610.19393
  • Fdili Alaoui, S., Bevilacqua, F., & Jacquemin, C. (2015). Interactive visuals as metaphors for dance movement qualities. ACM Transactions on Interactive Intelligent Systems, 5(3), Article 13. https://doi.org/10.1145/2738219
  • W3C. (2025). High Resolution Time Level 3. https://www.w3.org/TR/hr-time-3/