A reliable body-to-visual prototype needs only four stages: capture a pose or landmark stream, derive a continuous feature, smooth and timestamp it, then map it to a visual parameter. The important engineering choice is to expose latency and uncertainty rather than hide them behind a spectacular effect.
1. Capture a stream
MediaPipe Pose, webcam landmarks or an inertial sensor can supply the first stream. In a TouchDesigner Python Script DAT, keep the callback small and write only the values the visual layer needs:
# input: landmarks with x, y, z, visibility
shoulder = landmarks[11]
hip = landmarks[23]
dy = shoulder.y - hip.y
speed = (dy - op('null_prev')[0]) / max(me.time.rate, 1)
op('null_prev')[0] = dy
return dy, speed
The sketch is intentionally incomplete: production code should handle missing landmarks, camera mirroring and timestamps. A moving average or one-pole low-pass filter can reduce jitter, but every filter adds delay.
2. Extract a continuous feature
Avoid turning every threshold crossing into a trigger. Useful continuous features include centre-of-mass displacement, shoulder–hip angle, hand distance, velocity and a short-term energy estimate. Normalise against a calibration pose so the same visual range remains usable for different bodies. Store raw and normalised values separately; otherwise a calibration mistake becomes impossible to diagnose.
3. Map feature to image
Use a CHOP or Python mapping with explicit bounds. Map velocity to particle emission, direction to a field angle and accumulated energy to opacity. Clamp outliers, but log how often clamping occurs. A visual that is beautiful only because 40% of the signal is clipped is not responsive to the body; it is responsive to a hidden limiter.
4. Measure the loop
Record capture time, inference time, queue time, render time and display time. Test a slow reach, a fast turn and stillness. Report median and worst-case latency, not only a best-case demo. StreamDiffusion can reduce generation delay; AnimateDiff and ControlNet can add temporal structure or pose conditioning, but neither removes network, inference or display latency.
The practical question is whether the visual remains coupled to the movement. If a person changes direction and the image responds after the next phrase has begun, the system is sequential, not continuous. Design a degraded mode—simple shaders or cached motion—so the visual remains legible when generation stalls.
Honest limitations
Camera landmarks are estimates. They lose hidden depth, pressure, internal preparation and often the distinction between a quiet body and a poorly tracked one. A model-generated frame can be temporally smooth while still being wrong about support or contact. Validate important claims with a second signal or practitioner review. The archive’s signal-selection guide and interaction-aware training note provide useful checks.
References
Guo, Y., Yang, C., Rao, A., Wang, M., Yang, Z., Qiu, J., Zhang, C., & Lin, B. (2023). AnimateDiff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv:2307.04725. https://arxiv.org/abs/2307.04725
Kodaira, H., Xu, P., Liao, Y., et al. (2023). StreamDiffusion: A pipeline-level solution for real-time interactive generation. arXiv:2312.12491. https://arxiv.org/abs/2312.12491
Zhang, L., Rao, A., & Agrawala, M. (2023). Adding conditional control to text-to-image diffusion models. Proceedings of ICCV, 3836–3847. https://doi.org/10.1109/ICCV51070.2023.00355
Google. (2025). MediaPipe pose landmarker. https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker