Description

Motion Ray is an innovative, AI-driven interactive system that transforms a dancer's live movements into dynamic DMX commands for stage lights. It creates an immersive fusion of human performance and automated visuals, reimagining the relationship between performer and stage.

Instead of relying on rigid, pre-programmed lighting cues, Motion Ray treats the dancer as the conductor of the lighting console. By leveraging real-time pose detection, audio analysis, and a Conditional Variational Autoencoder (CVAE), the system enables the lights to respond fluidly and continuously to gestures. The result is a theatrical experience where fixtures act as an interpretive extension of the performer’s style, choreography, and the emotion that is intended to be conveyed.

Challenges, accomplishment and lessons learned

During Motion Ray’s development, the biggest challenges were establishing a clear conceptual framework and managing a complex system architecture. Conceptually, we needed reliable mappings from abstract performance inputs to lighting outputs—e.g., audio amplitude → brightness, performer velocity → movement speed, and extracting color palettes from uploaded images. Ensuring spatial consistency between a 2D camera feed and a physical 3D stage without prior geometry required an adaptive stage-sector identification system based on homography.

Technically, achieving true real‑time behavior was difficult. Our asynchronous pipeline initially caused desynchronization: OSC dispatch lagged behind YOLO detection and CVAE generation, producing a “time‑machine” effect with visual jumps and FIFO bottlenecks. We resolved this by redesigning thread coordination and state exchange.

We turned these issues into strengths. Implementing thread-safe deques and explicit synchronization (semaphores and threading.Event) eliminated latency buildup and enabled smooth interpolation of 15-frame sequences for pan, tilt, and color. On the AI side, training a Conditional VAE on sector-labeled lighting scenes allowed dynamic, spatially coherent lighting generations responsive to the performer’s live position. The result is a stable, low-latency pipeline ready for theatrical testing.

Key lessons learned: prioritize core features and standard protocols (OSC, QLC+) over monolithic custom builds; enforce strict concurrency and state-management policies (bounded queues, synchronization, frame discard or interpolation rules) for real‑time reliability; and design platform-agnostic code (relative paths, driver abstractions) to support collaborative development and cross‑OS deployment.

Technology

Python, PyTorch, Conditional VAE, Ultralytics'YOLOv11, librosa, OpenCV, React (GUI), DMX, OSC

Students

Luigi Biasi: VAE model, GUI, multithreading
Julien Boidevaix: movements boundaries, 3D environment research
Claudio Costantini: YOLO, stage sectors and positioning, dataset
Alinda Gercek: audio to brightness, color conditioning, music velocity classification

Media