Description

A description of what the project is about, what you want to convey, how do you envision it (an artistic installation? a webapp?) and what it actually does, so what is the user experience BrainViewer is envisioned as an interactive artistic installation. The project aims to convey a real-time, visual connection between musical stimulation, physical movement, and neural activity. When a user approaches the installation, they interact with a single web screen. From this interface, the visitor can select and play a musical track. As the user listens, they look at a 3D brain rendered as a glowing cloud of points, which represents individual neurons. The user experience is built around two main real-time reactions:
• Audio Reactivity: The 3D brain dynamically lights up and activates, lobe by lobe, based on the specific musical content and qualities of the chosen song.
• Physical Tracking: The installation acts like a digital mirror. Using a webcam, the brain tracks the user's physical presence and mimics their head movements.
Throughout the experience, the user can simply watch the brain react or read the on-screen information explaining which brain area is currently dominant.

Challenges, accomplishment and lessons learned

What kind of challenges did you run into for this projects, what kind of accomplishment are you proud of and and what did you learn during the project? Few lines for each question.

Challenges
The most difficult part of the project was managing the 3D activation of the TouchDesigner model and designing the underlying mapping. The goal was to achieve a visual effect that was aesthetically striking while remaining as faithful as possible to how a real human brain actually reacts to audio stimuli. To make this work, it required extensive parameter tuning, carefully balancing and summing the general, global features of the track with its specific, moment-by- moment acoustic variations.

Accomplishments
We are particularly proud of the rate-coding rendering model implemented in TouchDesigner. Instead of simply turning up the overall brightness of the active brain regions, the digital neurons simulate real action potentials using statistical probability and refractory periods. This creates a highly organic, lifelike swarm effect that dynamically responds to the music without ever completely saturating the screen.

Lessons Learned
We learned how to integrate and utilize new tools for audio feature extraction to accurately analyze the musical content. Furthermore, we gained valuable experience in leveraging a powerful software like TouchDesigner to handle the complex, real-time 3D rendering and visual mapping.

Technology

Technology: just a plain list of the main technology you used, which include libraries, coding languages, concepts, etc. • Languages & Runtimes: Python, JavaScript • Frameworks & Core Software: TouchDesigner, FastAPI, React, Vite • Libraries & SDKs: Essentia, librosa, TensorFlow, MediaPipe (Face Detection & Face Landmarker), OpenCV • Protocols & Networking: OSC (Open Sound Control via UDP), WebRTC, WebSockets, HTTP / REST • Concepts & Techniques: ◦ Audio feature extraction and timeline dynamics analysis ◦ Mood estimation (valence and arousal modeling) ◦ Real-time 3D point-cloud rendering and geometry instancing ◦ Computer vision head-pose tracking and mirror interaction

Students

:
Alessandro Antonio Lillo: TouchDesigner model implementation and mapping.
Filippo Garofalo: backend and music analysis
Lorenzo Nobile: face tracking, emotion estimation and frontend
Salvatore Pantusa: face tracking, emotion estimation and frontend

Media