Description

Hypnagogia is an interactive audio-visual installation that invites the user to step into the liminal space between wakefulness and sleep, the hypnagogic state. The user enters a fully navigable 3D Victorian living room in first-person rendered in open- Frameworks. A golden gramophone sits in the corner and an enchanted book rests on the main table. The user picks up the book and types (or dictates via speech-to-text by clicking a but- ton) a description of a dream they have had. Once submitted, the text is sent to a Python bridge that runs a transformer-based emotion classifier and outputs seven normalized scores (joy, sadness, anger, fear, disgust, surprise and neutrality). Those scores are streamed over OSC to both the C++ visual system and a SuperCollider audio engine, creating a shared emotional control layer that keeps image and sound synchronized with low latency. Visually, the front wall becomes a reactive surface where a GPU particle system adjusts gravity, turbulence, flow speed and color palette to the detected emotions, turning affect into motion and light. Soni- cally, SuperCollider generates a continuously evolving composition that crossfades sample layers, applies emotion-driven effects (pitch gravity, tremolo, granular flutter, bit-crushing and stutter) and performs binaural spatialization so the sound seems anchored to the gramophone and shifts with the listener’s position and orientation. The system relies on openFrameworks features (3D rendering, camera, collision, model loading via ofxAssimpModelLoader), OSC messages (ofxOsc and python-osc), GLSL and FBO ping-pong for particle simulation and custom binaural tech- niques (interaural time/level differences, head-shadow filtering, pinna/elevation cues, distance attenuation). Conceptually, the project explores the hypnagogic threshold between wakefulness and sleep by treating a dream as a living signal that can be analyzed, mapped and re-expressed as space, motion and sound. Interactable objects include the start button for voice input, the dream book for typing, the gramophone for music playback, the piano for performance, the sofa and Victorian chair for sitting, the bookshelf/Dream Archive for browsing saved dreams, the cat for a meow, the medicine/pill bottle for resetting the experience and the large clock that reveals the current time when targeted. The aim of the project is not to represent dreams in a literal or narrative sense, but to allow the user to re-experience the sensations associated with them. While fragments of dreams are often remembered as images or events, the emotional atmosphere in which they were lived tends to fade more quickly. Hypnagogia therefore focuses on reconstructing the affective dimension of dreaming, translating the emotional residue of a dream into immersive audiovisual stimuli so that the user can inhabit, once again, the feelings that may have accompanied that nocturnal moment.

Challenges, accomplishment and lessons learned

Challenges

Converting the abstract emotional states into logical visual and aural behaviors was the project’s most difficult task. The goal was to accurately capture the qualitative nature of each emotional state using particles and music, rather than just mapping numerical emotion values to param- eters. In order to achieve this, it was necessary to create particle dynamics that could percep- tually convey tension, lightness, turbulence, or stillness while also creating a music engine that was adaptable enough to capture minute emotional details without becoming disorganized or monotonous. In order to maintain an overall aesthetic unity, each emotion had to affect several layers of the system, including movement, color, density, harmony, timbre and spatial perception. The room’s design was equally important. The environment had to serve as the most appropriate spatial and atmospheric context for the project’s goal, not just a container for visual effects. The Victorian living room was thoughtfully designed to encourage closeness, reflection and story immersion. Its lighting, materials, scale and spatial arrangement were all constantly improved to make the music and particles feel naturally integrated into the room rather than superimposed. One of the main conceptual and implementation challenges was building a space that could accommodate and magnify the emotional changes while still being technically sound in real time.

Accomplishments

One of the main achievements was the effective use of closely related visual and aural responses to depict dream emotions. Both the particle system and the generative music engine are powered by emotion analysis, which results in noticeable changes to the particle behavior, color dynamics, musical texture and spatialization that mirror the narrative’s identified affective profile. From dream input to synchronized audio-visual feedback, the entire pipeline functions with minimal latency, maintaining immediacy and enhancing the perception of responsiveness. The development of a completely navigable 3D room with a game-like degree of presence and interaction was another noteworthy accomplishment. An experience that feels more like a co- hesive, livable space than a merely demonstrative prototype is produced by the environment’s integration of various interactive elements and support for real-time first-person exploration.

Lessons Learned

For complex creative systems, Open Sound Control (OSC) has shown itself to be a useful inte- gration layer. Iteration speed was increased, component coupling was decreased and independent development and testing were made possible by keeping emotion analysis, audio synthesis and visual rendering distinct processes. Both the expressive possibilities and the real-world difficulties of shader-driven workflows were brought to light by GPU-based particle systems. In contrast to CPU-based code, where inspec- tion tools are scarce and failures might go undetected, the ping-pong FBO technique provided an effective and scalable method for real-time simulation, but it also necessitated a different approach to debugging. Binaural spatialization highlighted the value of perceptual cues over rigorous physical accuracy in audio. The perceived realism of space was enhanced by head-shadow filtering, interaural time differences and interaural level differences more so than by traditional stereo panning alone. Lastly, inherent ambiguity was introduced by emotion analysis. Designing audio-visual mappings that are stable, expressive and aesthetically coherent even when confronted with unexpected or mixed emotion vectors is crucial because model outputs don’t always match human interpreta- tion.

Technology

• C++ / openFrameworks - 3D rendering, scene management, first-person camera, collision detection
• GLSL shaders - GPU ping-pong particle system (vertex + fragment)
• ofxAssimpModelLoader - loading and animating .glb 3D models
• ofxOsc - Open Sound Control messaging between all components
• ofxFastParticleSystem - GPU-accelerated particle framework
• SuperCollider (sclang + scsynth) - generative audio synthesis, emotion-reactive music engine, binaural spatializer
• Python 3 - bridge for speech-to-text and emotion analysis
• Hugging Face Transformers - NLP emotion classification model
• python-osc - OSC communication from Python
• Binaural audio techniques - ITD, ILD, head-shadow filtering, pinna/elevation modelling, distance attenuation
• OpenGL - depth testing, FBO render targets, custom shaders

Students

Bianco Lorenzo: Graphics: conceived and built the oniric room environment, defined its visual identity and atmosphere and implemented all lighting and materials.
Callandrone Pietro: Audio: designed and implemented the generative emotion-reactive music engine, the basic room functionalities and communication between all modules.
De Lellis Flavio: Particle system: developed the GPU-based wall particle system,the emotion-to-simulation mapping and the Python OSC bridge..

Media