Description

A description of what the project is about, what you want to convey, how do you envision it (an artistic installation? a webapp?) and what it actually does, so what is the user experience. A single webcam reads a visitor's facial expression and hand gestures in real time. Those signals decide which painting from a 4,000+ work archive appears, how its pixels dissolve into a living field of particles, and what generative soundscape fills the space around it — no screen, no wall text, no controller. It's designed for a gallery or museum context: a way to make a painting notice you back, and to offer a small emotional journey through art history instead of a wall label. Gesture in — interaction out. Swipes browse paintings and styles; thumbs up/down teach the system your taste (persisted across sessions); an "OK" sign toggles a mode where your hand stirs the painting into a particle cloud, scattering it on a push and pulling it back together on a retreat. Emotion in — painting out. A 7-class facial emotion model (with continuous valence/arousal) is projected into the WikiArt Emotions dataset's 20-dimensional affective space, and the closest-matching, still-fresh, aesthetically-strong painting is selected via cosine similarity. Everything out — sound + visuals. The painting's style picks one of five fully distinct synthesized soundscapes; gestures and mood continuously shape both the sound and the particle system live, so the image and the audio move as one.

Challenges, accomplishment and lessons learned

What kind of challenges did you run into for this projects, what kind of accomplishment are you proud of and and what did you learn during the project? Few lines for each question. The hardest part was making the 4 distinct tools that we used work as a coherent, real time system over OSC. These are computer vision, a PyTorch emotion model, SuperCollider for sound, and Processing for visuals. Designing generative audio that doesn't feel chaotic was another challenge. Translating the facial emotion into the emotional space of the WikiArt dataset, while staying coherent, was also difficult. Accomplishments: A fully working installation where a face and a hand simultaneously drive image selection, a generative soundscape, and reactive visuals live. The sound engine is synthesis-only, and the recommender is able to keep the experience fresh and even remembers a visitor's taste across sessions. Lessons learned: We learned how to better design a real time multi-stage system. We also learned that continuous signals (like valence/arousal and windowed motion) make an installation feel much more more alive than discrete on/off states, and that a huge share of the success lies in mapping and tuning, not in the models themselves. On the audio side, we saw that the line between generative and random is hard to draw and control but we learned how to use this fine line between them as a deliberate design choice.

Technology

just a plain list of the main technology you used, which include libraries, coding languages, concepts, etc. PyTorch (ResNet50 emotion model with valence/arousal) - OpenCV - MediaPipe (hand tracking) - CLIP / HuggingFace Transformers (visual embeddings) - NumPy / pandas · WikiArt Emotions dataset - SuperCollider (real-time generative synthesis) - Processing + oscP5 (particle-system visuals) - OSC via python-osc - Git LFS. Concepts: real-time computer vision, facial emotion recognition, gesture recognition, content-based retrieval and cosine similarity, embedding-based diversity, audio–visual synchronization, generative sound design, particle systems.

Students

Onur Arikan: emotion recognition + processing visuals and particle systems + gesture recognition
Thomas Guffroy: painting selection + emotion matching
Margarita Makurina: SuperCollider sound design + Processing visuals + particle system
Umut Özer: gesture recognition and integration

Media