Ego

Ego is a project that is meant to explore the idea of human perception, in particular the idea of identity and self consciousness and the way it is distorted and biased..

  • A.A. Year: 2022-23
  • Students

    Matteo Pettenò
    Sebastian Gafencu
    Samuele Del Moro
    Tommaso Gambini

  • Source code: Github
  • Web app: link

Description

We implemented this project as a webapp to make the experience available to everyone, but it can also be imagined in the context of an artistic installation: the user's first approach would be defined by the vision of a cloud of swirling particles, while an undefined drone sound is playing. Once the user is close enough to be detected by our system the cloud will start to slowly morph into a shape resembling its face: the user is now witnessing their identity take shape. A repeating melody would be heard, whose generation is based on the user's face, in particular on their somatic traits and their current mood. The intrigued user might now get even closer, in which case they'll hear the music become more and more intense, in a higher pitch and faster, representing an ever-growing feedback loop of self consciousness. The final phase of Ego will show the user their face being morphed into a new shape, reminiscent of a Rorschach Test, to finally express the idea of the distortion of the self caused by the filtering of external point of view.


Challenges, accomplishment and lessons learned

Multiple challenges came from working on different computers/OS. We had some compatibility issues with Max and some external plugins. Also working with different machines meant we had to find optimal parameters, as a compromise between quality and performance. Initially, communication between browser and external tools was also a problem but we managed to overcome it by implementing a bridge between WebSocket and UDP. Another challenge was to automate graphical parameters and at the same time guarantee a fluid experience. In the end we think we achieved good results in implementing a quite complicated particle system.


Technology

  • WebApp Framework: Svelte
  • Programming languages: TypeScript + GLSL
  • Rendering: Three.js
  • Face detection: MediaPipe Face Mesh Javascript Solution
  • Expression recognition: face-api.js
  • Sound design: Max 8, Tone.js
  • OSC: osc-js
  • Utilities: lygia, gsap, three-msdf-text-utils
  • Others: GPU Computation, Processing
  • CI/CD: GitHub Workflows
  • Hosting: GitHub Pages


Students

Matteo Pettenò:

  • Application core development
  • Three.js development
  • Shader and GPU computation
  • Parameters automation
  • Face and expression detection
  • Face triangulation
  • OSC communication
  • Application architecture
  • Repository administrator

Sebastian Gafencu:

  • Music generation
  • Perlin noise shape generation algorithm
  • Parameters automation
  • Video editing

Samuele Del Moro:

  • Patch Max 8

Tommaso Gambini:

  • Physics-based shape generation algorithm
  • Tone.js

Quotes


Media