Description

Signs in music is a site that aims to create an emotional bond between hearing and deaf people. The software contains a brief tutorial on how to reproduce some gestures of the American Sign Language. The sign done by the user is then recognized by a supervised trained neural network and a song with a title related to the done gesture is searched in the Spotify database. Some features of this song are extracted and mapped to a particle system whose goal is to visualize the selected song, which is also played synchronously with the particle system. In this way, a hearing user needs to learn the sign language to use the site, while a deaf user can achieve a sort of visualization of the song watching to the graphic interface.

Challenges, accomplishment and lessons learned

Challenges:
The main challenge was the compatibility between the site and the particle system, written in HTML and JavaScript, and the neural network that recognizes the gestures, coded in Python. Another big challenge was to find the best way to visualize the selected song. We tried to map the audio parameters of the song to the graphic parameters of the particle system in the most meaningful way.
Accomplishments:
For sure, one of the best accomplishments achieved was to be able to read an external library written by another user and to modify its source code based on our project requirements. In fact, the simple use of the library was not enough to achieve the wanted result.
Lessons Learned:
We learned how to divide the work into 3 main parts and how to write the code in a way such that the integration with the other two parts written by the other members could be as easy as possible.

Technology

• Tensorflow and OpenCV were used for the sign recognition model
• particles.js was the starting point of the particle system
• Flask for the communication between the site and the gesture recognition system
• TouchDesigner as the graphic engine

Students

Alice Sironi: interactive site development with tutorial and project explanation.

Cecilia Raho: gesture recognition model implementation.

Lelio Casale: song research, song features extraction and computation,particle system development.

Media