Description

This project creates a virtual piano experience. A camera tracks your hand, while a sensor glove detects touches for realistic note playing. Imagine playing on any surface - a table, your desk or everywhere you like! This project involves three different elements: the motion capture done by mediapipe in python, the pressure sensors connected to a bela board, running on C++ code and the core application in Processing, providing GUI and connecting the other parts.

Challenges, accomplishment and lessons learned

The biggest challenge we faced in our project was developing and integrating three systems in different programming languages that needed to function in parallel and in real-time. We had to use multiple communication protocols and ensure that the timing was compatible with our requirements. Additionally, we developed and optimized a multi-page application in Processing, which needed to be fast but also easily scalable. It was very stimulating to tackle the technical aspects of development while keeping the goal of making it creative and user-friendly in sight.

Technology

Python, Processing, C++, mediapipe, OpenCV, controlP5, javax.sound.midi, python osc, bela board

Students

Alessandra Moro: GUI and Processing Java code.

Mattia Massimi: piano-related python and Processing Java code.

Marco Furio Colombo: python model, Bela C++ code, sensors, Java midi handling.

Media