Description

BARD is a creative AI system that uses music as the starting point for generating a structured narrative and a visual atmosphere. We envision it as an immersive artistic installation for different kinds of performances, such as classical concerts, jazz sessions, electronic live sets, theatre pieces or experimental audiovisual events. The generated text and images are projected into the room where the performance takes place: on walls, curtains, scenographic elements or multiple surfaces, surrounding the audience with a story that appears to emerge from the music itself. The experience starts from a musical input: the system analyzes the audio, extracts emotional and musical cues, translates them into dramatic directions, generates a coherent story scene by scene, and associates each scene with visual elements. During the performance, fragments of text and image-based particles appear in space, creating the feeling of a living bedtime story generated from the expressive qualities of the music. The goal is not to describe the track literally, but to turn musical evolution into narrative progression, atmosphere and visual motion.

Challenges, accomplishment and lessons learned

One of the main challenges was connecting several complex systems into a single pipeline: cloud AI models, structured prompts, image generation, local visualization, audio timing and OSC communication. Another difficulty was translating music into narrative meaning maintaining consistency throughout the performance, regardless of its duration, while also dealing with latency, API quotas and model unpredictability. We are proud of having built a complete end-to-end pipeline, from audio input to music analysis, story generation, image generation and immersive visual output. A major accomplishment was the structured JSON-based architecture, which makes generative AI more controllable and allows each story fragment to carry text, mood, timing, visual prompts and continuity information. We learned that working with generative models is not only about prompting, but also about designing constraints, data structures and fallback logic. The project also taught us how important synchronization is in audiovisual systems, and how to balance technical reliability with narrative and visual expressiveness.

Technology

Python 3.11 back-end; Google Cloud Platform and Vertex AI for Gemini 2.5 Flash and Imagen 4 Fast; JSON schemas for structured model outputs; audio chunking; prompt engineering and narrative state management; Pillow, rembg, ONNX Runtime and U²-Net for image post-processing; OSC for synchronization between Python and the visual layer; Processing 4 with oscP5 for animated text, mood atmospheres and particle-based image visualization; Docker and Docker Compose for a reproducible Python environment; optional FastAPI/Uvicorn and Google Cloud Storage for future deployment.

Students

Filippo Longhi : back-end, environment setup
Anna Chiara Melioli : front-end, processing
Elena Tortorella: audio analysis, processing
Marco Porcella: story LLM, presentation materials

Media