Over a year ago, I posted a video where Gemini and I collaborated on an Ableton Live Session. Music AI is fun! Creating voice-native music software is something I'm really passionate about, so, here is Jamcat. Jamcat's concept is a voice controlled, generative music performance tool, designed to inspire live, in the moment jam sessions. It uses LLMs, with both text-to-audio and audio-to-audio models to create music in realtime, entirely hands-free. ➡️ Pipecat PhoneLLM Alpha 1 for session control, running on @modal. ➡️ "StemGen" routing and workload subagent. ➡️ Foundation 1, for melodic stems. ➡️ Various audio models on @fal (like the excellent @ElevenLabs Music) for vocals, drums, textures and one-shots. ➡️ ... and of course @pipecat_ai! Voice commands can be toggled via push-to-talk (or a foot pedal, if you're wielding an axe), or it can listen the entire time (diarization.) It makes some mistakes, or sometimes misses a beat, but most of the mistakes in this video were me not paying attention to the bar counts, like a true fake musician.

Here is an example Pipecat project for trying PhoneLLM Alpha 1. Deploy the model to a Modal endpoint with one click. Client from the video in repo too (with all that sweet sweet terminal-ish aura.) Next up, a TTS that can pronounce @bmervetan's name correctly? 😅