LiveSynth

A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument

Kyungsu Kim, Yejin Kim, and Kyogu Lee

Music and Audio Research Group, Seoul National University

LiveSynth generates audio from incoming MIDI events with timbre specified by a reference recording or text prompt. It operates causally at a 100-Hz frame rate and supports timbre morphing and autonomous continuation.

Audio samples Code Model weights

Zero-shot instrument cloning

Each row uses the same MIDI excerpt. VAE reconstruction isolates the quality loss introduced by the causal audio representation.

Example Reference Ground truth VAE recon. LiveSynth TokenSynth CTD

Example 1

Example 2

Example 3

Example 4

Example 5

Text-to-Instrument

A text prompt specifies the timbre while both models render the same MIDI excerpt. We show results using raw and Procrustes-aligned CLAP embeddings.

Prompt

“the sound of a acoustic flute”

Prompt

“the sound of a acoustic mallet”

Prompt

“the sound of a acoustic string”

Timbre morphing

The MIDI events remain unchanged while the timbre condition moves continuously between two reference recordings.

Example 1

Reference AReference B

Example 2

Reference AReference B

Example 3

Reference AReference B

Example 4

Reference AReference B

Example 5

Reference AReference B

Autonomous performance continuation

After a 5-second MIDI-conditioned prefix, LiveSynth continues from its causal context with the absent MIDI condition.

Example 110-s excerpt