To demonstrate the diverse and powerful generation capabilities of SongCreator, we present various samples about universal song generation.
Music is an integral part of human culture, embodying human intelligence and creativity, of which songs compose an essential part. While various aspects of song generation have been explored by previous works, such as singing voice, vocal composition and instrumental arrangement, etc., generating songs with both vocals and accompaniment given lyrics remains a significant challenge, hindering the application of music generation models in the real world. In this light, we propose SongCreator, a song-generation system designed to tackle this challenge. The model features two novel designs: a meticulously designed dual-sequence language model (DSLM) to capture the information of vocals and accompaniment for song generation, and an additional attention mask strategy for DSLM, which allows our model to understand, generate and edit songs, making it suitable for various song-related generation tasks. Extensive experiments demonstrate the effectiveness of SongCreator by achieving state-of-the-art or competitive performances on all eight tasks. Notably, it surpasses previous works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally, it is able to independently control the acoustic conditions of the vocals and accompaniment in the generated song through different prompts, exhibiting its potential applicability.
To demonstrate the diverse and powerful generation capabilities of SongCreator, we present various samples about universal song generation.
| Lyrics | Pre-determined Vocal | Pre-determined Accompaniment | SongCreator |
|---|---|---|---|
| Lyrics-to-Song | |||
| So if I let down my guard, if I rip up my scars, and I show you my heart, am I beautiful? If I tell you my secrets, show my dark and my demons, tell me, what do you see? Am I beautiful? | - | - | |
| Maybe I can, we just come meet for dinner, at benningan's, I never meant to make you cry, is it too late to try, don't you remember what we said, as long as we stick together. | - | - | |
| Back on my feet, back on my feet again, I gotta roll the dice, back on my feet, no time to feel the pain, back on my feet, back on my feet again, yeah yeah yeah yeah, back on my feet, no time to feel the pain. | - | - | |
| Well make it to the end, and I can probably have been a better, better friend, I just want to start again, could we go back to when, we used to listen to radiohead. | - | - | |
| We never sleep, we never try, when you are with me, I wanna stay, I wanna stay here with you (ooh), cause you make me feel like, I could be driving you all night, and I'll find your lips in the streetlights. | - | - | |
| And don't take this from me, I've been fading away, memories are turning to gray, but every step that I take, the closer I get, to finding there's more to me, than what these eyes have seen, it's within my reach, and I'll take it all. | - | - | |
| Stuck in your ways, where will you go? I was the one who held you up, whenever you were low, I was the only one who refused to see (refuse to see). | - | - | |
| Got you on my mind, back on my feet, back on my feet again, I gotta roll the dice, back on my feet, no time to feel the pain. | - | - | |
| Can't seem to get my mind off of you, back here at home, there's nothin' to do, now that I'm away, I wish I'd stayed. | - | - | |
| Baby and now we're six months older and everything you want, and everything you see is out of reach, not good enough I don't know what the hell you want from me. | - | - | |
| I've become so numb, I can't feel you there, become so tired, so much more aware, I'm becoming this, all I want to do is be more like me, and be less like you. | - | - | |
| Lyrics-to-Vocals | |||
| Cause I'm trying, trying, to set me free, will you let me know, if you have the key. | - | - | |
| Even though you hurt me, I feel blessed love, baby, I'm your puppet on a string, making me tumble and swing. | - | - | |
| Giving me the greatest gift you can, saying yes and now our life begin, choosing you daily, so take my hand now. | - | - | |
| Like a pocket knife shadow, I am confused about who I am, I see your shadow, now I am not afraid to hurt you, I have never quite hurt someone like this before. | - | - | |
| Stuck in your ways, where will you go? I was the one who held you up, whenever you were low, I was the only one who refused to see (refuse to see). | - | - | |
| Don't know why I run, don't know why I hide, don't know why I try to keep these issues. Running through my mind all to myself, don't know what I want, don't know who I am, don't even remember who I used to be, before the storm came crashing down. | - | - | |
| Accompaniment-to-Song | |||
| Too far from home, all I do is searching and wondering, never knew the one I hoped for is here waiting, do you feel the same? | - | ||
| Was it something that I said? I tried my best, yeah, all for you. I can see it in your eyes, the way you lie, I'm such a fool. | - | ||
| Just a smiling face, I can see she's headed for a tragedy, I wanna save the girl, but it's not my place. | - | ||
| And don't forget the lows everybody's got a story no one knows no one knows so just let them play until the credits roll. | - | ||
| Oh, could I take care? For the only one i've dreamed about, until you listened my last goodbye. | - | ||
| Will you notice me, are you what I need, I'm trying, trying, to set me free; | - | ||
| Vocals-to-Song | |||
| Don't know why I run, don't know why I hide, don't know why I try to keep these issues, running through my mind all to myself, don't know what I want, don't know who I am, don't even remember who I used to be, before the storm came crashing down. | - | One too many times, I'm the one to take the burden, oh, you know I've tried, but got nothing, I got nothing; | - |
| Enough, I've been waiting forever for you to change, I can't do this forever not one more day, never wanted to lose you this way, but you're stuck." | - | ||
| I've got better things to do, and better things to listen to, than all your ranting and raving on me, it's all of the time now, and it's too late for you to find reality here, now I want you out of my life; | - | ||
| You're stuck in my head, stuck on my heart, stuck on my body, I wanna go; get outta here, I'm sick of the party, I'd run away, I'd run away with you | - | ||
| "Even though it hurts and I still feel shame, best friends try to warn you about the game. | - |
To demonstrate the diverse and powerful generation capabilities of SongCreator, we present various samples about controllable song generation.
| Lyrics | Vocal Prompt | Accompaniment Prompt | SongCreator |
|---|---|---|---|
| Lyrics-to-Song with vocal prompt and accompaniment prompt from the same segment | |||
| The life I planned for me, and my dreams, I am losing control, of my destiny, it feels like I'm falling and that's what it's like to believe, so I'm letting go, giving in to your gravity. | |||
| And every second I waste is more than I can take, I've become so numb, I can't feel you there, become so tired, so much more aware, I'm becoming this, all I want to do is be more like me, and be less like you. | |||
| I was a weary one but you saved me, you picked me up and put me on your shoulders, and brought me to the tree; | |||
| You can see in his eyes, he's in love with her, you don't wanna be the one to cause more hurt, the damage is done, don't make it worse. | |||
| Only doing things out of frustration, trying to make it work, but man, these times are hard, she needs me now, but I can't seem to find the time. | |||
| Lyrics-to-Song with vocal prompt and accompaniment prompt from different segment | |||
| Draw me angels with flapping wings, ooh... Strange machines, cos after the daekness is sunrise, there'll be hugs and teary smiles. | |||
| If I get a second chance, I'll stop fighting jews, and I hope that they'll be nice to me too, I was so focused on promoting, my own aryan nation, I forgot about my friendship with jews. | |||
| Don't wish too hard, because they may come true, and you can't help them, you don't know what you might have set upon yourself, china in your hand. | |||
| Only doing things out of frustration, trying to make it work, but man, these times are hard, she needs me now, but I can't seem to find the time. | |||
| Black hole sun, won't you come, and wash away the rain? Black hole sun, won't you come, won't you come. | |||
| Lyrics-to-Song with only vocal prompt | |||
| And its unrelenting chase, I was on the edge of deception, caught up in my own hesitation, until your love took over me. | - | ||
| I'am take a sad girl, turn into a bad girl, even at your worst, you're better than the rest, find someone to kiss tonight, underneath the flashing lights, we gon' have the time of our lives. | - | ||
| Tell you what I want, tell me what you need, the things that I desire, the things that make you bleed, I'll lift the pain away, just give me all of you, I'll build it up then tear it if you want me to, but no, I won't do it, my body's in for the moment, i wanna feel you, but i don't wanna take. | - | ||
| Nothing can stop me, I'm the champion, I've been waiting for you, I've been waiting for you, when the shadows come and cover the horizon. | - | ||
| His mind full of deception, the prize he had no care for, to win was his desire, I stand before these walls, neglected but watched over. | - | ||
| Lyrics-to-Song with only bgm prompt | |||
| But am I gonna get enough from you, I don't know what it is about you, but I can't keep my eyes off you, let me know if you feel it too, I wanna make you mine right now, baby let me show you how. | - | ||
| When the rain, is blowing in your face, and the whole world, is on your case; I could offer you, a warm embrace. | - | ||
| Yeah, they're making us crazy, don't give up on me, baby. Oh, these times are hard, yeah, they're making us crazy, don't give up on me, baby, oh, these times are hard; yeah, they're making us crazy, don't give up on me, baby; | - | ||
| What brings me back to this? Is this something I need, or have I done something wrong? how did we even get here. | - | ||
| Aa-aa-ah, losing all emotions, I did'nt feel nothing, in need of attention but nobody's looking. | - |
To demonstrate the diverse and powerful generation capabilities of SongCreator, we present various samples about song editing. It is noteworthy that Original Song (Reconstructed) is reconstructed using BEST-RQ and LDM.
| Original Lyrics | Original Song (Reconstructed) | Edited Lyrics | SongCreator Edited Song |
|---|---|---|---|
| Song Editing | |||
| I made the right choice running away, all those years ago, I need a smoke pull over, haven't taken a drag since seventeen, when I thought you knew me better. | I made the right choice running away, all those years ago, let's stop for a moment, I crave a cigarette, it's been ages since seventeen, when I thought you knew me better. | ||
| Black and white, fever burning in the night, just a shade, in a room without a lightl, see your face, can't remember where I am, outer space. | Black and white, fever burning in the night, just a shade, in a room, can't remember where I am, outer space. | ||
| I'am take a sad girl, turn into a bad girl, even at your worst, you're better than the rest. | I'am take a sad girl, turn into a queen, even at your worst, you're better than the rest. | ||
| A voice to call the dead from their sleep, the wonder of it all is that you found me. | A voice to call the souls back to waking life, marveling at the serendipity all is that you found me. | ||
| But we always have been friends, I miss you, both we have to grow, both we got mature. | But we always have been friends, I miss you, now we both must grow and learn, both we got mature. | ||
| She's in line at the dole with her head held high. | She's in line waiting patiently at the dole with her head held high. | ||
| Never let me go, strange love. | Never let me go, hold me close in your arms, cherish every moment together, strange love. | ||
| I can't pull myself away, I can't pull myself away. | I can't pull myself away, trapped by this magnetic force, drawn irresistibly closer, closer despite my efforts, I can't pull myself away. | ||
| Vocals Editing | |||
| And the people bowed and prayed, to the neon god they made, and the sign flashed out its warning, and the words that it was forming. | And the people bowed and prayed, to the bright idol they raised, in the light that boldly gleamed, and the words that it was forming. | ||
| She's in line at the dole with her head held high. | She's in line waiting patiently at the dole with her head held high. | ||
| So if I let down my guard, if I rip up my scars; and I show you my heart; am I beautiful? If I tell you my secrets, show my dark and my demons. Tell me, what do you see? am I beautiful? | So if I let down my guard, if I rip up my scars, and I show you my heart, am I beautiful? Tell me, what do you see? Am I beautiful? | ||
| Fools said I, you do not know, silence like a cancer grows, hear my words that I might teach you, take my arms that I might reach you. | Fools said I, you do not know, silence like a cancer grows, hear my words that I might teach you, take these hands that I might reach you. | ||
| Vocals Editing in Song | |||
| We had the songs that we sang along to, you had the moves to make me dance with you, I always saw you reaching and catching stars, we had the thing that everyone wanted, hung on your sleeve, you wore your heart on it, did you get out? | We had the songs that we sang along to, you had the moves to make me dance with you, you seemed to grasp at dreams, pulling them close, we shared a magic that all wanted, hung on your sleeve, you wore your heart on it, did you get out? | ||
| You're stuck in my head, stuck on my heart, stuck on my body, I wanna go, get outta here, I'm sick of the party, I'd run away, I'd run away with you. | You're stuck in my head, stuck on my heart, stuck on my body, I wanna go, get outta here, I'm sick of the party, I'd escape this noise, I'd run away with you. | ||
| Telling me I'm on my own, do I have what it takes for this? (Will I fly? will I fall?) At the edge of all I've ever wanted, head is spinning stuck in a tug of war, tell me is it meant to be this way. | Telling me I'm on my own, do I have what it takes for this? (will I fly? will I fall?) Ready to leap beyond what I've dreamed, caught between hope and a hard place; tell me is it meant to be this way. | ||
| In restless dreams I walked alone, narrow streets of cobblestone, neath the halo of a street lamp, I turned my collar to the cold and damp. | In restless dreams I walked alone, silent paths of old stone, under a faint street light’s glow, I turned my collar to the cold and damp. | ||
| When my eyes were stabbed, by the flash of a neon light, that split the night, and touched the sound of silence. | When my eyes were stabbed, by the flash of a neon light, that split the darkness, it reached the sound of silence. |
To demonstrate the diverse and powerful generation capabilities of SongCreator, we present various samples about music continuation.
| ID | Music Prompt | SongCreator |
|---|---|---|
| Music Continuation | ||
| 1 | ||
| 2 | ||
| 3 | ||
| 4 | ||
| 5 |
We present some interesting examples of song generation without lyrics. However, please note that since SongCreator were not specifically optimized for unconditional generation, the vocal content is often akin to babbling.
| Vocal Prompt | Pre-determined Accompaniment | Song Prompt | SongCreator |
|---|---|---|---|
| Song Continuation | |||
| - | - | ||
| - | - | ||
| - | - | ||
| Vocals Continuation | |||
| - | - | ||
| - | - | ||
| - | - | ||
| Accompaniment-to-Song (no lyrics) | |||
| - | - | ||
| - | - | ||
| - | - | ||
| - | - |
We additionally retrained the SongCreator on approximately 20,000 hours of high-quality Chinese song data to verify its ability to generate high-quality songs in other languages. Here, we present some examples of generated songs, alongside comparisons with results from commercial model like Suno.
| Lyrics | Suno | SongCreator |
|---|---|---|
| Lyrics-to-Song | ||
| 这一路上走走停停,顺着少年漂流的痕迹,迈出车站的前一刻,竟有些犹豫,不禁笑这近乡情怯,仍无可避免,而长野的天,依旧那么暖. | ||
| 我曾难自拔于世界之大,也沉溺于其中梦话,不得真假,不做挣扎,不惧笑话,我曾将青春翻涌成她,也曾指尖弹出盛夏,心之所动,且就随缘去吧. | ||
| 想用倔强造句遣词,写最与众不同的诗,想用青涩涂抹画纸,绘我独一无二的样子.想用理想扬帆立帜,宣最风雨同舟的誓,想用时光温暖故事,为你蓬勃鲜艳的明日. | ||
| 就让爱照亮年华飞逝,陪伴着每寸年少心事,像风无声,慰抚尘世,一起勇敢闯未来未知. | ||
| 我问为什么,那女孩传简讯给我,而你为什么,不解释低着头沉默,我该相信你很爱我,不愿意敷衍我,还是明白你已不想,挽回什么. | ||
| 我怀念的,是无话不说,我怀念的,是一起做梦,我怀念的,是争吵以后,还是想要爱你的冲动. | ||
| Lyrics-to-Vocal | ||
| 越过绵绵的高山,越过无尽的沧海,如果期待依然在,总是春暖到花开,请你轻轻留下来,让梦卷走这尘埃,香飘在书厢之外,奏响美丽的天籁. | - | |
| 如果爱忘了,泪不想落下,那些幸福啊,让她替我到达,如果爱懂了,不能给我的,请完整给她. | - |
For each of the above tasks, we provide a detailed task description.
| Tasks | Description |
|---|---|
| Lyrics-to-Song | This task aims to generate a complete song that includes harmoniously integrated vocal and accompaniment from lyrics. SongCreator supports to control various acoustic conditions in the generated song by providing optional prompts. The vocal prompt can control speaker, vocal melody, and tempo, while the accompaniment prompt can control instruments, musical melody, and rhythm. The vocal prompt and accompaniment prompt can either be present simultaneously, exist individually, or be absent altogether. |
| Lyrics-to-Vocals | This task aims to generate the vocals without accompaniment based on the given lyrics. In this case, the vocal prompt can be provided to control the speaker, melody, and tempo of the generated vocals. |
| Accompaniment-to-Song | The purpose of this task is to supplement the vocal track of a song based on the given lyrics for a pre-determined accompaniment track. The vocal track in generated song should complement the input accompaniment track to create a coherent song. Similar to lyrics-to-vocals, the generated vocals can also be controlled using the vocal prompt. |
| Vocals-to-Song | Contrary to the accompaniment-to-song task, the purpose of this task is to generate harmonious accompaniment for the input vocal track and combine them to create a coherent song. |
| Song Editing | This task requires a model to alter a segment within a song to match a target lyrics. The modified segment must be coherent with the unedited parts of the original song, i.e., maintaining the speaker, instruments, melody and rhythm. |
| Vocals Editing | This task is similar to song editing, but the modification target is changed from the complete song to the vocals without accompaniment. |
| Vocals Editing in Song | This is a unique capability of SongCreator, which modifies the content of the vocal track in a song while keeping the original accompaniment track unchanged. It means that the modified vocal segment not only maintains coherence with the unedited vocal track of the original song but also harmonizes with the accompaniment in the original song. |
| Music Continuation | This task is expected to generate instrumental music, which is coherent with the accompaniment prompt in terms of instruments, melody, harmony and rhythm. |
| Song Continuation | This task is expected to generate a complete song without input lyrics, which is coherent with the song prompt in terms of instruments, speaker, melody, and rhythm. |
| Vocals Continuation | This task is expected to generate vocals without input lyrics, which is coherent with the vocal prompt in terms of speaker, vocal melody, and tempo. |
| Accompaniment-to-Song (no lyrics) | This task is similar to the accompaniment-to-song task, but instead of providing lyrics, it only offers the pre-determined accompaniment track, allowing the model to supplement the vocal track of a song. |