Automatic conversion from speech to rap music

Speech-to-music conversion has become more and more popular in recent years. However, existing approaches cannot be directly applied for automatic conversion from speech to rap, because rap is a special music genre that contains many stressed syllables aligned with beats of the music accompaniment. This paper presents the first speech-to-rap system. The system first applies forced alignment to both rap acapellas and speech with the same lyrics to obtain word segments, which are then used to compute the conversion factors for prosodie features such as pitch and duration. Then we employ a phase vocoder to convert the original speech based on the rap acapella's pitch and duration. After that, the rhythmic effect is added to the synthesized rap acapella according to the detected beat information via a beat tracking algorithm. Finally, the synthesized result is combined with the accompaniment track to form a rap song. A subjective test of mean opinion scores given by 22 subjects indicates an average score of 3.3 out of 5 possible points, demonstrating the feasibility (but still with room for improvement) of the proposed approach.