Publicação
MERGE Lyrics: Next-Generation Lyrics Music Emotion Variation Detection
| Resumo: | Lyrics Music Emotion Variation Detection (LMEVD), which focuses on tracking emotional shifts within song lyrics, is a growing sub-field of Music Information Retrieval (MIR). However, progress has been hindered by the scarcity of publicly available high-quality datasets, inconsistencies in annotation practices, lack of standardized benchmarks, and limitations in the employed data representations, e.g., textual features and word embeddings. This thesis addresses these gaps through a large-scale empirical study and the creation of a novel resource for the community. The core contribution of this work is the introduction of the Music Emotion Recognition - Next Generation (MERGE) Bimodal Music Emotion Variation Detection (BMEVD) dataset, comprising 152 songs and 926 manually annotated segments with decoupled audio and lyric emotion labels. This dataset enables granular, stanza-level analysis of emotional progression. Using this dataset and an established benchmark, we conducted a comprehensive evaluation of modern Natural Language Processing (NLP) techniques for LMEVD. We systematically compared traditional feature engineering against static and contextual word embeddings, a wide range of machine learning and deep learning classifiers, and various feature pooling strategies. Our experiments demonstrate that contextual embeddings, particularly fine-tuned RoBERTa, significantly outperform static embeddings and classic features. Crucially, we found that a well-tuned Convolutional Neural Network classifier paired with fine-tuned RoBERTa embeddings and no pooling achieved the highest performance, a macro F1-score of 75% on the BMEVD dataset and 70% on a previous benchmark, surpassing prior results. Building on this strong unimodal foundation, we proposed a novel bimodal framework for Music Emotion Variation Detection (MEVD), where structural segments were obtained using All in One, with lyrics automatically transcribed from the isolated vocals (Demucs) via Whisper. These transcriptions were processed using our best-performing lyrical model (fine-tuned RoBERTa + CNN). Despite leveraging this optimal configuration, the bimodal fusion, evaluated through averaging and arousal-valence (AV) strategies, did not surpass the audio-only model’s macro F1-score of 48.37%. The average method only achieved a macro F1-score of 43.44%, and the AV method (macro F1-score: 44.94%) showed a targeted improvement on quadrants (Q2 and Q4). However, the overall performance was ultimately limited by the poor quality of automated transcriptions (Word Error Rate (WER): 55%), with the lyrics-only model only achieving a 40.31% macro F1-score. These findings confirm that reliable lyrical extraction remains the critical bottleneck for effective bimodal MEVD. Finally, to bridge the gap between computational analysis and practical application, we developed an interactive dashboard that visualizes the emotional dynamics of songs using our best-performing model. This work provides a robust benchmark, practical guidelines for effective LMEVD, and a foundation for future research into music’s emotional narrative, with applications in recommendation systems and music therapy, for example. |
|---|---|
| Autores principais: | Mangara, Alice Fátima |
| Assunto: | Lyrics Music Emotion Variation Detection Music Emotion Recognition Natural Language Processing Word Embeddings Machine Learning Deteção de Variação de Emoção em Música Reconhecimento de Emoção em Música Processamento de Linguagem Natural Representações vectoriais de palavras Aprendizagem Automática |
| Ano: | 2025 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso aberto |
| Instituição associada: | Universidade de Coimbra |
| Idioma: | inglês |
| Origem: | Estudo Geral - Universidade de Coimbra |
| Resumo: | Lyrics Music Emotion Variation Detection (LMEVD), which focuses on tracking emotional shifts within song lyrics, is a growing sub-field of Music Information Retrieval (MIR). However, progress has been hindered by the scarcity of publicly available high-quality datasets, inconsistencies in annotation practices, lack of standardized benchmarks, and limitations in the employed data representations, e.g., textual features and word embeddings. This thesis addresses these gaps through a large-scale empirical study and the creation of a novel resource for the community. The core contribution of this work is the introduction of the Music Emotion Recognition - Next Generation (MERGE) Bimodal Music Emotion Variation Detection (BMEVD) dataset, comprising 152 songs and 926 manually annotated segments with decoupled audio and lyric emotion labels. This dataset enables granular, stanza-level analysis of emotional progression. Using this dataset and an established benchmark, we conducted a comprehensive evaluation of modern Natural Language Processing (NLP) techniques for LMEVD. We systematically compared traditional feature engineering against static and contextual word embeddings, a wide range of machine learning and deep learning classifiers, and various feature pooling strategies. Our experiments demonstrate that contextual embeddings, particularly fine-tuned RoBERTa, significantly outperform static embeddings and classic features. Crucially, we found that a well-tuned Convolutional Neural Network classifier paired with fine-tuned RoBERTa embeddings and no pooling achieved the highest performance, a macro F1-score of 75% on the BMEVD dataset and 70% on a previous benchmark, surpassing prior results. Building on this strong unimodal foundation, we proposed a novel bimodal framework for Music Emotion Variation Detection (MEVD), where structural segments were obtained using All in One, with lyrics automatically transcribed from the isolated vocals (Demucs) via Whisper. These transcriptions were processed using our best-performing lyrical model (fine-tuned RoBERTa + CNN). Despite leveraging this optimal configuration, the bimodal fusion, evaluated through averaging and arousal-valence (AV) strategies, did not surpass the audio-only model’s macro F1-score of 48.37%. The average method only achieved a macro F1-score of 43.44%, and the AV method (macro F1-score: 44.94%) showed a targeted improvement on quadrants (Q2 and Q4). However, the overall performance was ultimately limited by the poor quality of automated transcriptions (Word Error Rate (WER): 55%), with the lyrics-only model only achieving a 40.31% macro F1-score. These findings confirm that reliable lyrical extraction remains the critical bottleneck for effective bimodal MEVD. Finally, to bridge the gap between computational analysis and practical application, we developed an interactive dashboard that visualizes the emotional dynamics of songs using our best-performing model. This work provides a robust benchmark, practical guidelines for effective LMEVD, and a foundation for future research into music’s emotional narrative, with applications in recommendation systems and music therapy, for example. |
|---|
Atividades financiadas
Não foram encontrados projetos financiados.