2.1 Mood Taxonomies

The preparation of an annotated dataset requires the selection of proper mood taxonomies. Identifying an appropriate mood taxonomy is one of the primary and challenging tasks for mood classification. Mood taxonomies are generally categorized into three main classes namely, categorical, dimensional, and social tags ²⁰.

Categorical representation describes a set of emotion tags organized into discrete entities according to their meaning. The earliest categorical music mood taxonomy was proposed by Hevner¹⁰ and is known for its systematic coverage on music psychology¹³. Another traditional categorical approach uses adjectives like gloomy, pathetic and hopeful etc. to describe different moods²¹. On the other hand, Music Information Retrieval eXchange² (MIREX) community proposed a categorical mood taxonomy for audio based mood classification task¹⁴, which is quite popular among the MIR researchers. In case of Indian music mood classification, Koduri and Indurkhya worked on the mood classification of south Indian classical music using categorical mood representation and they considered the mood taxonomy consisting of ten rasas (e.g., Srungaram (Romance), Hasyam (Laughter), Karunam (Compassion) etc.)¹⁷. Similarly, Velankar and Sahasrabuddhe prepared data for mood classification of Hindustani classical music consisting of 13 different mood classes (e.g., Happy, Exciting, Satisfaction etc.) ³⁶.

Dimensional models of emotion categorization describe emotions with respect to one or more axes. The most well known example of such a space is the “valence-arousal"³¹ or “energy-stress"³³ representation. The valence indicates positivity and negativity of emotions whereas the arousal indicates emotional intensity¹⁶. One of the earliest researches carried out on the dimensional models was proposed by Russell ³¹. The author proposed the circumplex model of affect (consisting of 28 affect words) based on the two dimensions, denoted as “pleasant-unpleasant" and “arousal-sleep". In context of Indian music mood classification, most of the researches adopted the dimensional model. Ujlambkar and Attar used the Russell’s circumplex model of affect to develop a mood taxonomy of five mood classes and each of the classes consists of three or more sub classes ³⁵. Patra et al. ²⁷ have used five mood classes with three or more subclasses, which are the subsets of the Russell’s circumplex model of affect.

Social tags are generally assigned by the non-experts for their own personal use, such as listeners to assist in organization and accessibility of an item¹⁸. Tags are typically the words or short phrases or unstructured labels that describe resources. In case of Western songs, mood classification was also performed using social tags in¹⁸,²⁰.

It was observed that the Hevner’s adjectives¹⁰ are less consistent in case of the intra-cluster similarity, whereas MIREX mood taxonomy¹⁴ suffers with inter-cluster dissimilarity and confusion between the categories were observed²⁰. From the above, Laurier et al.²⁰ concluded that the psychological models have some similarity with the social tags though it may not be suitable for today’s music listening reality¹¹. In case of the Indian songs, no such social tags were collected or reported till date.

2.2 Music Mood Classification

The framework of classification systems was divided into three categories based on the type of features and experimental settings.

3.1 Proposed Mood Taxonomy

Most of the taxonomies in the literature were used for evaluating the Western music. Ancient Indian actors, dancers and musicians divided their performance into nine categories based on emotions and called the different emotions together as Navrasa, where rasa means emotions and nav means nine. Unfortunately in the modern context of music making, all the nine types of emotions are not frequently observed. For example, the emotions like surprise and horrific belonging to the Navrasa are rarely observed in current Hindi music. The emotion word Hasya (Happiness) need a further subdivision, for instance, happy and excited. Hence, this model cannot be used for analyzing the mood aspects of Indian popular songs³⁴.

Another interesting mood taxonomy for classifying Hindi music was proposed by³⁴ after consulting feedback of 30 users. Observation of many music tracks led us to believe that romantic songs may be associated with largely varying degrees of arousal and valence, making it difficult to categorize based on Thayer’s or Russell’s model. The songs from sad class need a further subdivision, because there are many sad songs with high arousal.

The comparative analysis of different mood taxonomies revealed that the clustering of similar mood adjectives has a positive impact on the classification accuracy. Based on this observation, we opted to use Russell’s circumplex model of affect³¹ by clustering the similar affect words located close to each other on the arousal-valence plane into a single class as shown in Figure 1. We considered the mood classes Angry, Calm, Excited, Happy and Sad for our experiments. Each of the classes contains another two nearby key affect words of the circumplex model of affect. Thus, our final mood classes are Angry (Alarmed, Tensed), Calm (Satisfied, Relax), Excited (Aroused, Astonished), Happy (Pleased, Glad) and Sad (Gloomy, Depressed). One of the main reasons for collecting songs and grouping the similar songs into a single mood class is to consider the significant invariability of the audio features at subclass level with respect to their main class. For example, a “Happy" and a “Delighted" song have high valence, whereas an “Aroused" and an “Excited" song have high arousal.

Fig. 1 Russell’s circumplex model of affect ³¹  

3.2 Data Preparation

In the present work, we collected the lyrics data from web archives corresponding to the annotated audio dataset available for Hindi songs in²⁷. The lyrics are basically written in Romanized English characters whereas the prerequisite resources like Hindi sentiment lexicons, emotion lexicons and list of stop words are available in utf-8 character encoding. Thus, we transliterated the Romanized English lyrics to utf-8 characters using the transliteration tool available in the EILMT project.⁴ We observed several errors in the transliteration process. For example, words like ‘oooohhhhooo’ ‘aaahhaa’ were not transliterated due to the presence of repeated characters. Again, the words like ‘par’ and ‘paar’, ‘jan’ and ‘jaan’ were transliterated into different words and , and , but, the above pairs are the same words and . Hence, these mistakes were corrected manually.

Each of the lyrics was annotated by at least three annotators aged 20±4 years, who were undergraduate students and research scholars and worked as volunteers for annotating the lyrics corpus. The lyrics were asked to annotate after reading it with either of the aforementioned five mood classes. Each of the lyrics was also annotated with positive, negative, and neutral polarities. In several cases, we observed that the mood class that was assigned to an audio is different from the mood class assigned to its corresponding lyric for some of the Hindi songs. The statistics of annotation during listening to audio ( 𝐿 𝐴𝑢𝑑𝑖𝑜 ) and reading of the lyrics ( 𝑅 𝐿𝑦𝑟𝑖𝑐𝑠 ) are provided in Table 1.

Table 1 Confusion matrix of annotated songs with respect to five mood classes [after listening to the audio (L_Audio) and reading of the lyrics (R _Lyrics ) 

The differences between reader’s and listener’s perspectives for the same song motivated us to investigate the root cause of such discrepancy. The authors believe that the subjective influence of music modulates the perception of lyrics of a song in the listeners. The poetic and metaphoric usage of language can be observed in the lyrics. For example, a song “Bhaag D.K.Bose Aandhi Aayi"⁵ has mostly sad words like “dekha to katora jaka to kuaa (the problem was much bigger than it seemed at first)" in the lyric. This song is annotated as “Sad" while reading and annotated as “Anger" as it contains mostly the rock music and the arousal is also high. Similarly, a song “Dil Duba"⁶ is annotated as “Sad" and “Happy" while reading the lyrics and listening to the audio, separately. This song portrays negative emotions by using sad or negative words like “tere liye hi mar jaunga (I would die for you)", but, the song contains high valence. The above observations emphasize that the combined effect of lyrics and audio is an important factor in indicating the final mood inducing characteristics of a music piece.

It was observed that the annotators were influenced by the moods perceived by the audio of the songs. It was also difficult to feel the mood of a song using only lyrics because the identification of metaphoric and poetic usage is hard without listening to the audio. Hence, the songs were considered which were annotated with the same mood class after listening to the audio as well as reading of its corresponding lyrics for further experiments. We have considered 27, 37, 45, 48 and 53 songs for Angry, Calm, Excited, Happy and Sad mood classes, respectively and each of these audio files was sliced into 60 seconds of clips. These clips were annotated previously done by Patra et al.²⁷.

The pairwise inter-annotator agreements were calculated on the dataset by computing Cohen’s 𝜅 coefficient⁴. The overall inter-annotator agreement scores with five mood classes were found to be 0.80 for Hindi lyrics. However, the inter-annotator agreement was around 0.96 for the lyrics data while annotating with positive, negative, and neutral polarity.

4.1 Audio Features

We have considered the key features like intensity, rhythm, and timbre for mood classification task. These features have been used for music mood classification for Indian languages in state-of-the-art systems . We have listed the audio features used in our experiments in Table 2 and these features were extracted using the jAudio toolkit⁷.

Table 2 List of features extracted from audio 

4.2 Lyric Features

A wide range of textual features such as sentiment lexicons, stylistic and n-gram features were adopted in order to develop the music mood classification system.

4.2.1 Sentiment Lexicons (SL)

We used three lexicons to classify the moods present in the lyrics and these lexicons are Hindi Subjective Lexicon (HSL)², Hindi SentiWordnet (HSW)¹⁵ and Hindi WordnetAffect (HWA)⁵. HSL contains two lists, one is for adjectives (3909 positive, 2974 negative and 1225 neutral) and another is for adverbs (193 positive, 178 negative and 518 neutral). HSW consists of 2168 positive, 1391 negative and 6426 neutral words along with their parts-of-speech (POS) and synset ids extracted from the Hindi WordNet.⁸ HWA contains 2986, 357, 500, 3185, 801 and 431 words with their parts-of-speech (POS) tags for angry, disgust, fear, happy, sad and surprise classes, respectively.

To the best of our knowledge, the performances of the available POS taggers and lemmatizers for Hindi language are not up to the mark. The CRF based Shallow Parser⁹ is available for POS tagging and lemmatization, but it also did not perform well on the lyrics data because of the free word order nature of Hindi lyrics. Thus, the number words matched with these sentiment or emotion lexicons are considerably less. The statistics of the sentiment words found in the whole corpus using three sentiment lexicons are shown in Table 3. We also extracted the positive and negative words that were annotated by our annotators. We found 641 and 523 positive and negative unique words from the total corpus.

Table 3 Sentiment words identified using HWA, HSL and HSW 

4.2.2 Text Stylistic (TS)

Text stylistic features have been used effectively in text stylometric analysis such as authorship identification, author identification²⁴. These features have also been used for mood classification from lyrics of Western music¹². The TS features such as the number of unique words, number of repeated words, and number of lines etc. were considered in our experiments. The detailed list of TS features along with their descriptions is given in Table 4.

Table 4 List of TSF and description 

5.1 Feature Selection

Feature level correlation is used to identify the most important features as well as to reduce the feature dimension³⁰. Thus, the correlation based supervised feature selection technique implemented in WEKA toolkit was used to find out the important contributory features for audio and lyrics. A total of 431 audio features were extracted from the audio files using jAudio. A total of 12 sentiment features, 12 textual stylistic features and 5832 N-gram features were also collected from the lyrics. The feature selection technique implemented using Weka yields 148 important audio features and 12 sentiment, 8 stylistic, and 1601 N-gram features from lyrics. We subsequently use these features for our classification purpose.

5.2 Mood Classification using Audio Features

For music mood classification using audio features, the linear kernel of LibSVM was selected since it provides the higher F-measure in our case as compared to the polynomial kernels. We performed the classification by adding the features one by one. Initially, the timbre features were used to classify the moods, then added intensity features and then rhythm features, incrementally. After adding all the features together, the audio based mood classification system achieved the maximum F-measure of 58.2%. The contribution of each feature in F-measure is given in Table 5.

Table 5 System Performances 

5.3 Mood Classification using Lyric Features

For mood classification using lyrics, the linear kernel was selected and the classification was performed by adding features one by one. Initially, the experiment was performed using only sentiment features, and then added other features subsequently. The maximum F-measure of 55.1% was achieved using the sentiment and N-gram features for five class mood classification of Hindi songs as shown in Table 5. We also annotated each of the lyrics with positive, negative, and neutral polarity in addition to five mood classes. The maximum F-measure of 69.8% was achieved for polarity classification system using the sentiment and N-gram features of Hindi song lyrics.

5.4 Multimodal Mood Classification

Finally, the experiments were performed using both audio and lyric features. Again, we used the linear kernel of LibSVM for the classification purpose. The TS features reduced the performance of the systems, thus these were excluded while developing the final multimodal system for mood classification of Hindi songs. The multimodal mood classification system achieved the maximum F-measure of 68.6% after adding all features for Hindi songs and the system performance is given in Table 5.

5.5 Observation and Comparison

The confusion matrix for the multimodal mood classification system is also given in Table 6. From the confusion matrix, it is observed that there is biassness in the classification system towards the nearest classes. There were confusions in between the mood class pairs such as “Angry & Excited", “Excited & Happy", “Sad & Calm", “Sad & Angry".

Table 6 Confusion matrix for the multimodal music mood classification system 

The audio based mood classification system developed in Patra et al.²⁹ achieved F-measure of 72%. The F-measure achieved by our audio based mood classification system was less as compared to the above system by around 14%. One of the main reason for such low performance is that the authors used 1540 number of audio clips whereas we used xx number of audio clips only. We had to select less number of audio clips to the cope up with the clash happened during mood level annotations of audio and lyrics. They used Feed-Forward Neural Networks (FFNNs) for identifying the moods of a song whereas we used LibSVM for mood classification. It is found in the literature that the performances of the state-of-the-art mood classification systems based on audio are better using the FFNNs²⁹,³⁰.

Ujlambkar and Attar³⁵ achieved the maximum F-measure of around 75-81%. Our system performed less as compared to the above state-of-the-art system by around 22%. They developed their system using a different mood taxonomy and with more number of audio files which are not available for research. Again, each of the clips was of 30 seconds for their experiment whereas in case of ours, the size of the music clips was 60 seconds. The performance obtained by our audio based mood classification system shows an improvement of around 8% over the state-of-the-art audio based mood classification systems²⁵,²⁶ (achieved accuracy of around 50%), because they used less number of audio files.

It was observed that the N-gram features yield F-measure of 46.5% alone for the mood classification system based on the lyric features. The main reason may be that the Hindi is free word order language and the Hindi lyrics are also more free in word order than the Hindi language itself. Whereas, the text stylistic features do not help much in our experiments as it reduces the F-measure of the system by around 2.7%.

The state-of-the-art mood classification system available for the Hindi songs in²⁸ achieved F-measure of 38% using lyric features only. Our system out performs the above system by around 17%. The main differences was that they annotated the moods to the lyrics after listening to the corresponding audio whereas in our case, we considered the moods annotated after listening to the audio and reading the corresponding lyrics to avoid the biasness in the mood annotation process. More number of lyrics were used for the current experiment. The polarity classification system outperforms by around 1% as compared to the polarity classification system available in . To the best of the author’s knowledge, there is no other mood classification system based on the lyric features available for Hindi songs till date. While multiple experiments were performed on mood classification of Western songs based on lyrics, but the differences in number of mood classes made comparisons among these works and our proposed method difficult.

The multimodal mood classification system achieved the maximum F-measure of 68.6% after adding all the audio and lyrics (excluding the TS features) features using LibSVM. To the best of our knowledge, there is no state-of-the-art multimodal mood classification system available for Hindi songs. Laurier et al.¹⁹ performed multimodal mood classification of Western songs using both audio and lyric features and achieved 98.3%, 86.8%, 92.8%, and 91.7% for Angry, Happy, Sad and Relaxed mood classes, respectively. They made the classification much easier by classifying one mood class at a time, i.e. for the first time they classified “Angry" or “not Angry" and so on. Hu and Downie¹² achieved 67.5% for multimodal mood classification using late fusion on a dataset of 5,296 unique songs comprising of 18 mood classes. Our multimodal mood classification system outperforms the above system by 1%.