3.1 Code Mixed IR

We assume L to represent the set of natural languages and S to be the set of scripts such that L=l1,l2,⋯,ln and S=s1,s2,⋯,sn respectively. There could either be a one-to-one or one-to-many mapping between the two sets. Let us assume that a word w written in a language with a particular script be denoted as li,sj. When i = j, we say that the word is written in its native script otherwise, it is in transliterated form.

Let q be a query given over a set of documents D where the IR task is to rank the documents in D so that the documents most relevant to q appear at the top. For a monolingual query q, we can assume that q∈l1,s1. In MSIR, q∈l1,sj where j may or may not be equal to 1 and therefore, the document pool is denoted as

where D1,k=d1,k,1,d1,k,2,⋯,d1,k,N are documents in language l1 written in script sk, i.e., for all m, d1,k,m∈l1,sk. It suggests that in the MSIR setup, the query and the documents are all in the same language, say l1, which are written in more than one different scripts. Therefore the IR task is to search across the scripts. However, in case of CMIR, query terms belong to different languages which may be either in their native scripts or transliterated form. Therefore, in this case, either q∈li∪lj,si∪sj or q∈li∪lj,sj for bilingual documents. This suggests that the documents are written in either or both the languages l1 and l2 in their native scripts s1 and s2 respectively or both l1 and l2 written in the same script (either s1 or s2). It is very less likely that in a practical scenario a query will be written in multiple scripts.

The document pool thus becomes

where L=l1,l2,⋯lN, DL,k=dL,k,1,dL,k,2,⋯,dL,k,N are documents in languages L written in scripts sk. This means for all m, dL,k,m∈L,sk. Therefore, the IR task in this case is not only to search across the scripts but also across the languages of the query terms and the documents.

3.2. Difficulties and Challenges in CMIR for Social Media Texts

The IR task under the CMIR setup is to search for documents written in multiple languages and rank them according to the query given. CMIR on social media texts introduces several challenges for the IR task. First, it is a difficult task to identify the language of each query terms for a given Code-Mixed query across Code-Mixed tweets where the document length is short. The second challenge is the presence of spelling variations in transliterated query terms. For example, “मै” in Hindi which stands for “I” or “me” in English, can be written in any one of the following Roman transliterated forms such as mein,me,mei,main,mai and so on. Therefore, for a query 𝑞 which has one of the given Roman transliterated search terms, only those documents (tweets in our case) in the document collection 𝐷 will be retrieved which matches with the exact terms. However, there may be documents in 𝐷 with same query terms with different spellings. A retrieval system in such a scenario will ignore all documents that do not contain the query search term with the exact spellings. Third challenge is to identify words which are represented using mixing of non-alphabetic characters. Tweets are known to be terse in nature and sometimes words could be expressed with non-alphabetic characters. For instance, “before” is often written as “b4”. Therefore, existing state-of-the-art systems are not appropriate for handling documents of such nature as discussed above.

4.1 Data Collection

For this study, we have collected Hindi-English tweets where the Hindi terms are in Roman transliterated form. Fetching Code-Mixed Hindi-English tweets is itself a difficult task. Tweets were collected on topics related to the events trending at that time. However, it is difficult to decide what is trending. Therefore, we started searching for tweets for events that were happening and making news at the time of collection. For example, for a topic like Delhi election which was held in February 2015, we started collecting tweets from November 2014 to April 2015 that is the period from political campaigning to declaration of the election results. For such collection of tweets, the probable query search terms are the entities such as names of the political parties and the candidates contesting the election. Initially we attempted to collect tweets based on the Entities related to the topic. However, it was difficult to retrieve Code-Mixed Hindi-English tweets, therefore, we simply issued bag of words as the query containing Hindi terms in Roman transliterated form mixed with the topic under consideration. The Hindi search terms are mostly stop words. For example, for tweets related to the Delhi election, queries such as “BJP aur Kejriwal” (BJP and Kejriwal) were issued. In the example query, “BJP” is the name of a political party, “Kejriwal” is the name of a candidate and “aur" is a Hindi stop word meaning “and” in English. In another example query, “aam aadmi party Delhi mein” where “aam aadmi party” is the name of a political party and “mein” (“in”) is the Roman transliterated Hindi word. In similar fashion, tweets related to other trending topics were collected such as Cricket World Cup 2015 which was held between February and March 2015, the Bollywood² controversy related to an Indian film actor. We also tried to collect tweets from other topics during the period from December 2015 to January 2016.Though several other topics were searched but we did not include them in the corpus because of their smaller presence (limited to 3 to 4 tweets only). We have used Twitter4j API ³ for collecting the tweets. Due to the API restriction, we could not fetch tweets beyond seven days thus resulting in limited number of tweets for some topics.

Tweets were collected based on different query search terms using AND and OR Boolean operators. For example, a search query like “AAP OR aadmi AND hai” which has three terms viz; “AAP”, “aadmi” and “hai” was used to fetch tweets containing either “AAP” only or “aadmi” and “hai” both. A total of 9,715 tweets were collected based on different query search terms. It has been observed that there were initially redundant tweets due to the presence of re-tweets, retrieval of the same tweets from two different queries and same tweets with trailing different hash tags, URLs and emoticons. By redundant, we mean in cases where the maximum text of two or more tweets match. Therefore, we have considered them as duplicate tweets and have removed them by a measure known as Jaccard similarity coefficient ⁴.It is interesting to observe that certain tweets were retrieved which are written in Devanagari script for the Hindi words. In some cases an entire tweet has been found to be written in Devanagari script. We have not considered such tweets for our work and therefore we have manually removed them from our corpus. As our initial step, we have used CMU tokenizer ⁵, ¹³for tokenizing the tweets. Finally, the corpus size was reduced to 9578 tweets out of which 6678 are Code-Mixed and the distribution is shown in Table 1.

Table 1 Topic wise distribution 

4.2. Language Identification

For any Code-Mixed corpus, it is of utmost importance to identify the languages at the word level. Automatic language identification from large multilingual corpora has great significance and therefore, we have used such a system² for our purpose. ² gave an F1 score of 91.1% for automatic language identification on our corpus. The tokens were tagged as hi,en,ne,acro,univ,undef. Here hi and en signifies that a token is either a Hindi or an English word respectively. Likewise, ne signifies that a token is a Named Entity, acro is an Acronym, univ is either an emoticon or a punctuation symbol and undef for tokens which are neither univ nor hi or en. An example annotation is shown below:

Tweet:

jaldi delhi .... thoda time aur ... aapni sarkar ko chuno ..... this time 75% voting .... :)

Annotation:

jaldi/ hi delhi/ ne ..../ univ thoda/ hi time/ en aur/ hi .../ univ aapni/ hi sarkar/ hi ko/ hi chuno/ hi ...../ univ this/ en time/ en 75%/ univ voting/ en ..../ univ : )/ univ

Meaning:

hurry delhi .... little more time ... choose your government ..... this time 75% voting .... :)

Tag wise distribution of the tokens from 6,678 Code-Mixed tweets is shown in Table 2. From Table 2 it is observed that Hindi word count is more than the English words in the corpus.

Table 2 Tag wise distribution of tokens in 6678 CodeMixed tweets 

4.3 Code Mixing Types

After automatic language identification of the words using², we have measured the level of Code-Mixing by calculating the Code Mixing Index (CMI) introduced in ³ and ⁴.CMI has been calculated to measure the level of mixing between Hindi and English in our corpus. In our corpus both intra and inter-sentence level Code-Mixing are present. We have not measured their percentage distribution because categorizing them requires Sentence Boundary Detection (SBD) which is itself a very difficult task. We therefore only report the distribution of Code-Mixed data in our corpus. However, there is another type of Code-Mixing occurring at word level which we have not found in our corpus. Table 3 lists the distribution of monolingual vs. multilingual tweets. We observed that 69.72% of our corpus is Code-Mixed and the remaining 30.28% is monolingual.2% of the corpus has been found to be of other monolingual type. A tweet is considered as “other monolingual” if there are no hi or en words.

Table 3 Type of Code-Mixing in our corpus 

5.1 Creation of Gold Standard Data

From the 6678 Code-Mixed tweets, we have taken 1959 tweets and conducted two annotation experiments. The reason for choosing the 1959 tweets for creating gold standard is that for our experiments, the chosen set of queries from few topics constitute only 1959 tweets. For the annotation purpose we could not deploy annotators from Amazon Mechanical Turk (AMT) due to financial constraints. Therefore, two in-house students studying Masters in Computer Science and Engineering, were selected for the annotation work who are well conversant with both Hindi and English languages. In our first experiment, each annotator was given a topic with query related to the topic and the list of tweets. Annotators were then requested to rank the tweets according to its relevance for the given query without consulting each other. The agreement score in the first experiment was very low with a Kappa⁶ value of 0.1312 only. After carefully investigating the queries and the corresponding tweets, we have found that for some queries it was comparatively easier to rank them whereas for some other queries, the ranking based on relevance judgement was difficult. It was difficult because the collection of tweets for certain queries were semantically diverse from each other with respect to the query. Therefore, in our second experiment, both the annotators were instructed to understand the expected outcome of a given query and to discuss the relevance of a tweet accordingly. After conducting the second experiment, we achieved inter-annotator agreement with a Kappa value of 0.6762. This suggests that relevance judgement for tweets is itself a very difficult task due to their terse nature.

5.2 Searching and Ranking

For evaluation of existing IR systems on our corpus of 1959 gold data, we have used Lucene⁷ for indexing, searching and retrieval. Lucene combines Boolean model (BM) with Vector Space Model (VSM) of Information Retrieval where each distinct index term is a dimension, and weights are 𝑡𝑓-𝑖𝑑𝑓 values and uses the practical scoring function⁸. We tested with 20 queries with an average query length of 2.67 terms and compared them against the gold data. For our experimental purposes, Boolean queries with AND and OR operators and queries as phrases were issued for searching and retrieving the tweets.

We have configured Lucene to retrieve the top 20 tweets for each query. Each tweet has been considered as a document for indexing. Therefore, for 1,959 tweets there are 1,959 documents that were indexed. The Lucene Standard Analyzer is capable of preprocessing (tokenization, stop word removal) only English texts. A total of 11,565 terms were indexed with three fields (contents, file name, file path). The top ten terms that Lucene identified are shown in Table 4. It is interesting to observe that most of these terms are actually Hindi stop words such as hai,ki,ko,ka.The Standard Analyzer could not detect the presence of Hindi stop words in their Roman transliterated form thus affecting the ranking. Also noted in Table 4 is that the frequency of relevant indexed terms such as kejriwal,sonia,khan etc. are comparatively low. An indexed term is considered relevant if it is close to the topic. For example, terms like kejriwal,bjp,modi etc. are considered relevant for the tweets related to Delhi election.

Table 4 Top 10 indexed terms and other relevant terms among top 100 terms 

5.3 Results

For evaluating the performance of Lucene on our gold corpus of 1,959 tweets, we initially noted the Mean Average Precision (MAP) of all the tweets ranked by Lucene for the queries against the gold standard. The MAP value we obtained is 4.254359863552588 x 10^-5 which is very low in terms of performance. This suggests that relevance judgement for ranking informal texts such as tweets, is a difficult task because relevance judgement is idiosyncratic in nature and therefore, creating gold standard data for IR is biased. Therefore, MAP is more likely to be low if there is a huge disparity between the system rankings and the gold rankings. Moreover, due to the terse nature of tweets, it is difficult to judge whether a particular tweet is more relevant than the other for a query. Therefore, we instead took a different approach for evaluating Lucene on our corpus. In our alternate approach, we first issued exactly the same queries to Lucene that were used to collect the tweets. Again we conducted two rounds of annotations. In the first round, tweets that were ranked by Lucene were given to the annotators for manual re-ranking. The inter-annotator agreement was then observed to be 0.7458, which is higher in comparison to the earlier annotation agreement score of 0.6762. In the second round, both the annotators were requested to come to a consensus and create the gold standard. Finally, gold data was created after manually re-ranking the system (Lucene) retrieved tweets. Queries have differing number of relevant documents. Therefore, we cannot use one single cut-off level for all queries. This would not allow systems to achieve the theoretically possible maximal values in all conditions. We have therefore, measured the 11-point average precision ¹¹ and Mean Average Precision (MAP) for the Vector Space Model based ranking mechanism. A total of 20 queries were issued and finally a MAP value of 0.186 was obtained which is reported in Table 5.

Table 5 11-point Average Precision and MAP