4.1 Paraphrases Extraction

We test a hypothesis that paraphrase graph construction method can be applied towards the problem of news texts (or rather headlines) clustering. Paraphrase graph is a graph where sentences are vertices, and edges connect headlines if they are para-phrases.

To test our hypothesis, we consider gold standard news clusters and construct paraphrase graphs for each of the clusters. The constructed graphs are supposed to be represented by single connected components with minimal number of outlying vertices.

To decide whether two headlines are paraphrases, we use our paraphrase identification model trained on the ParaPhraser corpus [⁶]. ParaPhraser is a project dedicated to the collection of the Russian paraphrase corpus. It consists of about 11 thousand of sentences pairs annotated as precise, loose or non-paraphrases. The sentences are derived from news headlines.

Our paraphrase identification model is actually an SVM with two types of features: shallow string-overlap-based features and se-mantic features (covering synonymy and word-building relations). Detailed description of the features can be found in [⁵]. This is our best paraphrase identification model so far (it obtained accuracy of 0.7448 and F1 of 0.8078 at Russian Paraphrase Detection Shared Task’2016 in binary classification subtask).

Thus, for each of the five news clusters in question, we extract their headlines and compare them pairwise inside each cluster. Each pair of news headlines inside a cluster is assigned a paraphrase class (1 - for paraphrases, either precise or loose, and 0 - for non-paraphrases) which is the output of our paraphrase identification model. The resulting set of sentences together with their paraphrase relations form a paraphrase graph for each news cluster.

It is assumed that since a news headline is a convolution of a news report and texts inside each cluster in question describing the same event, each cluster or at least the greater part of each cluster it will be represented by a single connected component in paraphrase graphs. The scope of our method can be evaluated by the number of the outlying vertices (headlines). In other words, to evaluate the appropriateness of our paraphrase graph construction method for the problem of news clustering, we calculate the portion of the vertices inside the largest connected component for each of the considered news clusters: the closer the portion value is to 1, the better our method is.

4.2 Extraction of Linked Text Segments

As a baseline we analyze texts inside their respective news clusters via their key-words.

When analyzing a text using keywords method, a focus can be on different units of analysis:

In this part of the research we analyze keywords in the context of the text, and the text as a part of the cluster.

The computational experiment in question is based on the calculation of the Dice coefficient. The experiment is conducted in two stages:

1. We calculate Dice coefficient for each pair of tokens for all text collections.

2. We link pairs of tokens together into text segments according to their Dice coefficient values and their context. As we move along the text, we decide whether cur-rent word should join the current text segment. Namely, we do not link the word to the preceding word if Dice coefficient for this pair is lower than the threshold or if it is lower than the average between the corresponding values for the left and the right pairs. As a result of such procedure, we obtain a list of linked segments, calculated for each text independently and then united into a frequency list. In other words, each text becomes marked up with linked segments (word sequences).

As a result of such experiment, we obtain:

At this stage of the research we prove the following hypotheses:

5.1 Results of the Experiment with Paraphrase Graphs

According to the algorithm described in Section 4.1, we have built paraphrase graphs for the news clusters in question. In the current section we provide a brief description of the constructed graphs and error analysis, i.e., analysis of the headlines which are missing from the largest connected components of each paraphrase graph.

We extracted nodes with the largest node degrees from each paraphrase graph. They are presented in Table 1 (top 3 nodes per graph).

It turned out that for each graph central nodes correspond to the shortest headlines. For all the clusters except for the Schwarzenegger cluster, central nodes are brief and informative and represent the compression of the news reports of the clusters. As for the central nodes of the Schwarzenegger cluster, the corresponding headlines are also laconic, but not purely informative, in fact they are publicistic rather than in-formative.

For each constructed graph, we extracted its connected components. We assume that since headline is a convolution of text and the texts inside each cluster refer to the same event, the whole cluster or at least the greater part of it will be one connected component. The outlying headlines, in their turn, reveal the scope of the method.

For each of the five constructed graphs we calculated percentage of vertices covered by their largest connected component (see Table 2). The closer these values are to 100%, the better performance our paraphrase graph construction method demonstrates.

Quite high percentage scores allow us to speak about the validity of the proposed method for the task of extracting thematically homogeneous news clusters. It can be seen that our method shows best results (93%) against the “Schwarzenegger” and the “Paul the Octopus” clusters, while the lowest result is achieved against the “Sobyanin” cluster. According to the results of our second experiment with linked text segments, publicistic headlines are abundant in the former two clusters, while the latter one (the “Sobyanin” cluster) being purely informative, includes several sub-events inside one macroevent, i.e. is the least homogeneous cluster of the five considered ones.

In the five constructed clusters, the outlying vertices are of the main interest to us because they reveal the limits of our clusters extraction method. Let us consider such vertices further in this section.

Firstly, news headlines are often represented by citations or snippets of dialogues - vivid and memorable phrases aimed to attract readers’ attention, which are not the convolutions of the news report (e.g., “У нас есть вакантное место” / “We have a vacancy”; “Медведев выразил сожаление, что у Шварценеггера нет российского паспорта, а то вакантное место есть” / “Medvedev sorry that Schwarzenegger has no Russian passport because there is a vacancy for him”; “Д.А.Медведев: Опыт Сколково надо тиражировать по всей стране” / “D. Medvedev: Skolkovo experi-ence should be …”)

In the “Schwarzenegger” cluster the headlines are often references to the “Terminator” movie (which is not surprising since A. Schwarzenegger is mostly known to the Russians thanks to this movie), e.g., “Восстание машин” / ‘Rise of the machines”, “Иногда они возвращаются” / “Sometimes they do come back”.

Short headlines (consisting of 2-3 words) especially without proper names, receive low value of the closeness measure with other headlines, and such low values are insufficient to get into the main connected component, e.g., “Военный мятеж” / “Military mutiny”.

Similarly, very long headlines consisting of a whole paragraph also appear to be outliers in paraphrase graphs: they receive low similarity value with other headlines because of the abundance of mismatched vocabulary and complex syntax, e.g., “На ликвидацию последствий разлива химикатов на алюминиевом заводе в Венгрии потребуется как минимум год работы и несколько десятков миллионов долла-ров” / “It will take at least a year of work and several tens of millions of dollars to eliminate the consequences of chemical spills at an aluminum plant in Hungary”.

Despite the abundance of emotional and attention attracting (purely publicistic) headlines, the “Schwarzenegger” cluster shows the maximal thematic homogeneity while the “Sobyanin” cluster consists of a multitude of events related to the inauguration of Sobyanin as the mayor of Moscow. However, our paraphrase identification model could not divide it into sub-events. Only a part of the sub-events formed separate components, but the larger part of sub-events were stucked to the main one.

It is worth noting that some errors were also initially contained in the news corpora, for example, “Тема недели”/ “Topic of the week”, “На идут уже 331 человек!” / “331 people being sent already!”. However, their percentage is very small (about 0.2%).

Thus, the result of news headlines clustering is influenced by the usage of specific expressions, especially references or emotional words, in different headlines and in different contexts.

General phrases, analogies, opinions and evaluations of events are not amenable to clusters. They have lack of proper names (or contain names of experts which further reduces the value of the closeness measure) and they have very tiny intersection with words that describe the event.

5.2 Results of the Experiment with Linked Text Segments

Results of the experiment with linked text segments have proved our ideas about the differences in the style of the considered news clusters.

At this part of our research we only worked with the three news clusters out of fixe: about the arrival of A. Schwarzenegger in Moscow, about the appointment of S. Sobyanin as the Mayor of Moscow and about the death of Paul the Octopus as they cover all types of news clusters we consider in this paper, and are the most typical representatives of different style.

In the cluster about A. Schwarzenegger the theme of investment and innovations turned to be less important (according to headlines analysis) than the ride of A. Shwarzenegger on the Moscow subway and his posting about it in Twitter: «шварценеггер_прокатился в московском_метро» /Schwarzenegger took a ride in the Moscow metro/ (8 occurrences), «медведев_прокатил_шварценеггера_на "_чайке_"» /Medvedev took Schwarzenegger for a ride in “Tchaika”/ (6 occurrenc-es), «шварценеггер_гуляет в московском_метро и переписывается с_медведевым через_Twitter» /Schwarzenegger takes a ride in metro and corre-sponds with Medvedev in Twitter/ (3 occurrences), «шварценеггер_хочет_вернуться в_кино» /Schwarzenegger wants to return to acting/ (3 occurrences).

Express analysis of the second cluster has shown that it has one main theme: the appointment of S. Sobyanin as the Mayor of Moscow.

The second most important theme is about the retirement of Luzhkov. Such conclusions are proved by the frequency list of linked text segments obtained from the corresponding news cluster: «новым_мэром москвы стал сергей_собянин» /Sergey Sobyanin has become a new Mayor of Moscow/ (7 occurrences), «мосгордума _утвердила сергея_собянина на_пост мэра_ москвы» /Moscow City Duma has approved Sobyanin as the Mayor of Moscow/ (4 occurrences), «сергей_собянин - новый_мэр_ москвы» /Sergey Sobyanin as a new Mayor of Moscow/ (4 occurrences), «мосгордума_ «мосгордума_проголосует по_кандидатуре собянина на_пост мэра_москвы» /Moscow City Duma will vote for Sobyanin as the Mayor of Moscow/ (3 occurrences), «в_москве назначили но-вого_мэра» /A new Mayor appointed in Moscow/ (3 occurrences), «москва_получила нового_мэра» /Moscow got a new Mayor/ (3 occurrences) и «москва_получит нового_мэра» /Moscow will get a new Mayor/ (3 occurrences), etc. There are much fewer news reports with headlines like «лужков_не пойдет на инаугурацию_преемника» /Luzhkov will not attend the inauguration of his successor/ (2 occurrences).

The main theme of the third cluster concerns the death of Paul the Octopus. Other less important themes are his predictions and other moments of his biography as well as the appointment of his successor. This is also confirmed by the list of linked text segments derived from the corresponding news cluster. Thus, most frequent text seg-ments are «умер_осьминог_пауль» /Paul the Octopus died/ (30 occurrences), «умер_осьминог-предсказатель_пауль» /Paul the Octopus, the predictor, died/ (20 occurrences), «в_германии_умер осьминог_пауль» /Paul the Octopus died in Germany/ (19 occurrences), others are quite similar: «скончался_осьминог_пауль» /Paul the Octopus deceased/ (9 occurrences), «знаменитый_ осьминог-предсказатель пауль_умер» /Famous Octopus-Predictor died”/ (6 occurrences), «в_германии_умер осьминог- предсказатель_ пауль» /The Octopus-Predictor died in Germany/ (5 occurrences). Only some of them convey some additional information, for example, «мемориал_ осьминога_пауля установят в_его родном_ немецком_океанариуме» /Paul the Octopus Memorial will be installed in his native German oceanarium/ (5 occurrences).

All the text segments were extracted using a program by Daudaravičius [³]. The evaluation of the corresponding algorithm was conducted by its author and it was shown that in the task of keywords extraction it helps to increase F1 by 17-27% de-pending on the data.

Some of the segments we obtained during express analysis of the clusters consist of 3 and more words, and such segments reflect the distribution of levels of importance of different structural components of the text. It correlates with the idea that the addressee of the texts tends to operate the largest operational units (see [¹¹]). The above supposition is part of the hypothesis that heterogeneous structure of a news text is reflected in the markup of the segments which are most important for the addressee’s perception (see [⁸, ¹⁰]):

Dynamic sketchy style of the most texts about A. Schwarzenegger is similar to that of Twitter. It is characterized by short structures and thus has short linked segments. There is no strict compositional structure of a plot although the text is certainly close to the narrative. We believe that the cluster about A. Schwarzenegger most clearly reflects the differences between the data obtained from the headlines and from the bodies of the news reports. Even on the baseline level of the linked segments there is a clear tendency: headlines are publicistic (trying to affect the addressee rather than share the information) while the bodies are informational and sound “more quiet”. Thus, the list of linked text segments gives us a useful insight into the news cluster in question.

In the cluster about S. Sobyanin there is no such striking contrast between the segments obtained from the headlines and the bodies of the news reports.

Texts are static, they tell about a single event and the main theme stands out very clearly. Linked segments derived from the headlines consist of informationally important fragments as well as the publicistic ones. Linked segments derived from the bodies of the news reports allow us to get an idea about important additional subtopics (e.g., the party path of Sobyanin, voting circumstances, etc.).

In the cluster about Paul the Octopus the intersection between the segments de-rived from the headlines and those from the bodies is minor. The texts in this cluster are static, they narrate about a single event and at the same time are full of various details which form the event and their compositional structure is not conventional.

There could be two reasons for such disconformity: firstly, a bright publicistic style both in the headlines and the bodies of the texts (which forces to abandon clichés and use a variety of design options for the messages), and secondly, the abundance of details, i.e., topics and subtopics, which are difficult to attribute weights of informational importance. Thus, if we consider lists of linked segments as the compressed text of the cluster, then the cluster about Paul the Octopus is the most complex one among the three considered clusters.