nueva página del texto (beta)
Inglés (pdf)
Artículo en XML
Referencias del artículo
Traducción automática
Enviar artículo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
1 Introduction
The available number of documents in electronic form has exponentially grown in recent times. It is impossible for a human to absorb this information to the rate that is produced. The multilingualism present in documents on the Internet does not make the problem any easier. One way to deal with the issues raised are the Automatic Text Summarization (ATS) algorithms [32, 33, 17, 16, 34, 24], a Natural Language Processing (NLP) task. ATS is a compression with losing information. A summarization algorithm aims to compress a source document in order to save their essential content. ATS can produce a relevant synthesis of the documents, giving the reader the decision to consult the source literature.
The production of summaries by humans is a highly subjective cognitive process: not only is it necessary to have knowledge of the language in which the document is written but extra-linguistic domain knowledge is also required in order to produce relevant abstracts.
There are different types of summaries depending on: i) the source documents: single-document, multi-document, monolingual, multilingual, generic, guided, etc.; ii) the genre of texts: news, scientific, encyclopedic, etc.; iii) the audience: expert, general public, etc. Other kinds of summarization are also possible: blogs summarization, tweets summarization, etc. For a more complete state of the art, the reader may consult: [17, 16, 24, 34], among other readings.
ATS algorithms, despite not generating quality summaries comparable to those produced by humans, they are capable of generating exploitable summaries that give a reduced view of the source document.
The term “exploitable” means that the automatically generated summary has certain qualities that make it interesting:
It has to be shorter than the original (measured in characters, words, sentences, etc.).
Good informative content coming from the source document (the informativity).
It has to be coherent, grammatically correct and readable (the form of the document).
However, the two latter qualities are highly subjective. How to correctly measure the informativity and the form at the same time? The answer to these questions is still open however partial answers exist [16, 8, 23].
In ATS two processes are clearly differentiated: a) the production of –relevant– summaries, and b) the –correct– evaluation of these summaries. It is not enough to create ATS systems, it is necessary to correctly evaluate the production of such systems. Process (a) presents several intrinsic difficulties: the “comprehension” of the text, the selection of the informative segments, their assembly and their possible rewriting.
The production of summaries can be realized by extraction or abstraction approaches. The extractive approach selects the most informative segments from the source, then the algorithm assembles them into a summary [15, 1, 3]. The abstractive approach aims to generate new and informative texts from the source document [16, 34]. Nowadays, most systems use an extractive approach because the algorithms for extracting sentences have shown technical superiority against complex methods of abstract generation (based on rules or others).
Process (b), quality summary assessment is a task with its own difficulties [24, 16]. In general, summary evaluation can be performed manually (reading and direct evaluation by human judges), automatically (comparing artificial summaries vs. references1) or automatically without references (comparing artificial summaries vs. the source document) [34].
The evaluation with references is dominant in the literature because it presents a good correlation with the manual evaluations [11].
In this paper we are interested in automatic evaluation without references. This way of evaluating has a great advantage: not needing human references or judges trying to analyze large volumes of sources and artificial summaries. In addition, this type of evaluation has shown a good correlation with methods using references [14, 34]. However the evaluation without references has incurred severe criticism.
It can be argued that the evaluation algorithm, being able to evaluate summaries, also –potentially– has the capacity to generate them. Supposing that when generating a summary using the weighting function of an evaluation algorithm and measuring the quality of this summary using the same evaluating algorithm is actually cheating. The aim of this paper is to show that, under certain assumptions, using the evaluation function during the generation process does not guarantee that the optimal summary will be obtained at the same moment when it is evaluated.
The rest of the paper is organized as follows: Section §2 discusses the generation and evaluation of summaries. In Section §3, two proofs show the difference between generating and evaluating summaries using divergence of probability distribution. Finally in Section §4 the discussion and conclusions are presented. Three annexes with linguistic motivation complete the paper: A) the representation of documents as probability distribution, B) the automatic generation of summaries and C) their evaluation.
2 Generating and Evaluating Summaries
Through a classical approach, textual documents can be normalized and represented in an appropriate vector space (see A). In this space, there are algorithms that can calculate a relevant summary of the text and other algorithms that evaluate the quality of the informative content of a summary already generated. In this section we will formally define the tasks of generating and evaluating textual summaries.
2.1 Automatically Generation of Summaries
Let
The extractive paradigm of summary generation [15, 1, 3, 16, 34] is currently dominant in literature. This paradigm involves the selection and extraction of the k most relevant (informative) sentences belonging to the source document
Formally, let a sentence fi = {w1, w2,...} be a sequence of words and a text
A summarizer is a function G that generates a subset
2.2 Automatically Evaluation of Summaries
There are several manual and automatic methods with and without references capable of evaluating text summaries (see C). In this section we only will deal with the automatic evaluations without references. This type of evaluation and its relationship with the summaries generation using Divergence of Probability Distribution constitute the critical point of the paper.
Automatic evaluations without references are based on a measure of divergence between two probability distributions [7, 12, 13, 14]. Two suitable divergence measures are the Divergence, Relative Entropy, or Kullback-Leibler Distance DKL [2] and the Jensen-Shannon Divergence DJS [4]. The divergence between the probability distribution T of the text and the probability distribution A of an automatically generated summary is calculated. The value obtained is a score of the summary information content. Less divergence means greater similarity between
Despite the good empirical results of the evaluation without references [12, 13, 14], a recurrent criticism towards this kind of algorithms is that nothing prevents using the divergence function between probability distributions in the process of summaries generation.
For example, at each step the algorithm could compute the weight Ω of the sentence fi, calculated as a measure dependent on the normalized divergence between the probability distributions of the source text
Supposing that a summarization algorithm G using a sentence weighting Ω with a DPD D. Suppose the divergence D is also used in an evaluation algorithm E. After the generation process, if the summary
3 Textual Divergences using Probability Distributions
3.1 Preliminaries
Let the random variable
This probability p(w) can be estimated by:
where
Evaluating a summary using a measure of divergence D is equivalent to calculating the divergence between the probability distribution of the source text
when the summary consists of k sentences.
In general D(T||A) > 0, intuitively, a value close to zero obtained by the equation 1 implies a high similarity of the summary
Generating a summary requires the use of a heuristic or m-steps algorithm. Heuristics can be constructive (by adding selective sentences) or exhaustive (using a combination of all sentences). The use of an exhaustive heuristic is dealt with section 4 and in B.
If the weighting Ω employs a divergence function D applied on the fi sentences, then generating the summary
This involves calculating the weight of subsets of the text (one sentence or a group of sentences) and selecting the best weighted subsets (with minimum divergence) and concatenating them to produce the required summary.
Supposing that in the m constructive step, k sentences have been retained to form part of the summary
In this article we prove for the asymmetric divergence of Kullback-Leibler [2] and for the symmetrized divergence of Jensen-Shannon [4] that evaluating a summary is different from generating it, even if the same divergence function was used for both tasks.
To prove equation 2 for these DDPs, we need the next three assumptions:
3.2 DDP Kullback-Leibler
Definition 1 The divergence, relative entropy or Kullback-Leibler distance between two probability distributions p and q over the same alphabet X is given by:
Hereafter, for purposes of facilitating reading, we shall agree to write: log2 = log, p(w) = pw and q(w) = qw.
In addition, we also need to establish some conventions: for some words w in
The Kullback-Leibler distance is not a metric, because even though it is true that DKL(p∥q) ≥ 0 with equality if and only if p ≡ q, it also occurs that DKL(p∥q) ≠ DKL(q∥p) and it does not satisfy the triangle inequality.
Despite this, in the area of automatic summaries, the Kullback-Leibler distance is used to measure the divergence between the probability distribution T of a source document and the probability distribution A of its summary, and it is defined as follows:
where
Theorem 1 Let
occurs only when the sentences fi and fj have length zero.
Proof 1 We will calculate first DKL(T||fi ∪ fj). To simplify the notation we will write
then:
On the other hand:
Matching the equations 5 and 6, and performing the operations we obtain that:
implies:
As we can see, the sum of the left side of the equation 7 is calculated on the set
is when:
From the above it is concluded that:
this is when the sentences fi and fj have length zero.
QED
3.3 DDP using Jensen-Shannon
The Jensen-Shannon divergence DJS [4], or symmetrized Kullback-Leibler distance between two probability distributions p on the alphabet
With the same convention (3) used in the divergence DKL.
Use DJS to measure the divergence between the probability distribution T of a text
with the same meanings for T, A, |
Theorem 2 Let
occurs only when the sentences fi and fj have length zero.
Proof 2 From now on, we will write:
We calculate first :
On the other hand, we have:
Matching equations (8) and (9), regrouping terms and doing the necessary operations we obtain:
As in Theorem 1, we can see that the sum of the left side of equation 10 is calculated on the set T\{fi ∪ fj} while the sum of the right side is calculated on the whole set T. Giving that (f1 ∪ f2) ⊂
is when: fi ∪ fj = ∅
From the above it is concluded:
this is when the sentences fi and fj have length zero.
QED
4 Discussion and Conclusions
The main contribution of this article is to show that a DDP used in a summary evaluator algorithm can not be used in a generator algorithm to construct a summary without introducing a bias. In other words, in spite of using the same DDP for both construction (aggregation of sentences using heuristics) and for the evaluation of a summary, it can not be guaranteed that the evaluation of the given summary can obtain the minimum divergence with respect to the source. Hence, it is not unreasonable to use the same DDP in both tasks.
In this article, we have shown that under the three next conditions:
The divergence definition D uses DKL or DJS,
The original document
The summary is obtained by extraction and contains at least 2 sentences,
it happens that D(T ||{fi ∪ fj}) ≠ D(T ||fi) + D(T ||fj).
We have tested it with the Kullback-Leibler probability distributions DKL and Jensen-Shannon DJS. In the case of the divergence DKL, given its asymmetry, it can be considered from the distribution of probabilities T of the text compared to the distribution of probabilities A of the summary and vice versa. In this article, we only consider the case DKL(T||A), but a similar analysis can show that the Theorem 1 is also fulfilled in the case DKL(A||T).
However, an additional case must be considered. In the extraction paradigm, a summary generator G∗ could use an exhaustive combinatorial approach combined with an a posteriori divergence evaluation E. That is, an algorithm G∗(
This number can be large enough even for moderate values of P and S. For example, from a source having P = 40 sentences and a compression rate ρ = 20%, the number of possible extracts of S = 8 sentences is r = 760904, 685. Generating and evaluating them exhaustively in order to keep less divergent summaries is not practical.
Avoiding the exhaustive algorithm, the argument that we have presented eliminates the main criticism towards summarization evaluations without references, in the sense that using DDP during generation and evaluation processes constitutes cheating. We can state that, under conditions mentioned above, evaluating and constructing a summary are two mathematically different tasks, although the DDP used in both is the same.
