2.1 Main Approaches for English Textual Entailment

Textual Entailment is treated in several works by assuming that it correlates with the similarity that can be captured at different levels depending on the chosen linguistic representation of the text (e.g. Bag of Words, Structured Representation, Logical Representation…). Lexical similarity methods depend on some measures of lexical similarities using lexical resources (e.g. [¹¹, ¹²] ). Some systems approximated the entailment problem as of checking if a good alignment exists between parts of the HYPOTHESIS and parts of the TEXT(e.g. [¹³, ¹⁴]). More structured approaches compute also similarities at the syntactic level by comparing syntactic representations. A common way to perform this comparison is by computing the lowest cost transformation of T’s representation to H’s representation by executing a sequence of elementary edit operations [¹⁵–¹⁸].

There are significant attempts in the literature to deal with Textual Entailment by transforming the pairs of texts (T, H) into logical representations. In this model, the Textual Entailment is handled in terms of logical entailment. Thus, inference holds between T and H if it holds between their logical representations, plausibly by applying a set of relaxations to make systems robust to errors (e.g. [¹⁹–²²]). Many systems combine logical based approaches with other techniques such as shallow semantic analysis. Boeing Language understanding engine (BLUE) [²³] is an example of this combination.

It uses two strategies implemented by two components on the pipeline. The first one is a logical component and the second is based on a bag of word model that ignore structured representation and acts as a post- process of the logical component to try to test inference validity for pairs that the logical component can’t classify.

Another example of hybrid approaches is that made in [²⁴], where authors extract a set of shallow features based mainly on a set of metrics extracted from a bag of word model, and other features based on a deep semantic analysis technique. Then a machine learning technique based on decision trees is applied to combine features extracted from both methods and produce the final entailment decision.

Machine learning is a crucial element in the architecture of the majority of RTE systems. A dominant approach in the literature casts the problem of RTE as a supervised classification.

Each pair (T, H) to check is represented by a feature vector that incorporates scores of multiple measures applied to the pair at different levels by exploring several aspects of natural language at different levels (e.g. lexical level, syntactic, structural, and semantic level). In order to generate the entailment label, this feature vector is given as input to a classifier trained based on the feature vectors of annotated examples. The definition of the feature space in which we represent the pair (T, H) is the key challenge faced by the machine learning approaches (feature engineering).

With the emergence of the techniques based on Deep Learning applied to NLP, recently promising results were obtained on Recognizing Textual Entailment by exploring Deep Learning techniques and sentence encoding, for instance [²⁵] and [²⁶].

Research in Textual Entailment has led so far to the creation of several datasets for English. For instance: RTE Pascal challenges [²–⁸]; The Stanford Natural Language Inference (SNLI) [²⁵]; Multi-Gene Natural Language Inference (MultiNLI) [²⁷] and SICK (Sentences Involving Compositional Knowledge) [²⁸]

2.2 Arabic Language and RTE Task

Although the considerable work that has been done in RTE, most studies have tended to focus on English. Particularly, a little work handles this task for Arabic. To the best of our knowledge, the first work have been done in 2011 by [²⁹], it investigates the effectiveness of some existing TE approaches when they are applied to Arabic.

The architecture of the implemented system described with more details in [³⁰] was similar to other existing works for English, however, the implementation of each stage attempted to propose some improvements or adaptations in the aim to deal with the problems raised by Arabic.

The main improvement was in the matching component, by extending Zhang-Shasha’s Tree Edit Distance Algorithm [³¹] to cover also operations on sub-trees instead of operating just on the nodes [³²]. Authors proposed also to enhance the results of the preprocessing stage through the combination of outputs of multiple syntactic taggers and parsers [³⁰].

Moreover, [³³] aimed to improve the accuracy of an existing Textual Entailment engine by giving attention to negation and polarity. Indeed, negation and polarity components are integrated into a pipelined architecture with the entailment engine. If the output of the former is "entails" then a set of negation rules are checked, if these rules induce a positive decision (entails) then the polarity component is asked to make the final decision.

Furthermore, [³⁴] experimented existing techniques for lexical and semantic matching to the Arabic language by implementing a classical pipeline including a preprocessing component followed by some similarity measures at the lexical and semantic level to produce the final entailment decision.

An approach to deal with Arabic RTE by using a logical representation of the TEXT-HYPOTHESIS pair based on first order logic is that presented in [³⁵]. The problem of Textual Entailment is cast as a binary classification problem, where the feature vector used is formed on some scores calculated using a set of similarity metrics applied to the logical representation of the text.

To address the lack of resource problems faced by the Arabic language, the work presented in [³⁶] tried to use word embeddings for Arabic RTE. The RTE task was presented as a binary supervised classification by experimenting with multiple supervised algorithms using features based mainly on distributional word representations using word embedding. Another approach to deal with Arabic RTE is an attempt to use text alignment [³⁷].

The problem of RTE is transformed into the problem of finding the best alignment between T components and H components.

This alignment is modeled as an assignment problem that consists of finding an optimal weight matching in a weighted bipartite graph. Then, a supervised classifier based mainly on a set of features that measure the quality of the alignment between T and H is asked to assign the final entailment label.

There is little work on the application of Textual Entailment as a subtask in Arabic NLP application. For instance, [³⁸] attempted to exploit Textual Entailment to improve Arabic text summarization. The entailment engine used is based on cosine directional similarity. Moreover, [³⁹] used Textual Entailment to perform the Arabic claim verification task. The Textual Entailment component developed in this work employed a cross-lingual approach and use an Enhanced Sequential Inference Model (ESIM) trained on a large corpus for English and evaluated in XNLI multilingual corpus [⁴⁰].

Unlike research carried out in this area for English, we did not find significant data for Arabic. To the best of our knowledge, the only resources publically available are the dataset ArbTEDS [⁴¹] and XNLI multilingual corpus [⁴⁰].

2.3 Word Mover's Distance

Word Mover's Distance (WMD) [⁴²] is a distance function that exploits word embeddings, which learn semantically meaningful representations for words, to compute a distance between text documents. It is defined as an optimization problem assuming that the dissimilarity between two text documents can be computed as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document. This distance is cast as a special case of the Earth Mover’s Distance [⁴³], a well-known problem that in turn can be cast as a transportation problem for which several solvers have been proposed.

The documents are presented using nBOW representation to represent the weight or the importance of each word. More precisely, let d1 and d2 be two documents after removing their Out-Of-Vocabulary (OOV) words (i.e., their words that not exist in the word embeddings vocabulary). Firstly, a vocabulary V is created of words contained in d1 and d2, and used to transform d1 and d2 on BoW representations (Word frequencies), and then each word frequency in document BoW representation is normalized by the length of the document. A distance matrix M between word vectors of the vocabulary V is constructed using Euclidean distance. Finally, the WMD is computed as the EMD between the nBow representations of d1 and d2 using the distance matrix M, formally:

3.1 Preprocessing and Enrichment

In the first step, we apply some basic preprocessing operations to pairs (TEXT, HYPOTHESIS) namely: segmenting the text, normalizing temporal expressions and numbers, annotating named entities, extracting lemmas, and removing stop words.

The result of this stage is formulated as a sequence of components belonging to one of the following types: Named Entity (NE), Temporal Expression, Number, ordinary word. Then the TEXT and HYPOTHESIS are represented as a bag of components with their vector presentations in a word embedding model, and each component 𝑐_i is associated with its frequency in the text:

In the following, we will refer to the frequency of a component in a text by the term weight.

To limit the effect of the order less property of the standard Bag-of-words model the tokens associated to named entities are grouped as a single component and temporal expressions and numbers are filtered out and handled separately.

Unlike in WMD, in our text representation, we do not normalize the weight of a component by the total weight of all components of the document (text). In fact, generally, T contains more words than H. Thus, if we make this normalization, a same word that appears both in T and H will have a higher importance in H. This can be a good choice to compare the similarity or the equivalence of documents, but this seems not to be the case for an asymmetric relation such as Textual Entailment.

Furthermore, one limitation of word embedding is that in order to have a vector representation of a word, it must be in the vocabulary of the used word embedding model. To limit this problem, we used lemmatization and we enriched words and named entities with a set of equivalents using external knowledge resources. If a component is not found in the word embedding vocabulary, it will be replaced by its equivalent found in the vocabulary.

Indeed, in addition to synonyms extracted from Arabic WordNet [⁴⁴], we used the free and open knowledge base Wikidata to automatically extract equivalents for named entities. This knowledge base contains data mainly structured as items, each one having a label, a description, a set of aliases, and statements that describe detailed characteristics of an item and consist of a property and a value such as "educated at". We exploited this information, to enrich named entities by their equivalents such as for instance "Donald Trump" can be replaced by "Donald John Trump" or "Donald Trump" and "President of the United States" or even "USA".

3.2 RTE Task as Earth Mover’s Distance

In the Textual Entailment task, the TEXT generally contains more information than the HYPOTHESIS. While WMD distance performs total matching between two documents, in our case of Textual Entailment, we considered that the comparison between TEXT and HYPOTHESIS should be conducted by a partial matching, and this partial matching is allowed in Earth Mover’s Distance. We correlate the problem of recognizing Textual Entailment to the value of EMD between TEXT and HYPOTHESIS representations. As generally, the total weights of the representations of the TEXT and the HYPOTHESIS are unequal we consider the case where weight flows from the heavier representation (the TEXT) to the lighter one (the HYPOTHESIS) until all weight in the lighter representation has been covered.

A formal definition of the Textual Entailment task as an Earth Mover’s Distance problem between the representations of the TEXT and HYPOTHESIS is given in the rest of this section.

Let T and H be, respectively, be the TEXT and the HYPOTHESIS. Each component 𝑡_𝑖 in T (respectively ℎ_𝑖 in H) is given a weight 𝑤_𝑖 (respectively 𝑢_𝑖), in such way, T and H can be defined as follows:

Let W and U be the total weight in T and H respectively (i.e. W=∑i=1mwi and U=∑i=1nui.

And 𝑉(𝑡_𝑖) and 𝑉(ℎ_𝑖) the vector representations of 𝑡_𝑖 and ℎ_𝑖 respectively in a word embedding model.

According to EMD nomenclature, a flow between T and H is defined as a matrix: 𝑓 = (𝑓_𝑖𝑗 ) ∈ 𝑅^𝑚×𝑛 where 𝑓_𝑖𝑗 represents the flow between 𝑡_𝑖 and ℎ_𝑖 (i.e. the amount of weight at 𝑡_𝑖 which is matched to weight at ℎ_𝑖 ).

A flow 𝑓 is feasible between the TEXT and the HYPOTHESIS iff the following EMD constraints are respected:

The work done by feasible flow 𝑓 in matching T and H is defined as follows:

While: 𝑑 (𝑡_𝑖, ℎ_𝑗) = 0 if 𝑡_𝑖 and ℎ_𝑗 are equivalents, else 𝑑 (𝑡_𝑖, ℎ_𝑗) is the Euclidean distance between 𝑡_𝑖 and ℎ_𝑗 vector representations 𝑉(𝑡_𝑖) and 𝑉(ℎ_𝑗)

Finally, we define our measure for capturing Textual Entailment between T and H as the EMD between T and H but without normalizing by the total weights in H, as given by the formula:

While F(T, H) is the set of all feasible flows between T and H. The measure 𝐸𝑀𝐷(𝑇, 𝐻) is the main feature used with three other features to make the final entailment label by a machine learning classifier. These features are:

4.1 Experiments and Result

To validate our approach we used the Arabic Textual Entailment dataset [⁴¹] for evaluation, which comprises 600 (T,H) pairs. This dataset also allowed us to compare the accuracy of our system with other works that used the same data. We preprocessed this dataset using Farasa [⁴⁵] for segmentation, lemmatization, and named entity recognition, and AraTimex [⁴⁶] for temporal expressions and numbers normalization. For word embeddings, we employed n-gram model of AraVec [⁴⁷] useful in our case to retrieve named entities containing more than one word.

Taking into account the small size of this dataset, we proposed two evaluation strategies to avoid biased results and then to have a realistic estimate of the generalization error of the model.

In the first strategy, we split the dataset in such a way that 75% of pairs are used for training and model hyperparameters tuning using Grid Search, while the 25% remaining data is used for testing (held out). In the second strategy, we executed Nested 10 K-Fold cross-validation on the entire dataset to avoid the risk of optimistically biasing our model evaluations and yielding an optimistic score. In this case, first, an inner cross validation is used to tune model parameters using Grid Search and then select the best model. Second, an outer cross validation is employed to evaluate the model selected by the inner cross validation in unseen data.

We conducted our experiments with various classifiers (Extra Trees, Random Forest, AdaBoost, Gradient Boosting, Logistic Regression, SVM) using scikit-learn package. The only results reported are those of the best performing classifier (SVM with RBF kernel). Furthermore, to evaluate the impact of combining knowledge extracted from external resources (Wikidata and WordNet) and the use of word embedding, we considered two configurations of the system in these evaluations:

Additionally, we report results obtained using the same previous evaluation settings, while using WMD instead of our extended measure for Textual Entailment.

Table 1 shows the accuracy results of each evaluation strategy.

To compare with state of the art work, we also reported in the same table, the results obtained by previous systems on the same dataset, these systems are: LR-ALL [³⁶] and ARTESys+ [³⁷] described previously in section 2.

All these results obtained by different evaluation strategies, show the success of our approach based on the Earth Mover's Distance for Textual Entailment. Indeed, the results obtained are comparable or exceed the results obtained previously by the main previous studies on Arabic Textual Entailment. Additionally, results show clearly that our measure performs better in Textual Entailment than WMD created for measuring document similarity, and enhance the accuracy by about 11% in nested cross validation.

Furthermore, results show the effectiveness of combining word embedding and knowledge extracted from resources such as Wikidata and WordNet to enhance the accuracy of recognizing Textual Entailment. In fact, the configuration C.C outperforms significantly the configuration B.C in all evaluations.

4.2 Error Analysis

In our system, we assume that Textual Entailment is in correlation with the similarity measured using EMD between T and H representations. This measure does not take into account the asymmetric nature of Textual Entailment while computing the distance between components of T and H. For instance, in the following example (a) entails (b) while (b) does not entails (a). Our system will label the pair as Entails either if we consider the first sentence as T or as H because the distance computed between the vector representations of the words "assassinated" and "dead" will be always the same:

(a) Ahmed is assassinated.

(b) Ahmed is dead.

We think that accuracy can be improved if we take into account this characteristic of Textual Entailment. Additionally, the enrichment process implemented using knowledge resources Wikidata and WordNet makes it possible to limit the problem of Out-Of-Vocabulary (OOV) words and not to solve it completely. Thus, we think that it is important to quantify the effect of remaining (OOV) words after enrichment described previously. This effect quantification of OOV should be returned by the system together with 𝐸𝑀𝐷(𝑇, 𝐻) and it can be used as an important element to judge the confidence in the entailment decision.

Another type of error rises from the text representation used that ignores extra-propositional aspects of meaning such as modality and negation and some important linguistic phenomena not taken into account by the bag-of- word model such as co-reference and syntactic relations between words.