1 Introduction

Arabic WordNet (AWN) ^[2,12,5,6] is a lexical database for the Arabic language. It has been developed following the development process of Princeton WordNet (WN) ^[15,24,25] and Euro WordNet ^[36]. There is a degree of uncertainty around the term wordnet. It was created as a lexical database, however, some researchers refer to it as a semantic network because of its hierarchy in the way entities are seen as nodes and connected with edges ^[16]. Others define it as an ontology regarding some specific relations between conceptual categories ^[9]. However, these denotations cannot disguise the fact that many wordnets represent the largest publicly available lexical resource for many language. AWN is one of them. It is widely used in various fields of Natural Language Processing (NLP). Therefore, a considerable volume of research has been published to improve its content, in term of quantity ^[1,3,29] and quality ^[2,9]. Along with this growth, however, there is increasing concern over the incompleteness and lack of the relationships in AWN.

These missing inter-relationships should be able to be addressed through the development of auto-reasoning within AWN. Auto-reasoning is most commonly seen in people, in what is usually called 'commonsense reasoning'. For instance, if a new species of plant is discovered, we do not have to specify that it has leaves. That is learned 'commonsense' knowledge which people can impute from existing knowledge without reference to external resources, otherwise defined by Davis et al. ^[10] as a simulation of how humans think and react in their every-day situations. It is therefore essential to look for missing relationships to improve AWN.

Much of the current research is focused on enhancements of existing knowledge bases (KB), using patterns or classifiers applied to external corpus, regardless of the type of KB ^[39,21,37]. This approach has limitations however, given that not all the embodied knowledge in a text is explicitly presented, and particularly, in respect of the Arabic language, as it is a low-resource language.

Our major objective is to find a way to predict new relations from existing ones in the AWN. To do this, we embrace the Neural Tensor Network (NTN) proposed by ^[32] because it gave better results when they applied it to WN and Freebase ^[7]. Also, It uses only the database for the prediction, no external resources. NTN operates by transforming entities into vectors - one vector representing the distribution of an entity in the database and its relationship with others. Relationships between entities are then expressed using a group of parameters within the NTN. This word-embedding feature makes the model more accurate when compared with other models.

We have structured this article in six sections, including this introductory section. Section 2 will describe the AWN and what version are we going to work on. Related works will be detailed in section 3. Section 4 will describe the word embedding technique that we use and the NTN model. An outline of some of the tests used and the evaluation of their results is provided in section 5, with our conclusions following in section 6.

2 Overview of Arabic WordNet

Arabic WordNet is a large lexical database of Arabic, which is available for public use, free of charge¹. It comprises 5 parts of speech: nouns, verbs, adverbs, adjectives, and adjective satellites, which are categorized into groups of synonyms called synsets. Each synset expresses a distinct sense or concept. They are interconnected by semantic and lexical relations. It is freely and publicly available for many tasks of NLP concerning the Arabic language. Currently, there are two versions of AWN. The version we are interested in is the LMF version. It is structured under the LMF standard, which makes it a practical tool for computational linguistics.

We find in the LMF Version² 60,154 entities in total. Most of them are verbs (42,298 entities) and nouns (16,432 entities). The rest are adverbs with 771, adjectives with 270, and adjective satellites with 386. With this variety, entities are connected with only 41,135 relations (5 type of relations). Our attention has been drawn to make progress in this version since it is rich in terms of entities but not in term of connection between them. First, we are going to cite some interesting works in this field.

3 Related Work

There is a large and growing body of literature which has been dedicated to an exposition of the incompleteness of existing KBs. The open-source KB, Freebase, before it closed, only held information regarding the nationality and birthplace of a small minority of those listed on it (25 and 29 percent, respectively) ^[11]. Studies of other KBs such as DBpedia ^[4] and YAGO ^[13] reveal that up to 99 percent of categorizes lack at least one property that others in the same class possess ^[34]. Galarraga et al. ^[17] discovered in 2016 that Wikidata knows the father of only 2 percent of all the people in it. For that, many attempts have been made to address this issue ^[14,31]. Most of them used external resources in order to mine relevant rules such as if X has children then probably X is married. Thus, they can impute new facts and relations between entities. Use of externals resources is an important factor but this can pose problems for AWN, given the low-resource nature of the Arabic language when compared to other like English.

There is nonetheless a significant minority of projects which have focused solely on improvements to a KB itself. In their major study, Nickel et al. ^[26] presented RESCAL, an approach based on the factorization of a three-way tensor. The main idea is to match the latent semantics of entities and relations, where a KB was regarded as a three-dimensional tensor. The matching score of a triplet is given by a bilinear function. Similarly, Jenatton et al. ^[20] proposed another tensor approach to model multi-relational datasets. It transformed entities into vectors and set a matrix for each relation. To reduce the number of parameters and avoid overfitting, relations were represented as a combination of latent factors. As Jenatton et al. observe, the bilinear structure of a latent factor model can reveal unseen shared similarities between different relation types.

The impact of tensor factorization has been explored in several studies of NLP. Setiawan et al. ^[30], He et al. ^[19], and Qiu et al. ^[28] proposed a different type of neural network that includes tensor decomposition. Sutskever et al. ^[35] proposed the Bayesian Clustered Tensor Factorization (BCFT). Their principal objective was to find interpretable structures within a KB and predict the accuracy of unobserved relationships within that KB. This model clusters entities and relations to share statistical strength through a three-way interaction using a bayesian non-parametric method. Socher et al. ^[32] found a way to predict new relations based on others. They developed the Neural Tensor Network (NTN), which is a tensor decomposition in a neural network architecture. It has been applied to WN and Freebase. Each relation in the two resources has its own tensor parameters and each entity is represented as an average of its constituent word vectors in high dimension.

Socher et al. also claim that representing entities with a single vector does not allow the sharing of statistical strength between similar entities. For instance, 'living room' and 'dining room' are two entities that have much in common, yet previous approaches are likely to embed each one separately. So, they embedded each of 'living', 'room', and 'dining' and then built the representations for entities as the average of the vectors of the three words. For that purpose, they initialized word vectors with pre-trained vectors using a word embedding model, demonstrating that this can increase the reasoning accuracy. In the same vein, Bordes et al. ^[8] proposed the Structured Embedding model. The main idea is that two entities of a correct triple should be closely related to each other in the relation space. Hence, two entities are defined by two (-head, and tail) relation-specific matrices and the score function correlates with the degree of relationship between the words.

While there is are many different models for word embedding, all of which work in their different ways, ultimately, in every model words will be represented as vectors in a continuous space. In 2013, Mikolov et al. ^[23] developed the famous word2vec, a model for word embedding with two different architectures, the continuous bag of words (CBOW), and the skip-gram (SKIP-G). The difference between them is that the first one learns to predict the word from its context and the second one learns to predict the context from the word. But, either way, they both represent the word in vector space according to its local context. CBOW is faster to train and works better with frequent words. SKIP-G performs well with a small training set and works with rare words.

A year later, Pennington et al. ^[27] presented an alternative model called GloVe. The major difference between GloVe and word2vec is that it leans by constructing a co-occurrence matrix: in other words how frequently a word appears in a context. In several cases, word2vec has proven to be a more successful algorithm for unsupervised learning of semantic similarity and relatedness, when compared with GloVe ^[22].

Recently, in 2016, a group of researchers on Facebook proposed fastText ^[18]. Rather than a word by word approach, like word2vec and GloVe, it breaks words into n-grams. For instance, the tri-grams for the word fruit is fru, rui, and uit and the vector of the word will be the sum of the n-grams. This approach provides a significant advantage in respect of rare words, and words that are not presented in the training set. The authors claim that this outperforms word2vec since it can present a vector for any word, something that neither word2vec nor GloVe can do. However, research has consistently shown that both these models require large datasets for the training process.

In a major study concerning the Arabic language, Zahran et al. ^[38] built a vectorized word embedding for the Modern Standard Arabic (MSA) using CBOW, SKIP-G, and GloVe. They also evaluated them: intrinsically, using the task of standard word similarity: and extrinsically, using two NLP application, Information Retrieval, and Short Answer Grading.

Soliman et al. ^[33] presented the AraVec, a set of pre-trained distributed word embedding for the Arabic language. Its first version has six word embedding models trained on three datasets (Wikipedia, Twitter, and texts from Arabic web pages) with more than 3,300M tokens. The second version now contains 12 models. The main idea is to present each one of the datasets with the two models of word2vec (CBOW and SKIP-G). This outperformed fastText because it has been trained on a large dataset includes tweets (Arabic dialects).

The most obvious finding to emerge from these studies is that the tensor approach is very effective in predicting accurately missing information in linguistics resources. Also, this has enhanced our understanding of representing entities as feature vectors to preserve the knowledge of the original data and present the interesting ability of generalizing new reasonable relations.

4 Proposed Model

In this section, we focus on how to complete missing relationships between words in AWN. As shown previously, the neural tensor network (NTN), alongside word embedding techniques, arise naturally with the representation of multi-relational data, like AWN. For that, first, we prepare all the entities in AWN and then build the model by training it and adjusting its parameters.

4.2 Neural Tensor Network

We adopted the NTN model provided by Socher et al. ^[32]. Not only has NTN performed well when used with NW and Freebase, it also outperforms other neural networks, such as the single and bilinear networks in tests, due to its ability to link entities across multiple dimensions.

As we can see, figure 1 represent entities e_n and relations r_k as triples. This is called a three-way tensor. We can tell that a slice of the tensor is defined by a matrix when r is fixed. If we have two entities e₁ and e₂ with a relation r, the score of their relationship is a function s(e₁, r, e₂):

se1,r,e2=urTfe1TWr1:ke2+Vr,e1e2+br, (1)

Fig. 1 Illustration of the tensor model 

F is a tanh nonlinear function. e1TWr1:ke2 is a tensor product where W1:k is the tensor and k represent the slices of the tensor. U_r, V_r, and b_r are the parameters of a standard neural network. The non-linearity of this model is the main advantage because it joints the entities directly through multiplicity. Each relation r has its own parameters as it is illustrated in the figure 1. A visualization of the tensor layer is shown in figure 2 provided in ^[32].

Fig 2. Neural tensor Layer parameters with 2 slices 

The model needs to be trained. This yield to adjust all the parameters U, E, W, V, and b by minimizing the hinge-loss function 2. The main idea is to give a higher score to a positive triple e1i,ri,e2i than a negative triple e1i,ri,ec , in which e_c is a random entity:

JΩ=∑i=1N∑c=1Cmax⁡0,1-sTi+sTci+λΩ22. (2)

With Ω represents all the parameters of the network. 1 is the margin and s is the score.

N and C are the set of positive and negative samples, respectively. Ti and Tci are the positive (correct) and negative triple, respectively. λ is the regularization parameter to prevent overfitting. To update these parameters and calculate ∂J/∂Ω, we use the backpropagation algorithm.

5 Test and Evaluation

The concept is easier to understand using an example. If ³ (Bosporus) is a (strait) and a is a (canal) then is a too. This is simply how the model will predict new relations in a transitive way. To make the semantic and lexical sharing stronger, entities should be presented with as many relations as possible. For that, we keep only the accurate ones that are presented with more than 2 relations. In total, we have 41,122 connected triples with 5 type of relations. We did not work with the antonym relation, and we concatenated the hyponym/hypernym (is a) and HasInstance/isInstance (instance) as one relation, respectively. We end up with 3 types of relations and we split the data as showed in table 1.

We do not have a large data to be able to train the model perfectly, so we used the k-fold cross validation (we choose k = 10) to better entertain the training and the test sets. For each k, we combine the training and validation sets and run the algorithm once again. Also, we notice that many words do not have any relations at all. So, we created manual associations with at least one relation between a word and a triple and had that verified by a lexicographer. To add more precision to the model and to force it to focus on harder cases, we add some of them to the validation and the rest to the test set, since the main goal is to predict new relations. Negative examples are created randomly by switching one entity in a correct triple in the training set.

Another matter that needs attention is the complexity of the model. The complexity is represented by the number of the parameters within the tensor that need to be trained. It lies exactly in the tensor. To reduce complexity in the model, we choose to work with only 3 slices of tensor. For instance, if the dimension of the entities vectors is d=100ei∈R100 then the tensor WR1:k∈Rd*d*k10000*k and the standard parameter VR∈R2*d2d*k . In the end, with 3 slices, we have over 30,000 parameters to train. For that, we could not add a new slice otherwise we have to add new relations between all the entities.

In general, the accuracy of the model is based on how correct new triples are.

The disparity as shown in table 2 lies in the difference between the relations in terms of the number of triples in the training set. There is also the complexity of the relation. For instance, it is easier to reason for example over the is a relation than the instance. Furthermore, the vector representation takes part of this disparity.

Table 3 show how much the initialization with pre-trained vectors help.

We tested the model with a random initialization and the two sets of AraVec: 100 and 300 dimensions. Both of the distribution of AraVec gave almost the same results but the random initialization showed lower accuracy. This due to the multi-word expressions presented in the AWN. We considered by presenting the multi-word expressions as the average of the words that combine them it will strengthen the semantic sharing between them. We can explain the low accuracy by the lack of the triples in the training set. Most importantly, this model can reason without recourse to external resources.

6 Conclusion

Our research addressed the incompleteness of the Arabic WordNet using the natural tensor network as a tool. Results achieved using the NTN have improved through the incorporation of pre-trained word vectors. The intersection between relations made via a tensor layer. The study has demonstrated meaningful results extending the number of relations within Arabic WordNet. Such an automated process inevitably brings with it the risk of adding wrong information. Further research will be focused on error detection in the Arabic WordNet and an investigation of the quality of its information.