2.1 Creation of the Corpus

Wikipedia in Spanish gathers a total of 1,636,082 articles, annexes and categories, ranking it in ninth place in Wikipedias based on the total content pages. Figure 2 shows the amount of articles in the Spanish Wikipedia over the years.

Fig. 2 Spanish Wikipedia statistics 

This makes Spanish Wikipedia a great source of information, which allows us to construct a corpus using this information, limiting it to the domain of computer science and related areas.

WikiExtractor^fn was used to extract and clean the information from the database of that source. WikiExtractor converts Wikipedia articles into plain text files or json files, which makes it possible to have a set of files free of HTML tags.

This tool is invoked using a Wikipedia dump file^fn. The output is stored in different files, which will contain various articles in xml and json format.

Additionally, it was identified that some Wikipedia articles contain a section called infobox^fn, here the content of the site is summarized allowing the reader to find fragments of information more easily. Taking into account the tabulated nature of this section, questions were generated from the elements it contains. So the main topic is identified by being the heading of the table; the items in the left column are Question Focus; and those in the right column are the immediate answers.

The generation process implements the text analysis described in 2.2 and the algorithm described in 2.3.

With this information, a corpus of 500 questions was generated which were classified according to the question classes of the TREC classification dataset. Table 1 shows the distribution of the questions in these classes which corresponds to Entity, Numeric, Location, Description and Abbreviation, respectively.

It is important to note that the 500 questions generated are simple questions, that is, they have a single interrogative term. Multiple questions like: ¿Cuándo apareció Python y cuáles son las extensiones comunes? (When did Python appear and what are the common extensions?), ¿Cuál es la última versión estable de Python y por quién fue influido? (What is the latest stable version of Python and by whom was it influenced?) or ¿Cuándo apareció Python y cuáles son las extensiones comunes? (When did Python appear and what are the common extensions?) were generated manually by combining two or more elements of the corpus.

Therefore, to create a multiple question, the selection of between two and three questions on the same topic is used as a starting point. Of the two questions shown in Figure 3 the interrogative words are identified and positioned in such a way that all the questions maintain coherence among themselves, being separated by commas and the conjunction y. Followed by each interrogative word, the focus of the question is embedded in case the focus of each question is different (3a). On the contrary, if the focus of the question matches with the others, it is added after the last question word (3b). Finally, the main topic or entity on which the questions are being asked can be added immediately after the first question focus (3a) or at the end of the sentence (3b).

Fig. 3 Multiple question construction 

Taking this into account, two resulting corpus were obtained: the first of these is composed of simple questions, which was used to evaluate the classification of questions in Spanish using a Convolutional Neural Network; and the second corpus, which consists of 1140 multiple questions, was used to evaluate the module of reformulation of complex/multiple questions.

2.2 Analysis and Treatment of Questions

The objective of this stage is to apply Natural Language Processing techniques to identify the elements present in a multiple question in order to reformulate it into individual questions for classification with a Convolutional Neural Network. This stage receives as input a multiple question from the corpus described in the previous section. This process begins with the extraction of the grammatical category of each word, as well as the parallel execution of the tokenization and the Named Entity Recognition and Classification using Freeling [²⁰].

The tasks already mentioned provide fundamental information for each of the different grammatical categories. For example, Table 2 shows the PoS tag assigned to each word by Freeling based on the categories proposed by EAGLES^fn for tagging. The information obtained allows the module to identify the different PoS tags and thus be able to reformulate the multiple questions into individual elements. As an example, the tag VSIS3S0 provides the following information: (V ) determines the category, in this case verb; The next (S) represents the type, semi-auxiliary; the third position (I) corresponds to the mood, imperative; later the tense is indicated(S), past; immediately the person is specified (3), third; followed by this, if plural(P) or singular(S); finally, the gender (0)^fn.

On the other hand, for the NERC, in addition to Freeling, a heuristic of item skip parsing was developed, improving the the Named Entities recognition. This heuristic was also used for the generation of the individual questions.

2.3 Question Reformulation

Once the above information was obtained, an algorithm was developed to classify the grammatical elements in the sentence, as well as for the generation of adjectives, determiners, pronouns or adverbs, identifying the gender, number and tense from the PoS labels obtained, this allows to create a correctly structured sentence.

In addition, the algorithm performs a search for keywords throughout the sentence, these, in most cases, correspond to the intention of the question based on the interrogative term. This allows identifying in the first instance the possible number of questions in the current sentence.

In addition, considering that a certain type of question can use different keywords, lists of words were created according to the type of question, for example: fecha[cuándo, fecha, año, mes, día]], ubicación[dónde, lugar, país, sede]], among others.

Once the previous information was obtained, it was necessary to locate the Focus of the Question, which is a statement or phrase that makes the question flow. This element must be related to the content or the desired results. For this, the algorithm LocalMax [⁸] was used on the corpus built. LocalMax algorithm assumes that Multi-Word Terms (MWT) have a high rate of adherence to each other.

The authors propose an association metric called Symmetrical Conditional Probability (SCP) to measure the correlation between two words using Equation 1:

where p(x,y); p(x); and p(y) are the probabilities that the bigram (x,y), the unigram p(x) and the unigram p(y) appear in the corpus respectively.

To generalize this measure of n-grams, the authors introduce the Fair Dispersion Normalization, that splits an n-gram w1,w2,……,wn at different points of dispersion and considers it as combinations of the two parts.

FDS uses the average of the products to normalize the association measure for a given n-gram (Equation 2). To measure the cohesion between words in an n-gram, the average of the products for the two parts at different points of dispersion of the n-gram is calculated (Equation 3):

where n is the length of the n-gram and p(w1,……,wn) is the probability of the sequence of words wi.

Based on FDS, the LocalMax algorithm tries to find an n-gram with the highest SCP__f that any (n − 1)-gram in this and any (n + 1)-gram containing it, allowing to identify MWT and with that avoid the loss of information. To this process the heuristic of item skip parsing is added to this process using stop words, PoS and NERC tags.

Once classified, a set of patterns were constructed for the reformulation of questions. This is done by adding new items and concatenating existing ones to give sense to individual questions. In most cases, grammatical elements such as articles, prepositions or determiners are added. These are necessary to generate a coherent sentence, as required. Table 3 shows 8 out of a total of 19 generated patterns.

In Table 3, FIA represents the opening question mark (¿), PT corresponds to the Interrogative Word that determines the type of question, VS,VM, P0, DT,...,n indicate the verb, determinant or information to generate a coherent sentence. In the case of infoboxes: QF, NE y MWT Question Focus, Named Entities and Multi-Word Term respectively, are determined based on the position in the table, the left column (QF) and the infobox title (NE); finally, FIT represents the closing question mark (?). The importance of this reformulation lies in the individual analysis of words to identify enough information to generate coherent questions, respecting aspects of gender and number agreement, as well as the proper use of verbs, determiners, prepositions, among others. Additionally, this allows us to eliminate words that are not strictly necessary in individual questions.

2.4 Word Embedding Creation

Taking into account the need to use word vectors in Spanish, it was decided to use Spanish Billion Word Corpus [⁴], which contains around 1.5 billion words collected from different information sources such as: SenSem, Ancora Corpus, Tibidabo Treebank and IULA Spanish Treebank; the OPUS project comprising: the books aligned by Andras Farkas, the compilation of legislative texts of the European Union by the CCI, the News Commentary corpus and United Nations documents compiled by Alexandre Rafalovitch and Robert Dale; the Spanish portion of the European Parliament by Philipp Koehn; and finally, snippets from Wikipedia, Wikisource and Wikibooks up to 2015, using Wikipedia Extractor.

Therefore, the process for its creation was replicated using FastText [³], which is a library written in C++ for the efficient learning of word representations and sentence classification. FastText allows you to train supervised and unsupervised representations of words and sentences. These can be used for different data compression applications as features in additional models, for candidate selection or as initializers for transfer learning [⁹].

In this work, the Skip-gram model was used using negative sampling and softmax. This tool achieves good performance for the aforementioned tasks, but also in the case of rare words using information at the character level.

2.5 Question Classification Method

The importance of identifying the type of question that will be processed by a QAS is fundamental for this task, as shown [⁷, ²⁷, ²⁸, ¹⁶, ¹⁹, ¹⁸, ³¹, ¹², ¹⁷, ³²]. Consequently, a Convolutional Neural Network model was used to classify questions in Spanish.

To understand how a CNNs work in [¹, ²³, ²³] this architecture is described, in which each layer of the network is three-dimensional, having a spatial extent and depth corresponding to the number of features. The notion of depth of a single layer in a CNN is different from the notion of depth depending on the number of layers.

In the input layer, these features correspond to color channels like RGB (red, green, blue), and in the hidden channels these features represent hidden feature maps that encode various types of shapes in the image. The architecture contains two types of layers: convolution and pooling.

For convolution layers, a convolution operation must be defined. Afterward, a filter to assign triggers from one layer to the next is used. A convolution operation uses a three-dimensional weight filter with the same depth as the current layer, but with a smaller spatial extent. The dot product between all the weights in the filter and any spatial region options (the same size as the filter) in one layer defines the value of the hidden state in the next layer (after applying a trigger function like ReLU). The operation between the filter and the spatial regions of a layer is performed at every possible position to define the next layer (in which the triggers retain their spatial relationships from the previous layer).

Generally, in the Natural Language Processing task for text classification [¹³] uses filters that move over entire rows of a word-based matrix. Where, the width of each filter is usually the same as the width of the input array. The height or size of the region can vary, but the sliding of the windows is usually two to five words at a time.

In this research, different sets of questions of p words and the length of the input sentences defined by n words were used, where each word is represented by a vector, in this case a Word Embedding in Spanish language described in 2.4.

Therefore, an array of n⋅⋅k is obtained, where k corresponds to the length of the Word Embedding (300). To this, the batch is added and it is denoted by b, which means the total number of inputs (questions) that the neural network will process simultaneously. A padding is applied to all those sentences that do not match a previously defined maximum length. For the convolution operation, the aforementioned input n⋅⋅k is used, now called x, and a matrix of weights W(m⋅⋅k), which generates as output a vector h that is obtained with the Equations 4 and 5:

In turn, different filter sizes are defined that are executed in parallel. Each convolution outputs a hidden vector of dimension 1⋅⋅n, which is concatenated to generate the input of the next layer with a dimension q⋅⋅n, where q is the number of parallel layers that will be used. Therefore, assuming the output of the layer h is of dimension q⋅⋅n the pooling layer will produce an output of size q⋅⋅1 denoted as h′′ (Equation 6):

Figure 4 shows the architecture used for this Convolutional Neural Network. This shows the vector corresponding to the tokens of the input sentence, followed by the Word Embedding used to calculate the vector of real value for each word using its 300 dimensions.

Fig. 4 CNN architecture 

Subsequently, the convolution and max pooling layers to finally reach the fully connected and softmax layer. With a total of 100 filters with dimensions of sizes 2, 3 and 4, an embedding layer dimension defined at 300, with an input size defined based on the vocabulary size, a dropout of 0.5, applying 2D convolutions. In this way, the neural network will be able to classify the Spanish questions.

Of the aforementioned datasets, only TREC focuses on the question classification task, therefore, the classes defined for this dataset were: Entity, Description, Person, Numeric, Location and Abbreviation.

The rest of the datasets were classified manually and with the help of an algorithm based on the use of keywords for each category. Table 4 shows the total elements per class for each dataset.

3.1 Question Reformulation Evaluation

To evaluate the reformulation of questions, 250 questions in Spanish were randomly selected from the corpus of multiple questions generated in 2.1. To measure the effectiveness of the method, the following metrics were selected based on the literature. Although these were originally created to evaluate other tasks of the NLP, they have achieved good results in this task. These metrics perform the calculation considering a reference sentence and a candidate sentence.

3.2 Question Classification Evaluation

Taking into account the small amount of information in Spanish for training the Convolutional Neural Network, each of the aforementioned datasets was translated with the help of online tools.

For the evaluation, the CNN was trained individually for each corresponding dataset for 35 epochs. Figure 5 shows the Accuracy and Loss values at each epoch for each dataset in the training and validation sets.

Fig. 5 Accuracy and Loss obtained in training and validation of each dataset used 

In Figure 5a the corresponding accuracy is shown. Here it can be seen that the set with the highest value was Curated TREC in the training set at 100% throughout the 35 epochs, which means an overfitting in the Neural Networks. On the other hand, the WebQuestions validation set had an accuracy above 80%.

Also in Figure 5b the loss is shown for each training and validation set according to the corresponding dataset in the 35 epochs. Here it is possible to see that the set with the lowest loss was Curated TREC in training, which could support that the model has been overfitted. On the other hand, the SimpleQuestions validation set was the one with the highest loss.

Below, a set of Figures (6-10) are shown corresponding to the confusion matrices for each dataset respectively in the test set, this tool allows to visualize the performance of the implemented CNN. Each column in the matrix represents the percentage of predictions for each class, while each row represents the instances in the actual class. This tool allows us to see if the system is confusing which classes.

Fig. 6 Curated TREC confusion matrix 

Fig. 7 SimpleQuestions confusion matrix 

Fig. 8 TREC 10 confusion matrix 

Fig. 9 WebQuestions confusion matrix 

Fig. 10 WikiMovies confusion matrix 

Figure 6 shows the corresponding confusion matrix for the Curated TREC dataset. Each class is represented by the corresponding number: {’NUM’: 0, ’LOC’: 1, ’PER’: 2, ’ENT’: 3, ’DESC’: 4, ’ABR’: 5}. Here it is possible to see that classes 0, 1 and 2 are those with the best classification results. However, classes 3, 4 and 5 present a deficiency in the classification, this is due mainly to the syntactic similarities between each type of question. In future, we plan to apply syntactic n-grams for analysis of these similarities [²⁶, ²⁴, ²⁵].

For the SimpleQuestions V2 dataset the following notation was used: {’ENT’: 0, ’PER’: 1, ’LOC’: 2, ’DESC’: 3, ’NUM’: 4}.

Omitting class 5 due to the absence of elements. Unlike the previous figure, Figure 7 shows more uniform results in the classification of classes 0, 1, 2 and 3. However, the classifier presents a significant error in class 4.

The confusion matrix shown in Figure 8 corresponds to the TREC 10 dataset. The following notation was used in it: {’DESC’: 0, ’ENT’: 1, ’PER’: 2, ’NUM’: 3, ’LOC’: 4, ’ABR’: 5}. For this dataset, classes 2, 3, and 4 had a lower error rate. However, class 5 had a clearly questionable performance, this is associated with the low amount of training data.

On the other hand, Figure 9 shows the results for the WebQuestions dataset, and, as with SimpleQuestions, class 5 is ignored for the same reason. Using the notation: {’ENT’: 0, ’PER’: 1, ’LOC’: 2, ’DESC’: 3, ’NUM’: 4} there is a clear balance between classes. However, class 3 shows a higher percentage of error.

Finally, Figure 10 displays the confusion matrix for the WikiMovies dataset with the notation: {’PER’: 0, ’ENT’: 1, ’NUM’: 2, ’DESC’: 3, ’LOC’: 4}, obtaining good results except for class 4.

Next, Table 10 shows the accuracy and loss values for each dataset respectively, in the validation set. This shows that Curated TREC achieved the highest accuracy percentage with a 98.82%, as well as the lowest loss with 0.044. On the contrary, SimpleQuestions had the lowest accuracy with 87.80% and WebQuestions had the highest loss with 0.347.

Taking into account that the questions refor-mulated in 2.3 may contain errors that interfere with the identification of the type of question, the manual classification and the prediction of the corresponding class was carried out using CNN. Figure 11 show the corresponding confusion matrix.

Fig. 11 Confusion matrix of question reformulation 

Here it is possible to appreciate that classes 2 and 3 are those with less precision in the classification. This is due to the syntactic similarity of both kinds of questions.

3.3 Discussion of the Results

The previously described metrics have shown promising results. For example, both ROUGE and BLEU rely on n-grams to measure the similarity between the output generated by a system (candidate) and the output generated by a human (reference). However, the difference between ROUGE-n and BLEU is that BLEU introduces a penalty term, and also calculates n-gram matching for various sizes of n-grams, as opposed to ROUGE-n, where there is only one size of n-gram.

Furthermore, METEOR has several features not found in other metrics, such as stemming and synonymous matching, along with standard exact word matching. This has reached a good correlation with human judgment at the sentence level. Finally, WER was used in order to identify the range of correct words, substitutions, deletions and the average number of word errors per sentence.

Therefore, taking into account that the average results of these metrics are similar, it is possible to conclude that, in general terms, the reformulation of the questions tends to be closer to the reference sentences, showing good results.

For the question classification it was possible to determine that, indeed, for the uni-class classification, the variables of the length and complexity of the questions have a direct impact on the correct classification.

In the same way, since CNN has not been trained with examples with these characteristics, the multiple questions are not correctly classified. In general, an accuracy of 98.82%, 87.98%, 94.96%, 88.94% and 87.80% was obtained for the Curated TREC, WebQuestions, WikiMovies, TREC 10 and SimpleQuestions datasets respectively.

On the other hand, for the questions that were previously reformulated, the results by class were 96.84%, 90.91%, 87.37%, 82.50% and 96.63% respectively.

However, when a multiple question was submit-ted to CNN, the class in more than 80% of the cases was entity type, whether or not this type of question was present.

This confirms the importance of question reformulation for this research.