2.1 Graphs as a Text Representation Structure

A detailed study of different uses of graphs for natural language processing tasks is explained in [¹⁰], where it presents algorithmic formulations for:

In addition to the algorithms and applications covered in [¹¹] several research works have used graphs for representing text data and solving many natural language processing problems.

In the task of document representation, Pinto et al. [¹⁸] and Gómez-Adorno et al. [⁵] proposed a reliable graph-based representation schema of textual documents that incorporates different levels of formal representation of natural language, and taking into consideration many linguistic levels, such as lexical, morphological, syntactical and semantics and by also extracting useful text patterns in the graph. They state the successful use of their schema in the broader framework of document understanding.

3.1 Graph Structure

A graph G=(V,E) is a structure consisting of a set of vertices V={vi|i=1,n}, some of which are connected through a set of edges E={(vi,vj)|vi,vj∈V}. In a weighted graph Gw=(V,E,W), edges have associated a weight or cost wij. W = {wij|wij is the weight/cost associated with edge (vi,vj), wi,j∈ℝ} . Edges can be directed or undirected.

Depending on the application for the graphics to be used, the nodes and edges can represent a variety of units and links. Nodes can represent text units as basic as words or as complex as documents.

The edges can represent the relationships between these text units, such as: co-occurrence, placement, syntactic structure and lexical similarity [¹⁷].

3.2 Named Entities

NER is an important task in the field of information extraction systems since it aims to locate and classify named entities in raw text into categories previously defined (e.g. Person, Location, Organization). In this way, texts can be represented by their named entities. It is emerged in the Sixth Message Understanding Conference in 1995 [¹²]. Although, sometimes many of the named entities can be ambiguous to be classified in more than one class, e. g. the automotive company created by Henry Ford in 1903, where “Ford“ can be referred to many entities (Name, Company, etc.).

On the other hand, NER systems require a large amount of highly accurate training data to perform well at the task named entities recognition [²⁰]. In this way, excellent training data can be achieved by human feedback, since humans can easily differentiate from one context and another, assigning the correct tag to each named entity in the texts.

4.1 Syntax and Structure

The graph is constructed from a finite set of named entities is stored in a .json file. Thanks to the simplicity of this format, it allows great ease of use for many programming languages and also due to its lightness in data storage. Since the JSON objects are a key-value data format it is convenient to use them to store the information of named entities.

The format used as well as each of the characteristics of the entities that have been employed are illustrated in Figure 1.

Fig. 1 Storage format for Named Entities 

Each node is represented with two curly brackets that encapsulate its properties in key-value pairs populating the space between them. These properties are:

Fig. 2 Possible states of a node in the graph 

4.2 The Process of Construction

The set of named entities that are used as Gold Standard is stored in this graph structure, reading one entity at a time until all are completed. The graph begins with a single root node from which branches are added based on the different categories or classes existing for the set of named entities.

In the process of construction of the graph the following steps are taken into consideration:

Fig. 3 Coupling of named entities by classes and common shared words to reduce the size of the graph 

5.1 Used Corpus

The corpus was constructed by the collaborative web-based tagger tool for named entities developed in [¹⁴]. Since this tool is easy to use and intuitive, allowing to create the necessary classes as well as a fast labeling of documents in plain text. And finally getting the documents labeled in a structured double column format that contains the data prepared to be used in the training of a classifier model provided by Stanford NER [¹⁶]. The first column contains the words or tokens of the document and the second column represents the class to which it belongs if it belongs to one, in other cases the value of the second column is 0.

The corpus is made up of job offers in the field of IT, where these documents can be represented by the most relevant concepts it contains. These concepts are grouped into 6 predefined classes:

All definitions for classes were taken from the Cambridge Advanced Learner’s Dictionary & Thesaurus [³].

5.2 Data Pre-Processing

To carry out the pre-processing of the information, the same software used to label the documents [¹⁴] allows to export them to a structured double-column format, as well as to download the corpus in two data sets: one for training and one for testing (always in a random way, which is perfect for this task.).

It has been implemented considering the principles of V-fold cross-validation method. So, only 25% of the whole data set is taken as validation data for testing the model and the remaining 75% is used as training data (corpus). The cross-validation process is then repeated 10 times, each time with random sets of documents (for both the training data set and the test data set) [²].

5.3 Measures and Training

In order to measure the performance of the NER classifiers, three well known measures are taken into consideration: Precision, Recall and F1. To calculate those measures, first is important to define four main aspects (as shown in Figure 4):

Fig. 4 Parameters to calculate Precision, Recall and F1 

Once knowing the essentials of the different metrics, it is possible to define the performance measures in the following way:

5.4 Experiments

For the training of the classifiers, 75% of the documents were randomly obtained with the collaborative web-based tagger tool and the remaining 25% was used to test the classifier, repeating the process 10 times and using exactly the same sets for the 3 classifiers.

The results obtained for the different classes were averaged by execution and classifier due to the large amount of data, and then contrasted according to the three measurements, these results are presented in detail in Table 1. From the results shown in the table it is possible to observe that the best scores for Precision are always achieved by the CRF++ classifier, leaving far below the graph proposed in each execution. This means that the number of false positives produced by the CRF++ classifier is very low, or in other words when the model predicts a class for a named entity, it is correct on average 75% of the time.

On the other hand, as regards the Recall measure, the highest scores are always reached by the proposed graph, showing a great difference in contrast with the CRF++ classifier.

This means that the number of false negatives produced by the proposed graph is low, or what in other words happens is that the proposed graph correctly recognizes on average 64% of the named entities.

However, the best results obtained in the F1 measure are always achieved by the Stanford classifier, which means that this classifier has a better balance between Precision and Recall, making it the best performing classifier of the three. For a better understanding of these results, the averages have been captured in charts (from Figures 5 to 7) where it is easy to appreciate the variations and levels reached by each classifier.

Fig. 5 Average results in all classes for the Stanford classifier 

Fig. 6 Average results in all classes for the CRF++ classifier 

Fig. 7 Average results in all classes for the Graph Structure Proposed 

The Precision, Recall and F1 measures for the Stanford classifier can be seen in Figure 5, where Precision values are maintained between 65% and 71% in all iterations, with the exception of execution 8, where it drops surprisingly to 56%. This happens because not all documents have the same number of examples for each class. So, in this execution part of the documents that were used for the training provided very few examples of each class in comparison with the other executions.

Regarding the measure of Recall, the values are kept constant in an interval ranging from 42% to 48% throughout the experiment. Finally, the F1 measure remains with small variations between 51% and 56% throughout the experiment, showing good performance for the Stanford Classifier.

The second classifier to be compared is CRF++ classfier, and its results for Precision, Recall and F1 measures are showed in Figure 6. Here can be observed that the performance of the Precision measure is very high compared to the Stanford classifier, being the minimum value reached in iteration 1 with 71% and the maximum value reached in execution 4 with a value of 82%, and the other obtained values oscillate between these two ranges without presenting as major variations. In contrast, the Recall measure is shown too low with a minimum value of 22% in execution 1 and a maximum value achieved in execution 2 with 28%. Presenting even more slight variations in the rest of the executions throughout the experiment. This means that a large number of false negatives is produced by the classifier, or in other words, few truly relevant results were returned. With measures so distant from each other, the F1 measure is maintained in a range that goes from 33% to 41%.

Finally, the results of the graph-based structure proposed are shown in Figure 7. The results are opposite to those of the other classifiers with respect to Precision and Recall since the values appear to be inverted, with a very low Precision but a very high Recall. For the Precision it can be observed that the values remain very similar with a range that goes from 20% to 26%, meaning that, when the classifier predicts a class for a given named entity, it is correct on average 23% of the time. This happens because the proposed model produces a huge number of false positives, so, this model could be improved if a balance of the used classes were achieved.

In the same way for Recall the values remain little variant between 61% and 67%, which shows a more uniform behavior throughout the experiment in comparison with the Stanford and CRF++ classifiers, this means that the proposed model produces a few false negatives. This occurs thanks to the weights assigned in the graph when more than one named entity is similar in the words that make it up and belong to the same class, so the model clusters the named entities similar and by class and manages to correctly recognize a large proportion entities named in each class. Thus, the harmonic average of these measures represented by F1, can be observed without large drops or sudden peaks throughout the entire experiment, and oscillating between 31% and 37%.

In general, it can be observed that the Stanford classifier performs better in the task of recognizing named entities, maintaining a great balance in Precision and Recall. In second place of performance is the CRF++ classifier with a notorious difference below the performance achieved by Stanford.

Finally, the graph structure proposed here is in the last place, but with results not so far from those achieved by the CRF++ classifier. Notwithstanding the above, it is possible to point out that the disk space consumed by the graph proposed here is much smaller than that occupied by the other classifiers, in addition to the training time that is prolonged for the other classifiers. So the proposed graph considerably lighter and faster than the other classifiers.