4.1 Data Preprocessing

The following list shows the preprocessing for this research work:

For the second experiment, we used a stop words list, to not discard combinations. In Table 3, we can see examples of N-grams used in this research work.

5.1 Strategy to Compare the Proposed Approach

In this research work, we would like to compare with other research approaches but authors in [¹²] and [⁷] do not provide a gold standard to compare with our approach. They provide a corpus, so, the strategy to compare our approach with other approaches is evaluating they corpus and our corpus in the same way.

To achieve this goal is necessary an estimation. One form is using a confidence interval for a population proportion. ”A population proportion means the proportion of units in a population that possess some attribute of interest.”^fn We used it to estimate the veracity of our results. The formula used it is (7):

where P is the number of your random sample divided by the total number of the sample rn. And the value 1.96 means the 95% confidence interval for a population proportion.

The formula (7) gives us two values. The values mean an approximate confidence interval, in this case with the 95% confidence.

5.2 Intersection

We implemented Likelihood-ratio positive because we are only interested in positive results. A positive result means an evaluation of the occurrence of an N-gram in the corpus and a negative result is the evaluation that an N-gram does not occur in the corpus. We create a filter derived from the aforementioned measurements, we take the results of each one and we intersect them giving a subset. That is to say, each one has its own range, so only took the best results of each one. We represent the set PMI as set A, Likelihood-radio as set B and Chi-square as set C. Thus we get the following intersections.

In Table 4, we can observe A∩BA ∩ B (see Fig. 1). This intersection between two sets of values PMI and Likelihood-ratio, where both have high values and we see the 10 first trigrams with the highest value. To do the intersection only 50% was taken that is to say one subset from A and another B. You can see the difference in the N-gram competitive salary earn has in PMI 331.944 higher than Likelihood-ratio with 21.049.

Fig. 1 Set intersection 

In Table 5, we can observe A∩C (see Fig. 1). This intersection between two measures PMI and Chi-square, where both have high values and we see the 10 first trigrams with the highest value. In special the term equal opportunities employer start to obtain key terminology. If compared Table 4 with Table 5 some trigrams are removed.

In Table 7, we can observe B∩C (see Fig. 1). This intersection between two measures Likelihood-ratio and Chi-square, where both have highest values and we see the 10 first trigrams with the highest value. We can see that measure Chi-square discard N-grams because their N-grams have low values and when we used different percentages, in view of Likelihood-radio gave us N-grams with higher values, these values were discarded in this intersection.

In Table 6, we can observe A∩B∩C (see Fig. 1). This intersection between three measures PMI, Likelihood-ratio and Chi-square. It is one of the main objectives of this research work, because we can observe how begin to filter the information. You can see the respective measure of each one. When comparing the Tables 4, 5 and 7, we can see that Chi-square measure discarded more N-grams.

5.3 Experiment 1

Experiment 1 presents the set intersection. The set intersection is when the results of each measure match in the ranking and does not comply with the POS patterns for one experiment and does comply with the POS patterns for the rest. This set intersection is between the PMI, Likelihood-ratio and Chi-square measures used to rank terms but without POS filter and order by Likelihood-ratio. We can see the different results in the Table 8 and 9.

In Table 8, we can observe that the measures Chi-square and PMI are not congruent in a descending or ascending form. This is due to the fact that many terms were discarded by the intersection. The Likelihood-radio results are ordered in a descending form but between each value, there is a big difference, this is also due to the fact that N-grams were discarded.

To explain better why N-grams are discarded when the intersection of the three measurements is done. It is necessary to know that an intersection is a subset of other sets, in this case of three sets (measures). We call full intersection to this subset (see Fig. 1).

5.4 Experiment 2

Experiment 2 is defined by the intersection of sets generated by the three collocation measures defined and a POS filter. We also used tokenization with POS tags. The POS filter consists in to verify if the first word is labeled by a JJ or NN followed by any other tag or couple of tags and ending with a tag NNS or NN. For instance, in Table 9 we can see N-grams filtered by discarding mainly verbs.

In Table 10, we can observe the N-grams that did not follow the POS pattern defined. We can see a pattern at the beginning of the N-grams that start with the following tags: IN, VB, VBG or RB. Taking into account this pattern, the filter was created discarding all the N-grams that had that pattern. We called this discarding as POS filter.

It is important to note that we only defined the POS pattern at the beginning and at the end of the N-grams that means that in the middle of the N-grams could be any other N-grams with any POS tag.

5.5 Experiment 3

In the manual labeled corpus also called positive tags. The reason for the name positive tags is because one human expert labeled careful each keyword. The size of the labeled corpus is 50 job descriptions. For each job description file, there is another file with labels contained bigrams, trigrams, and quadrigrams with the next structure:

We discovered four hundred twenty-four patterns of which one hundred fourteen only have frequency 1, Seventy-three have frequency 2, thirty-four have frequency 3, twenty-three have frequency 4, nineteen have frequency 5, sixteen have frequency 6, twelve have frequency 7, nine have frequency 8 and nine have frequency nine. With previous numbers, we decided to remove patterns with frequency from 1 to 9.

Since if we take a range higher such as frequency from 1 to 15 the recall measure rapid decreases because above of frequency 9 there are many keywords that depend on it. In an opposite way, when is below of frequency 9 the recall measure rapid increase but the precision measure decrease and at the same time F-measure decrease.

In total were three hundred nine patterns removed. In addition, the proposed approach will have problems with the recall measure because eight hundred eighty-two tags do not exist in the results.

The main patterns results are in the Table 11 they frequency is between thirty-five and one thousand and they were the patterns that gave us good results for the next experiment. We can see the patterns diversity than we occupy in this research work to develop the automatic key extraction. In the first row appears one thousand times that is mean that is very important. Also, we can see that a lot of them start with NN, NNS, VBG or JJ. Also, the first three in the Table 11 gave us unnecessary keywords, for this case is like a balance the POS pattern gave us necessary and unnecessary keywords.

Once we got the results we decided to create the POS patterns filter only taking the beginning and the end (see Fig. 3). We discovered that filter comes more unnecessary tags in the final results because the precision it was lowest.

Fig. 2 Labeled corpus 

Fig. 3 First POS filter 

Then we decided to use exactly the POS patterns extracted from the manually labeled corpus to filter and reduce the unnecessary keywords. This action had an increase in precision measure and F-measure.

We can observe five groups (see Fig. 2) and each group contain ten job description manuals labeled. With this corpus, we can compare the results obtained. It was training with three parts of the corpus and then take the POS to prove in the rest. There are ten possible combinations:

Where the first column is the set used to train the proposed approach with that patterns and the second column is the set used to prove it.

For this comparison only need the positives tags but for the second we needed the negatives tags. The negative tags were obtained from the ones that are not in the labeled corpus set.

When we add the negatives we create a new set (see Fig. 5). The new set was bigger than the first one, now contain 100 jobs descriptions divided into 5 groups, each group contain ten positive tags and ten negatives tags. There are 10 possible combinations:

Fig. 4 Second POS filter 

Fig. 5 Labeled corpus with negatives 

Where the first column is the set used to train the proposed approach with patterns combined positives and negatives and the second column is the set used to prove it also combined with positives and negatives.

5.6 Experiment 4

In this experiment, we decided to use three measures to increase the precision and F-measure. To obtain better results we decided to use an intersection between the measures. There are for possible combinations:

Fig. 6 Intersection A∩C 

Fig. 7 Intersection A∩B 

Fig. 8 Intersection A∩B∩C 

Fig. 9 Comparison of results 

Fig. 10 Intersection B∩C 

As each measure has its own value, we decided that we would have to base ourselves on different percentages in order to be able to discard any possible unnecessary labels.

From the intersection results, the values were sorted and some percentages were extracted. We decided to get the following percentages: 30%, 50%, 60%, 70%, 80%, 90% and 100% in order to determine the best subset with the highest number of keywords. For this paper we refer to the percentages as sets, with the following names, 30% PA, 50% PB, 60% PC, 70% PD, 80% PE, 90% PF, and 100% PG. The reason was to avoid having a single focus and prove which one gives us better results.

Each intersection has the its own results which are explained with supporting detail in the the following graphics.

In Fig. 6 we can see the intersection between PMI and Likelihood-Radio. The recall increases faster in each percentage, but we can see when PA starts, it is the lowest but it becomes the highest measure.

The precision measure in PA is higher because there are not many keywords and the few keywords are in the set labeling manually. As it was expected after the PA was the lowest with a slight decrease in each one because the number of keywords has rapid increase but the keywords are not in the set labeling manually. The F-measure start between recall and precision measure and after that always had a slight increase.

In this intersection we can recognize the higher value for each one, for recall of PG was 0.6896, because when more files are used the probability to find a keyword that matches with the corpus labeled manually increases. The precision measure of PB was 0.1326, this happens because it was a balance in the numbers of keywords obtained for the proposed approach matching with the corpus labeling manually. Finally, the F-measure was PG with 0.2139, this happens because the F-measure is the combination between two measures: recall and precision, so, when the recall measure has a high value this implies the F-measure will increase.

In Fig. 6 we can see the intersection between PMI and Chi-square. we can see the same behavior that graph (see Fig. 7) the recall increase faster in each percentage first start with the lowest percentage but after that always remain in the head and his higher value is PG with 0.6896, this happen for the same reasons when the corpus is big, so there are more keywords to math with the corpus labeling manually. For precision had a slight decrease and his higher value is 0.1454. Finally for F-measure is PG with 0.2139, although is a different intersection using different measures throw almost the same results.

In Fig. 10, we can see the intersection between Likelihood-Radio and Chi-square. We observe this intersection has the same behavior that the previous graphs (see Fig. 6 and 7), but change the data results. For this intersection is interpreted as the previous ones. Showing the importance of the POS filter.

In Fig. 8, we can see the intersection between PMI, Likelihood-Radio and Chi-square. The higher value for recall is PG with 0.6896 and the lowest value is PA with 0.0922.

For the precision measure with the higher values is PA with 0.1454 and the lowest is PE with 0.1245.

Finally, F-measure with the higher value is PG with 0.2139. The comparison of the graphs (see Fig. 7, Fig. 6 and Fig. 10) have the same behavior and this can be explained because the POS patterns are useful to reduce the unnecessary keywords extracted by the collocation measures. Although, it should be noted that have the same behavior but no the same values in the results, and we can say that for this research work the best experiment was the intersection using the three measures.

It should be mentioned that the PG of each intersection is exactly the same, this is because when using the whole information does not have any label discards. In the graph (see Fig. 9) we observe the results between the PG set with and without POS patterns filter. The first measure to explain is recall, in this one, we observe that the difference is 0.2408 being the higher that does not has POS filter. This easy to interpret, because when it does not have the POS patterns filter the proposed approach extracted a lot of keywords gave us a higher value for recall measure. In the same way when there are a lot of keywords that are not in the corpus labeling manually reduce the precision measure and increase when has the POS filter the difference is 0.0699. Finally for F-measure is a combination between precision measure and recall measure so for this research work is a high importance. We can see that the higher is the first one with 0.2139 and the difference is 0.107, thus we can say that the POS patterns filter method is better.

In summary, the last pattern lost 24% in recall measure but increase in precision with 6.9% and the F-measure with 21.3%.

5.7 Comparison with other Dataset

In this experiment, we used another corpus to demonstrate that POS patterns work not only in job descriptions.

The information has 454,580 N-grams divided in two:

In Fig. 11, we observe the PMI behavior. The gray line means the Good and the black line means the Bad. The Good means the N-gram is correct because comply with our patterns and the Bad means the appositive. In this pattern, we observed that PMI is good with a large information volume. The Good part always remains above the Bad with a big difference.

Fig. 11 PMI measure 

In Fig. 12, we observe the Likelihood-ratio behavior. In this behavior begins with the same proportion of Good and Bad. But by 100,000 the percentage of Bad is a little greater than Good. Then it changes a little before reaching 400,000 for good ones have a slightly higher value. This happens because the Likelihood-radio put a value to each trigram and that value put high values to trigrams that no have the POS pattern.

Fig. 12 Likelihood-ratio measure 

In Fig. 13, we observe the Chi-square behavior. In this behavior we can see that it looks like PMI behavior. With the difference that a little less distance between the Good and the Bad. Remaining with a higher value the Good ones. This happens because the Chi-square put similar values to trigrams that do comply with POS patterns and those that do not comply as well.

Fig. 13 Chi-square measure 

In Fig. 14, we observe the subset full intersection without sorted. Here we can see the three collocation measures together and the Good percentage. Likelihood-ratio stays below the other two measures. In the beginning, it was not constant. We can see that the one maintained with better results was PMI but is very similar to Chi-square.

Fig. 14 Measures intersection without order 

In Fig. 15, we observe the subset full intersection with sorted. In this graph unlike the Fig. 14 are ordered seeing three important things:

Fig. 15 Measures intersection with order