3.1 Knowledge-based Approach

A human HR executive can easily spot the project sections in any given resume. Given the endless variations in which the projects are written, this knowledge is quite non-trivial and does not consist of simple rules. In fact, we found that the humans have a dynamic, context-sensitive way of rearranging the project sections boundaries. We have tried to capture this expert knowledge in the form of an algorithm (Algorithm 1). The basic idea is that the human reader identifies the markers (already discussed), and then rearranges the project section starting point by understanding the spatial relationships among the marker positions.

Algorithm 1: identify_project_sections 

The essence of this rearrangement is as follows. It is usually true that a project section begins with the project marker (e.g., project title), and all other markers (e.g., client, teamsize etc.) come afterwards within the project section. But occasionally some of the markers may occur just before the project title. Thus we need to recognize such deviations from the typical sequence of markers within a project section, and adjust the starting sentence of the project section accordingly. Since a lot of such variations among marker sequences occur in practical resumes, there is much heuristic post-processing still required after algorithm identify_project_sections (Algorithm 1), which we have omitted.

3.2 ILP based Approach

We model the "repeated section identification" problem using the Integer Linear Programming (ILP) formulation. Table 3 depicts the input parameters, the constraints and the objective used for the ILP formulation. The 9 boolean arrays corresponding to project section markers described earlier are the input parameters. Another input for the ILP formulation is S, the maximum number of project sections possible for the current resume.

The output representation is in the form of two matrices (x and y) of N × S binary variables. If j ^th project section begins at the k ^th sentence, then the j ^th column of x will contain all 0's before the k ^th row and all 1's after that till the end. Similarly, if j ^th project section ends at the k ^th sentence, then the j^th column of y will contain all 0's before the k ^th row and all 1's after that till the end. In other words, x is used to mark the beginning of individual project sections and y is used to mark the end. Hence, xi,j-yi,j will be 1 if and only if i ^th sentence is part of j ^th section. Also, by definition of x and y,xi,j-yi,j will be 1 for consecutive sentences only. The objective is to minimize following 3 terms:

Various constraints are defined to capture different desired properties of the output project sections. Table 3 describes these constraints in detail. A separate ILP program is created for each resume and is solved to compute optimal feasible values for output variables, which in turn translate to predicted project sections.

3.3 Sequence Labeling Approaches

In addition to the unsupervised approaches (knowledge-based and ILP-based), we also explored two supervised approaches. Here, we model the "repeated section identification" problem as a sequence labeling task where an appropriate label is assigned to each element in a sequence. Each resume is represented as a sequence of sentences and 3 labels (B, I and O) are used to represent the project section information. First sentence in each project section is labeled with B and all subsequent sentences within the section are labeled with I. Sentences which are not part of any project section are labeled with O. In order to learn a sequence labelling model, we explore following two approaches:

3.3.1 Conditional Random Fields (CRF):

In this approach, each sentence is represented by a set of features which are designed to capture various characteristics of the sentence. Some of the representative features are as follows: presence of the project section markers in current, previous and next sentences, number of consecutive blank lines before and after each sentence, number of words in current sentence, distinct words present in current sentence. We use Conditional Random Fields (CRF) ^[6] for training a sequence labeling model, which is used to predict BIO labels for any unseen resume and predicted project sections can be derived from these labels.

3.3.2 Long Short-Term Memory (LSTM):

The CRF-based approach requires explicit feature engineering for the task. Also, the markers-based features are dependent on accuracy and coverage of the regular expressions used to identify the markers. Hence, we also developed an LSTM-based ^[5] approach which bypasses the need for explicit feature engineering. Here, the model for section identification is built in two phases. Initially, our aim is to learn embedded representations for sentences in resumes. For this purpose, we train an LSTM-based sequence autoencoder ^[1] which consists of an LSTM encoder layer and another LSTM decoder layer. The detailed architecture is depicted in the Figure 1. Here, a sequence of words in a sentence is passed through the encoder LSTM layer, so that the output of the final time step provides the representation of the whole sentence. Words are represented using 100 dimensional pre-trained GloVe ^[9] word vectors. The decoder LSTM layer then tries to reconstruct the same sequence of the words using this representation. The model is trained in an unsupervised fashion to minimize the Mean Squared Error at the decoder output. We train this model on a large corpus of 810 resumes having more than 140,000 sentences, using 100-dim word vectors and 200-dim vectors for sentence representations.

Fig. 1 LSTM-based sequence autoencoder used to learn embedded representations for sentences in resumes 

Using the encoder LSTM in our sequence autoencoder, we can now get an embedded representation for any new or unseen resume sentence. As discussed earlier, a resume is a sequence of sentences and our aim is to assign an appropriate label (from BIO labels) to each sentence to identify project sections.

For this purpose, we design another Bidirectional LSTM model ^[7] where input for the Bi-LSTM layer is a sequence of embedded representations of sentences in a resume. For the i ^th sentence, Bi-LSTM provides two representation vectors:

These two context vectors are concatenated and passed to a softmax layer for final prediction of BIO labels. The detailed architecture of this model is depicted in Figure 2.

Fig. 2 Bi-LSTM sequence labelling model for assigning BIO labels to resume sentences 

This model is then trained in a supervised fashion (similar to CRF) using labelled dataset of resumes annotated with gold-standard project sections. Once the model is learned, it can be used for predicting BIO labels for any unseen resume and predicted project sections can be derived from these labels.

Moreover, we tried another variant of this Bi-LSTM based model, where for each sentence, we augment the sentence representation (provided by the LSTM encoder layer in our sequence autoencoder) with a vector representing the presence of project section markers.

This vector is a binary vector containing K bits corresponding to each project section marker. If any marker is present in a sentence, then its corresponding bits are set to 1, otherwise they are set to 0. The value of K = 10 was empirically found to be suitable.

4.1 Datasets

We manually annotated 366 resumes for project section information. For each resume, we identified sentence numbers corresponding to the first sentences of project sections. The dataset was partitioned into two parts: D₁ (206 resumes) and D₂ (160 resumes). The resumes in the dataset D₁ were used to fine-tune the patterns for project markers as well as the rules used in knowledge-based approach. The supervised sequence labeling approaches (CRF and LSTM-based approaches) are trained on D₁ . The dataset D₂ is a blind test set for all approaches.

4.2 Intrinsic Evaluation

For intrinsic evaluation of the proposed approaches, we compare the predicted project sections with the manually annotated (gold-standard) project sections at 3 different evaluation levels (see Table 4):

Table 4 shows the comparative performance of the proposed approaches. All the approaches are evaluated on both the datasets D₁ and D₂. For the supervised approaches (CRF and LSTM-based), the dataset D₁ is used for training the models which are then applied on the dataset D₂. Also, these approaches are also evaluated on D₁ using 5-fold cross-validation. It can be observed that the knowledge-based approach outperforms all other approaches on D₁. But its performance degrades on D₂ which is the blind test set.

Although such performance degradation is observed for all approaches, it is more pronounced for the knowledge-based approach. As the rules in the knowledge-based approach are designed by observing project sections in D₁, this approach seems to have over-fitted on D₁. Although, ILP-based approach lags behind the knowledge-based approach, its performance is more consistent across D₁ and D₂. As ILP-based approach is a more principled way of representing the domain knowledge, it is also easier to maintain.

On the dataset D₂, the LSTM-based approach using markers outperforms all other approaches. Moreover, this approach provides more consistent performance across both the datasets D₁ and D₂. Although, the CRF-based approach lags behind in F1, it provides the best precision on D₂. In future, we plan to explore an ensemble of CRF and LSTM-based approaches to exploit the high-precision nature of CRF-based approach as well as the high-recall nature of LSTM-based approach.

The LSTM-based approach without using markers also outperforms knowledge-based, ILP-based and CRF-based approaches. It is important to note that this approach has the least dependence on the domain knowledge because it does not need information about project section markers. It only depends upon the sentence representations learned in an unsupervised manner using sequence autoencoder. Hence, the LSTM-based approach without markers can be easily re-trained for any other domain for identification of other types of repeated sections.

4.3 Extrinsic Evaluation

As mentioned earlier, several practical applications need project sections to be identified in resumes. So we used the resumes with or without identification of project sections to evaluate the output of some typical end-user queries. For example, a recruitment executive or project leader is interested in candidates who have used skill X in at least one project. Since the given skill X may occur outside project sections as well, one should only identify those candidates for whom X occurs within at least one project section. Not using project sections will retrieve all resumes in which X occurs somewhere in the resume, not necessarily within any project sections.

The results are shown in Table 5 for 3 different skills. In setting S₁, the resumes were retrieved without using any project section information. For settings S₂ to S₆, the resumes were retrieved using the project section information predicted by knowledge-based approach, ILP-based approach, CRF-based approach, LSTM-based approach without markers and LSTM-based approach using markers, respectively. Using ground truth, we computed precision, recall and F -measure for these 6 settings. Other example queries where project sections are important are: find candidates who have used skill X for at least 12 months and find candidates who have used skill X in role developer.