3.1 Domain-Independent Intent Extraction

In the research presented by Luong et al. ^[19], authors defined the user explicit intent as a 5-tuple:

in which:

And according to Luong et al. (2017) ^[21], authors proposed a domain-specific intent extraction model, where they tried to extract w and p with the assumption that d had been identified. It means that the intent information can be extracted only if the domain of the intent have been known. However in real applications, the user's intents are very diverse, they may be want to buy a house", or are going to travel", and even need to borrow some money" etc.

And that leads to a large amount of intent domain types, such as Tourism, Transportation, Finance, Real Estate, Education... And as we mentioned above, the more the number of the intent domains are, the more complicated it is to extract the intent information. Firstly, one needs to identify the intent domain d. And secondly, for each domain d, one has to build a set of specific labels to identify necessary attributes. For example, one could arrive at 15 specific labels for Tourism, 18 specific labels for Real Estate and 17 specific labels for Transportation domain respectively (see tables 1, 2, 3). Finally, after combining these three sets of specific labels, one could have the set of 33 specific labels (see table 9).

Therefore, if the number of intent domains get bigger, the number of labels will grow sharply. As a result, the bigger the number of labels is, the more difficult the predict model has to face. In this paper, we proposed a new method to identify the user intention that does not depend on the domain d. We still formulate our work as a sequential labeling problem, but the main improvement is the idea of generalizing a new set of domain-independent labels that we will describe more clearly in the next subsection. Instead of building a set of specific labels for each intent domain, we try to build the most suitable set of labels for all intent domains.

Then we built a carefully experimental strategy to verify our assumption that the set of general labels is more effective than the set of specific labels for intent extraction problem when intent domains are scaled up. This assumption lead us to the novel approach to identify users intents that be called domain-independent method. This approach allows us to extract w and p without having to identify d.

This will help us to bypass one difficulty. Moreover, we are free from the worry of the number of labels increasing when a new domain comes, because we only need a few general labels as will be described in the next subsection for every domain. In the next subsection, we will present three sets of specific labels that we proposed for three domains Tourism, Transportation and Real Estate. Especially, we will explain more carefully about the way that we map from the set of general labels to these three sets of specific labels.

3.2 Domain-Independent Labels Versus Domain-Specific Labels

With three domains that we chose to crawl the data for model training (Tourism, Transportation, Real Estate), we built three specific sets of labels. We have 15 labels for Tourism, 18 labels for Real Estate, and 17 labels for Transportation, and they are described in detail in table 1, table 2 and table 3 respectively.

After conducting surveys carefully all the crawled data from three domains (Tourism, Transportation and Real Estate) and also some online data from other domains, we decided to proposed a set of 10 general labels. The table 4 presents these ten general labels and especially shows the mapping from the set of general labels to three sets of domain-specific labels. Some information/properties exists in almost sort of intent domains, such as intent, object, price..., then they are treated as general labels. Some other properties are just specific for each intent domain, for examlpe time period in Tourism domain, acreage in Real Estate domain or color in Transportation domain will be aggregated to make the label description in the set of general labels.

3.3 Intent Extraction Models

Given a post contains a user's intent which belongs to any intent domain, our model desires to extract the intent keyword (such as buy, sell, hire...) and all the necessary information that relates to the user's intention. So we proposed to use two advanced machine learning models to build our models, namely CRFs and LSTM.

3.3.1 Conditional Random Fields

Conditional random fields ^[16] are probabilistic models has shown a great success in segmenting and labeling sequence data.

Given o = {o₁, o₂, …, o_T} as input observation sequence data, CRFs identifies s = {s₁, s₂,…, s_T}, which is a finite set of state associated with a set of labels y_i = ( y_i ∈ L = {y₁, y₂, …, y_M}) by a probability function:

where Zθo=∑s'exp∑t=1TFs',o,t is the normalizing factor to ensure that pθso is a probabilistic distribution, and Fs,o,t=∑iλifis,o,t is the sum of CRFs feature f_i with the feature weight λ_i correspondingly. CRFs is trained by searching the set of weights θ*=λ1*,λ2*,…,λn* to maximize the log likelihood function. When the labels make the state sequence unambiguous, the likelihood function in exponential models such as CRFs is convex, thus searching the global optimum is guaranteed.

It has been shown that quasi-Newton methods, such as L-BFGS ^[18], is the most efficient for this issue. In our work, we utilized pycrfsuite (https://python-crfsuite.readthedocs.io/en/), which is a fast implementation of Conditional Random Fields on Python. We chose linear chain CRFs architecture because of faster training time. State features used in our model were as following:

3.3.2 Bidirectional Long Short-Term Memory (Bi-LSTM)

LSTM was developed based on recurrent neural network (RNN) architecture by Hochreiter and Schmidhuber (1997) ^[9] and it is known to be the most effective deep learning model in natural language processing problem. Given the input (x₁,x₂,...,x_n), we have LSTM model computes the state sequence (h₁,h₂,...,h_n) by iteratively applying the following updates:

where σ is the element-wise sigmoid function and ⊙ is the element-wise product. LSTM have been designed to combat RNN issue by incorporating a memory-cell and have been shown to capture long-range dependencies. They do so using several gates that control the proportion of the input to give to the memory cell c_t, and the proportion i_t to forget from the previous state.

In sequence tagging task, we have to take care of both past and future input features for a given time, so we chose Bi-LSTM network ^[5] to do our second experiment. The figure 1 illustrates the structure of Bi-LSTM model that we used.

Fig. 1 The Bi-LSTM Model 

With this model, l_i represents the word i and its left context and r_i represents the word i and its right context. Then these two vectors will be combined to create the result vector represents the word i in its context, c_i.

Following the Bi-LSTM architecture in ^[17], we trained our Bi-LSTM model with the following set up:

Specifically, the size of char embedding and the number of char long short-term memory unit in our model are both 25. These ones for the size of word embedding and the number of word long short-term memory unit are both 100.

We also used dropout technique to reduce the over-fit phenomenon. Our optimization method was Adam with learning rate, learning rate decay and clip gradients initialized by 0.001, 0.9, 5.0 respectively. All of these hyper-parameters would be tuned during training phase.

3.3.3 Bidirectional Long Short-Term Memory -CRFs (Bi-LSTM-CRFs)

Instead of making tagging independently, a CRFs layer is added at the end of the tagging process of a Bi-LSTM model. The output of Bi-LSTM layer had been considered as the input of CRFs layer and the output of CRFs layer will be the final tags. Based on the model described in ^[17], we utilized Bi-LSTM-CRFs model described in figure 2 for our problem. The initialization of this model was same as the one described in Bi-LSTM model above.

Fig. 2 The Bi-LSTM-CRFs Model 

4.1 Experimental Data

In our work, we used the data from online forums, social media network and other websites. We collected data for Tourism domain from two main sources: https://www.webtretho.com/forum/f110/ and https://dulich.vnexpress.net/. In Real Estate domain, data was mostly crawled from https://batdongsan.com.vn/. Some Facebook public groups, such as: https://www.facebook.com/groups/xemay-cuhanoi, were used for collecting data for out last selected domain, Transportation. We only used the posts which have length from 30 characters up to 800 characters in order to reduce noisy data come from advertisement posts.

Overall, our built dataset contains about 3000 posts for each domain. After that, we had a group of 5 students to tag the data with the labels that we had built. We carefully did the cross-check among of these students works to choose the most suitable annotation. Finally, for each post, we have two sets of labels to tag. The first one is the set of specific domain labels of a post and the second one is the set of general labels. Figure 3 presents an example post be tagged with domain-specific labels and domain-independent labels in turn.

Fig. 3 Tagging posts with specific labels and general labels 

After that, we carried out the experiments with both tagged types, the results and discussion will be presented bellow. We used 60% of data to train our model, 20% of data to tune the hyper-parameters. Finally, to evaluate our model we used the remaining 20% of our collected data.

4.1.1 Results on a Specific Domain

4.2 Evaluation Measures

For all experiments, precision, recall and F1-score at the segment (or chunk-based) level are used as the evaluation measures. Specifically, assume that the true segment sequence of an instance is s = (s₁,s₂,...,s_N) and the decoded segment sequence is s'=s1',s2',…,sK'. Then, sK' is called a true positive if sK'∈s. The precision and recall are the fractions of the total number of true positives among the total number of decoded and true segments respectively. We report the F1-score which is computed as 2.precision.recall/(precision + recall). Besides, we have the support as the number of the true segment corresponding to each label in the testing set.

4.3 Experimental Results and Discussions

We conducted the experiments with three models as we described above, namely CRFs, Bi-LSTM, and Bi-LSTM-CRFs. With an attempt to prove our assumption that using general labels is more effect than specific ones if the number of intent domains increases, we carefully conducted totally 42 experiments, including:

In the next sub section, we will present some of the most interesting results and their discussions.

As mentioned above, we did the experiments for each of three specific domains: Tourism, Transportation and Real Estate with both the set of general labels and the set of specific labels. Table 7 presents the overall results for these experiments with CRFs, Bi-LSTM, and Bi-LSTM-CRFs respectively. The highest F1-score we received when conducting experiments for each domain separately are the results of Tourism domain, they are 83.33% for extracting general labels and 82.01% for specific labels.

As described above, Tourism domain has least number of labels compared to Real Estate and Transportation (15, 18 and 17 labels respectively). Moreover, after carefully analyzing data from three domains, we found that Tourism domain contains less noisy data, such as improper abbreviation, emoticons than two remaining domains. Table 7 also shows that Bi-LSTM-CRFs achieves better results than CRFs and Bi-LSTM our experiments, although there isn't any hand-crafted feature was used in Bi-LSTM-CRFs. It proves that Bi-LSTM-CRFs is the most suitable model for our problem.

For more detail, table 6 shows the best chunk-based results when applying Bi-LSTM-CRFs model for the Tourism data using the set of general labels. And table 5 shows the best chunk-based results when applying Bi-LSTM-CRFs model for the Tourism data using the set of specific labels. As we recognize, it is better to use the set of specific labels when extracting users intents in each individual domain. The reason is for a specific domain, the disparities in accuracy between using general labels and specific label are small, see table 7, while specific labels can describe the entities in greater detail.

So, we then present the best chunk-based results of Transportation domain and Real Estate domain when doing experiments with the set of specific labels in the table 8. We find that almost labels that benefit from high number and also their values have the recognizable form, such as Intent, Price, Contact,..., usually get high accuracy in all of three domains. However, some labels although have quite high number, such as Location, Description, Equipment and Context, they still get not really high accuracies. This can be explained by their complicated and barely recognizable value forms.

4.3.1 Using General Labels or Specific Labels when Scaling up Intention Domains

We would like to show our results and discussions to do the comparison between using the set of general labels and the set of specific labels. Figure 5 presents the average F1 score when we apply CRFs model in experiments with 1 domain using the set of domain-independent labels (general labels) and the set of domain-specific labels (specific labels) respectively, and the corresponding results when we increase the number of domains to 2 domains, 3 domains. Similarly it also shows the results when applying Bi-LSTM model and Bi-LSTM-CRFs model respectively.

Fig. 5 The average F1-score when applying CRFs, Bi-LSTM and Bi-LSTM-CRFs models in experiments with 1 domain, 2 domains and 3 domains using general labels and specific labels correspondingly 

We realize that it is usually gets better results when using the set of general labels rather than using the set of specific labels. So we could come to the conclusion that it would be better to use the set of general labels when identifying user's intent from collections of data combining from various domains. And as we mentioned above, one more reason for this conclusion is using the set of general labels help to get rid of rebuilding a new set of labels when a new intent domain comes.

4.3.2 The Best Result for the Combination of Three Domains

Figure 4 shows the results when we apply CRFs model, Bi-LSTM model and Bi-LSTM-CRFs model to extract users intention for the combination of data from three selected domains. For each model, we conducted experiments with the set of general labels and the set of specific labels respectively. In this situation, Bi-LSTM-CRFs model still achieves higher average F1-score than the two remaining models and the experiments using the set of general labels alway show higher results than the set of specific labels.

Fig. 4 The average F1-score for the combination of three domain datas with Bi-LSTM-CRFs, Bi-LSTM, CRFs models correspondingly 

Table 9 and table 10 below show the best chunk-based results when we do the experiment with the set of 32 specific labels and the set of 10 general labels for the combination of data from three selected domains respectively. This is the result when we applied Bi-LSTM-CRFs method into our model. With the set of general labels we find that the accuracies for almost labels are quite stability. They are almost over 70%, except the label Context. This can be explained by the number of the Context labels are small, moreover the description of the Context labels in this problem is very diverse and complicated as can be seen in the table 1, 2 and 3. Moreover in the experiment with the set of general labels, Intent and Object labels, which are the most important labels to identify users intents, always achieve higher F1-score than themselves in the experiment with the set of specific labels.

All in all, it reconfirms that Bi-LSTM-CRFs and the set of general labels are suitable for identifying users intents in the combination of various intent domain.