3.1 Sentence Representation

Neural networks are effective in representing semantic and syntactic information of sentences in low-dimensional vectors. This paper investigates two unsupervised neural models, i.e. Deep Auto-Encoder and Long Short-Term Memory (LSTM) Auto-Encoder [⁸] for sentence representation.

3.1.1 Deep Auto-Encoder

An Auto-Encoder neural network is a generative model that aims at reconstructing its own inputs. Our deep Auto-Encoder model is introduced in Fig. 2. It has four encoding layers:

h1=σ(W1⋅X), (1)

h2=σ(W2⋅h1), (2)

h3=σ(W3⋅h2), (3)

h=σ(W4⋅h3). (4)

Fig. 2 Deep Auto-Encoder: h (the red block) is used for sentence representation 

A sentence X is put into the network with tf-idf weights. X is very sparse because it only contains a small number of words while its dimension is the size of vocabulary. The Auto-Encoder can learn a distributed semantic representation with low dimension. The layer h is used for sentence representation. Decoding layers are:

h′3=σ(W′4⋅h), (5)

h′2=σ(W′3⋅h′3), (6)

h′1=σ(W′2⋅h′2), (7)

X′=σ(W′1⋅h′1), (8)

where sigmoid function is:

σ(x)=11+e−x. (9)

The squared error loss is:

J(X,X′)=‖X−X′‖=∑V(Xi−X′i)2, (10)

where V is vocabulary size.

3.1.2 LSTM Auto-Encoder

Deep Auto-Encoder doesn’t capture syntactic information in word order. We propose using LSTM Auto-encoder (Fig. 3), which was first introduced in [⁸]. This model learns sentence in an unsupervised manner and captures both syntactic information in word order and semantic information in word embeddings:

ht(enc)=LSTMencodeword(et,ht−1(enc)), (11)

Fig. 3 Long-short-term-memory Auto-Encoder: The last encoding LSTM cell (the red node) is used for sentence representation 

h_ends is used to present the input sentence

es=hends, (12)

ht(dec)=LSTMdecode(et,ht−1(dec)). (13)

The decoder sequentially predicts sentence words using a softmax function:

P(x′t|Δ)=softmax⁡(et−1,ht−1(dec)), (14)

e_t is an embedding for word at position t and generated by the LSTM_decode. The encoder and decoder use two different LSTMs with two different sets of parameters.

Our loss function:

J(X,X′)=1/N∑i<NH(ei,e′i), (15)

where H is the Cross-entropy error function. The LSTM model at time t is defined as follows:

[itftotlt]=[σσσtanh⁡]W⋅[ht−1et], (16)

ct=ft⋅ct−1+it⋅lt, (17)

ht=ot⋅ct. (18)

3.2 Extractive Summarization

MMR is applied to generative extractive summaries (Algorithm 1). It is a greedy algorithm which incrementally selects a sentence by maximizing a linear combination of query relevance and summary diversity (line 3). Here the hyper-parameter κ takes a value in [0, 1]. Sim(s, q) and sim(s, s′) are sentence similarity. q is the question. S is the set of all sentences in the answers. L is the limit length of a summary. R is the set of summary sentences.

Algorithm 1 Maximal marginal relevance (MMR) 

Sentence similarity is computed by cosine similarity:

4.1 Datasets

L6 - Yahoo! Answers Comprehensive Questions and Answers corpus³ from Yahoo Webscope was used for unsupervised learning of sentence representation (Table 1).

We used the test dataset in [¹⁷] for evaluation⁴. The dataset contains manual summaries with the limited length of 250 words. In our experiments, limited summary length was selected accordingly (L = 250 in MMR).

4.2 Experimental Setup

Each input sentence vector put into AE is represented using tf-idf. The vocabulary was created by lowercasing, removing the stopwords, rare words (below 10 times), stemming, and normalizing number. The auto-encoder has four layers for encoding, and four layers for decoding. Layer h with 100 dimensions is used to present sentence. Learning parameters for back propagation and Adam algorithm[⁶] were: learning rate η = 0.001; batch size = 128 sentences; 20 epochs. The model was trained on Yahoo-webscope in eight hours with a machine of 20 CPUs.

Word embeddings from word2vec⁵ on Google news of size 300 were fed into LSTM-AE. When a word was not in the vocabulary of pre-trained word embeddings, its embedding was sampled from a normal distribution. Commas, colons were converted to <dot>.

Periods, end marks were converted to <eos>. Learning parameters were: batch size of 128 sentences, 20 epochs, learning rate η = 0.001. It took three weeks with a machine of 20 CPUs to train this model on Yahoo-webscope. Both AE and LSTM-AE were implemented on Tensorflow.

4.3 Experimental Results

ROUGE metric [⁹] was used to evaluate text summarization. At first, the results of two baselines, tfidf and tf-idf weighted average word embeddings, are shown in Table 3. AE, LSTM-AE and a combination of AE and LSTM-AE by concatenating the two sentence embeddings (mentioned as CONCAT) are compared. The results are in Figure 4.

Fig. 4 Performance on varying κ in MMR 

As we only have the test dataset, experiments with different values of κ as the only hyper-parameter (of MMR) were conducted. LSTM-AE with κ = 0.3 was selected as our representative to compare with related works. Last but not least, with κ = 0.3, linear combination of AE and LSTM-AE similarities was investigated (Table 5):

where α is hyper-parameter.

As expected, Word2vec outperforms tfidf by large margin (Table 3) thanks to low dimensional vectors and semantic information. However, Word2vec is not on par with AE and LSTM-AE (Figure 4). This is because the former straightforwardly derives sentence embeddings from word embeddings by weighted average; while sentence vectors are parameters of the two latter models that need to be learned from data. With κ < 0.5, LSTM-AE beats AE on all the metrics. When κ > 0.5, AE performs better on ROUGE-1 and ROUGE-2. This is possible because a large value of κ prefer diversity to relevance. Overall, LSTM-AE is a better choice. It is worth noting that concatenating the two models doesn’t bring significant improvement (Figure 4).

LSTM-AE with κ = 0.3 was compaired to state-of-the-art methods. DOC2VEC [¹⁴] uses Paragraph Vector [⁷] to generate sentence representation and sparse coding to detect salient sentences. However, it is not clear on which data Paragraph Vector was learned and how sentences were represented. CNN learns sentence embeddings from annotated answer sentences, i.e. sentences with labels as summary or non-summary. Relevant sentences from Wikipedia are also retrieved to overcome sparsity. Low-dimensional sentence vectors are first put into sparse coding and then MMR to generate summaries. Here, the baseline BestAns selects the best answers as summaries.

Interestingly, our unsupervised sentence representation performs slightly better than supervised one without annotated data (Table 4). LSTM-AE outperforms DOC2VEC. The reason could be two-fold: i) Paragraph Vector introduces paragraph (i.e. sentence in this case) context via so-called paragraph id additional token in the input layer, and sampling several windows through the sentence. Meanwhile, LSTM-AE captures semantic and syntactic of the sentence in the last encoding LSTM cell and uses it for sentence representation. ii) LSTM-AE was trained on Yahoo-Webscope, a large corpus of questions and answers from communities.

This could make sentence representation more suitable to CQA tasks. On the other hand, we have no clue on which data Paragraph Vector is trained in DOC2VEC; and why ROUGE-2 reported in [¹⁴] is higher than both CNN and our method. In the future, we are going to reimplement DOC2VEC, with Yahoo-Webscope as training data for Paragraph Vector, to investigate in more details.

Table 5 shows that linear combination of sentence similarities is more effective than concatenating the representations of sentence pairs (Figure 4).