1 Introduction

Topic models are widely used in text processing and Latent Dirichlet Allocation (LDA) [³] is the core of a large family of probabilistic models. LDA provides an efficient tool to analyze hidden themes in data and helps us recover hidden structures/evolution in big text collections. The key problem in topic models is to compute the posterior distribution of a document given other parameters. The posterior inference problem in topic models is to infer the topic proportion of documents and topics which are distributions over vocabulary. Large datasets or streaming environments contain huge number of documents, hence the problem of estimating topic proportion for an individual document is especially important. The quality of learning for LDA is determined by the quality of the inference method being employed.

Unfortunately, solving directly a posterior distri-bution of a document is intractable [³]. There are two main approaches to tackle it.

One is approximating the intractable distribution by tractable distribution, for example Variational Bayes inference (VB) [³]. The other is a sampling method, which draws numerous the samples from target distribution then estimating the interesting quality from these samples. The well-known method is Collapsed Gibbs Sampling (CGS) [⁸]. There are also famous methods such as Collapsed Variational Bayes (CVB) [¹⁵], CVB0 [²], Stochastic Variational Inference (SVI) [¹⁰], etc.

To our best knowledge, there are not any mat-hematical guarantees for quality and convergence rate in existing approaches. Therefore, in practice we do not have any ideas about how to stop the methods we are using but trying, observing and retrying again to reach the best solution.

Another way to solve the posterior distribution is to view it as an optimization problem. To infer about topic proportion of a document is to solve the maximum a posteriori of topic proportion given words in this document and all topics of corpus [¹⁶]. This optimization problem is usually non-convex and NP-hard [¹⁴]. There is very few theoretical contributions in non-convex optimization literature, especially in topic models. Online Maximum a Posteriori Estimation (OPE) [¹⁶] which is an online version of Frank-Wolfe algorithm [⁹] is a stochastic algorithm to solve such kind of non-convex problem.

OPE is theoretically guaranteed to converge to a local stationary point [¹⁶]. Although OPE is easy to implement and has fast convergence and mathematically guaranteed, it remains some problems. The weakness of OPE is that it is not well adaptive with different data sets because of the uniform distribution in its operation. We will exploit this crucial point to propose a new and more general algorithm based on OPE. When changing its operations, we have to retain the advantage of the original algorithms, that is theoretical guarantees.

Our main contribution is following:

Organization: The rest of this paper is organized as follows. In Section 2, we introduce an overview of posterior inference with LDA and main ideas of existing methods. In Section 3, our new algorithm G-OPE is proposed in details. In Section 4, we conduct experiments with two large datasets with state-of-the-art methods in two different measures. Finally Section 5 is our conclusion.

Notation: Throughout the paper, we use the following conventions and notations. Bold faces denote vectors or matrices. x _i denotes the i ^th element of vector x, and A _ij denotes the element at row i and column j of matrix A. The unit simplex in the n-dimensional Euclidean space is denoted as Δn={x∈ℝn:x≥0, ∑k=1nxk=1} and its interior is denoted as ∆ _n . We will work with text collections with V dimensions (dictionary size). Each document d will be represented as frequency vector, d = (d ₁, .., d _V ) ^T , where d _j represents the frequency of term j in d. Denote n _d as the length of d, i.e., nd=∑jdj. The inner product of vectors u and v is denoted as 〈u,v〉. I(x) is the indicator function which returns 1 if x is true, and 0 otherwise and E(X) is expectation of random variable X.

2 Related Work

LDA [³] is the basic and famous model in topic modeling. It models each document as a probability distribution θ _d over topics, and each topic β _k as a probability distribution over words. In Fig. 1, K is number of topics, M is number of documents in corpus, N is number of words in each documents.

Fig. 1. Latent Dirichlet Allocation 

Note that θ _d ∈ ∆ _K , β _k ∈ ∆ _V . The generative process for each document d is as follows:

The most important problem we need to solve in order to use LDA is to compute the posterior distribution p(θ, z|w, α, β) of hidden variables in a given document d. However, it is intractable. There are many ways to handle it. Variational Bayesian Inference [³] approximates p(z _d , θ _d , d|β, α) by obtaining a lower bound on the likelihood which is adjustable by variational distributions. CVB and CVB0 deal with p(z _d , d|β, α), CGS draws samples from p(z _d , w|β, α) to estimate it. Eventually, all methods try to estimate the topic proportion θ _d .

In this paper, we infer topic proportion for a document directly by solving the Maximum a Posteriori Estimation (MAP) of θ _d given all words of this document and parameters of the model. The MAP estimation of topic mixture for a given document d:

using Bayes’ rule, we have:

Under the assumption about the generative process, problem (2) is equivalent to the following:

Within convex/concave optimization, problem (3) is relatively well-studied. In the case of α ≥ 1, it can easily be shown that the problem (3) is concave, and therefore it can be solved in polynomial time.

Unfortunately, in practice of LDA, the parameter α is often small, says α < 1, causing problem (3) to be non-concave. Sontag et al. in [¹⁴] has showed that problem (3) is NP-hard in the worst case when parameter α < 1. Consider problem (3) as a non-convex optimization problem, the gradient-based methods such as Gradient Descent (GD) and its variants are ineffective because of the existence of saddle points and flat regions, hence we need an effective random method to avoid them. OPE [¹⁶] is an efficient iterative algorithm for solving problem (3). It is a good solution in escaping saddle points and flat regions.

In the literature of iterative optimization algorithms, in each iteration, they try to build a tractable function that approximates true objective function, then optimize approximating function to reach the next point. The various algorithms have different techniques to build their own approximation. For example, using Jensen’s inequality, Expectation-Maximization (EM) [⁵] or Variational Inference (VI) [³] calculate the Evidence Lower Bound (ELBO) then maximize it. Gradient Descent constructs its quadratic approximation in each step and minimizes the quadratic. OPE solves the problem (3) by constructing an approximate sequence by stochastic way and solve it by Frank-Wolfe update formula [⁷].

Details of OPE is in Algorithm 1. The idea of OPE is quite simple. At each iteration t, it draws a sample function f _t (θ) and builds the approximation F _t (θ) which is the average of all previous sample function. The most interesting idea behind OPE is that the objective function is the sum of a likelihood and a prior. In each step, it builds an approximate function F _t (θ) by choosing either likelihood or prior with equal probabilities {0.5, 0.5}. That means when inferring about the topic proportion of a document, we use either the evidence of the document (likelihood) or knowledge we have known before (prior). This behavior is very natural to human. However, OPE considers likelihood and prior with the same contributions by using uniform distribution.

Algorithm 1. OPE: Online Maximum a Posteriori Estimation 

In fact, when humans deal with a new sample, one can rely on more likelihood if we have observed enough evidences, or rely on more prior knowledge if we have been lack of evidences. This simple idea leads us to build a more general and flexible version of OPE by using Bernoulli distribution instead of uniform distribution.

3 Generalized Online Maximum a Posteriori Estimation

In this section, we introduce our new algorithm, namely Generalized Online Maximum a Posteriori Estimation (G-OPE) based on OPE. OPE operates by choosing the likelihood or prior at each step t, then builds the approximation F _t (θ) which is the average of all parts draw from previous steps and current step. In G-OPE, in order to introduce the Bernoulli distribution into the sampling step, we need to modify the likelihood and prior so that the approximation function F _t (θ) → f(θ) as t → ∞. Denote:

then the true objective function f(θ) includes two components:

where g ₁(θ) and g ₂(θ) are the log likelihood and prior respectively.

Denote:

where G ₁(θ) and G ₂(θ) are the adjusted likelihood and prior respectively.

G-OPE is detailed in Algorithm 2. In Algorithm 2, f(θ) is the true objective function we need to maximize. At t ^th iteration, we draw sample function f _t (θ) from set of adjusted likelihood G ₁(θ) and prior G ₂(θ), then we build the approximate function F _t (θ). Because G-OPE is stochastic, in theory we consider T → ∞, where T is number of iterations for whole algorithm.

Algorithm 2. G-OPE: Generalized Online maximum a Posteriori Estimation 

We use Bernoulli distribution with parameter p to replace for uniform distribution in OPE. At t ^th iteration, we pick f _t (θ) as Bernoulli random variable with probability p from {G ₁(θ) , G ₂(θ)} where:

In statistic theory, as t increases (at least 20) and it is better to choose p not close to 0 or 1. Consider t independent Bernoulli trials with probabilities:

we build a stochastic approximate sequence:

We find out that F _t (θ) is the average of all sample functions drawn until current step.

So it is guaranteed to converge to f(θ) as t → ∞, which will be shown in Theorem 1. The Bernoulli parameter p controls how much likelihood part and prior part contribute to the objective function f(θ). We can utilize this point to choose the most suitable p in each circumstance. OPE is a special case of G-OPE when Bernoulli parameter p is chosen equal to 0.5. So OPE is not flexible in many datasets. G-OPE adapts well with different datasets, we will show it in the experiment section. In the rest of this section, we will show that G-OPE preserves the key advantage of OPE which is the guarantee of the quality and convergence rate. This character is unknown for the existing methods in posterior estimation in topic models.

Theorem 1 (Convergence of G-OPE algorithm)

Consider the objective function f(θ) inEq. 3, given fixed d, β, α, p. For G-OPE, with probability one, the followings hold:

1. For any θ ∈ ∆ _K , F _t (θ) converges to f(θ) as t → +∞.

2. θ _t converges to a local maximal/stationary point of f(θ).

Proof: Before the proof, we remind some notations: B(n, p) is binomial distribution with parameters n and p (Bernoulli distribution is a special case of the binomial distribution with n = 1), N(µ, σ ²) is normal distribution. E(X) and D(X) are expectation and variance of random variable X respectively.

We find out that problem 3 is the constrained optimization problem with the objective function f(θ) is non-convex. The criterion used for the convergence analysis is importance in non-convex optimization. For unconstrained problems, the gradient norm ‖∇f(θ)‖ is typically used to measure convergence, because ‖∇f(θ)‖ → 0 captures convergence to a stationary point. However, this criterion can not be used for constrained problems. Instead, we use the ”Frank-Wolfe gap” criterion in [¹³].

Denoted:

and

so, f(θ)=g1(θ)+g2(θ)=p.G1(θ)+(1−p)G2(θ).

Pick f _t follows the Bernoulli distribution from {G1(θ),G2(θ)} where:

Let a _t and b _t be the number of times that we have already picked G ₁(θ) and G ₂(θ) respectively after t iterations.

We find that a _t + b _t = t or b _t = t − a _t . We have a _t ∼ B(t, p) and E(a _t ) = t.p , D(a _t ) = t.p.(1 − p).

We have:

where S _t = a _t − t.p.

We have:

then S _t → N(0, tp(1 − p)) when t → ∞.

So S _t /t → 0 as t → ∞ with probability one. From (4), we conclude that the F _t → f as t → +∞ with probability one.

Consider:

Note that g ₁(θ), g ₂(θ) are Lipschitz continuous on Δ¯K. Hence there exists a constant L such that:

We have:

Since e _t and θ _t belong to Δ¯K then |〈G′1(θt)−G′2(θt),et−θt〉| and ‖et−θt‖2 are bounded above for any t.

Therefore, there exits a constant c ₁ > 0 such that:

Summing both sides of (5) for all t, we have:

As t → +∞, f(θ _t ) → f(θ*) due to the continuity of f(θ). As a result, (6) implies:

Note that Sh=O(h log⁡ h) [⁶], and hence Σh=1∞c1|Sh|h2 converges in probability one. Moreover, the term Σh=1+∞1h2 is bounded.

So Σh=1+∞1h〈F′h(θh),eh−θh〉 is bounded above.

Because et=arg⁡ max⁡x∈ΔK〈F′t(θt),x〉, so 〈F′t(θt),et−θt〉≥0.

If exists t0>0, c3>0 such as 〈F′t(θt),et−θt〉≥c3∀t>t0 then Σt=1∞1t〈F′t(θt),et−θt〉>Σt=1∞c3t. And because Σt=1∞1t is not bounded above, so Σh=1+∞1h〈F′h(θh),eh−θh〉→∞, which contradicts with the clause we claimed. Therefore:

〈F′t(θt),et−θt〉→0 as t→∞.

Since Stt→0, then 〈f′(θt),et−θt〉→0. Apply Frank-Wolfe gap criterion, θ* is stationary/local maximum of f, which completes the proof.

Besides, in the non-convex optimization field, the idea of how to build the approximate function in G-OPE can be utilized in the case of objective function f which is the sum of two parts f = g + h. In each step, choose g or h in Bernoulli distribution with parameter p, and adjust p to adapt with different circumstance. Randomness can help algorithms jump out of local minimum/maximum.

Therefore, to design new stochastic algorithms, we begin with a deterministic version, add a sequence of approximation in the G-OPE style, working with each approximation at each iteration by deterministic update formula. This is an open idea for our future works.

4 Experiments

In this section, we will investigate the performance of G-OPE in real world datasets. G-OPE can play as the core inference step when learning LDA, we will investigate the performance of G-OPE through the performance of Online-OPE [¹⁶] when changing its core inference method. So we derived Online-GOPE.

We conducted two experiments. The first one is the effect of parameter p in G-OPE when learning LDA and the second is in comparison Online-GOPE with the current state-of-the-art methods.

4.2 The Effect of Bernoulli Parameter p

In this experiment, we investigate how important the value of parameter p is. Because p ∈ (0, 1), and p is good if it is not close to 0 and 1. So we choose p respectively in {0.1, 0.15, ..., 0.9}, then run Online-GOPE in two datasets. We report the performance of Online-GOPE in Fig. 2 and Fig. 3. We can easly observe that p affects very much in the performance in terms of both measures. In Fig. 2, Online-GOPE reaches the best performance on New York Times for LPP measure at p = 0.35 and for NPMI measure at p = 0.75. In Fig. 3, Online-GOPE reaches the best performance on Pubmed for LPP measure at p = 0.4, for NPMI measure at p = 0.45.

Fig. 2. Online-GOPE with different values of p on New York Times 

Fig. 3. Online-GOPE with different values of p on Pubmed 

This results support our idea about the contributions of likelihood part and prior part of topic proportion inference for a document. The different dataset has the suitable value of p. If we want to get the best performance on the generalization or on semantics quality of topics, we have different p to choose. Therefore G-OPE is very flexible in the real world dataset.

The good values of p depend on how much likelihood part and prior part possess in total. The likelihood depends on the length of the documents. In our datasets, the average length of a document in New York Times is 329 while the average length of a document in Pubmed is 65. That explains why we have different best values of p for each dataset.

4.3 Comparison of G-OPE with Novel Algorithms

In this experiment, we compare Online-GOPE with the best value of p in previous experiment to the original Online-OPE and other methods: Online-VB, Online-CVB, Online-CGS. All of these algorithms try to learn the topics over the words β or variational parameters λ. The difference among these algorithms is the inner inference procedures.

The results is shown in Fig. 4 and 5. With suitable parameter p, we obtained G-OPE which was better than OPE, VB, CVB, and CGS on LPP measure. For NPMI measure, all algorithms perform the same, but G-OPE is one of the tops.

Fig. 4. Online-GOPE compares with Online-OPE, Online-VB, Online-CVB and Online-CGS on New York Times dataset. Higher is better 

Fig. 5. Online-GOPE compares with Online-OPE, Online-VB, Online-CVB and Online-CGS on Pubmed dataset. Higher is better 

This results show that Online-GOPE performs better than not only original OPE, but also the current novel methods. G-OPE works well because of the right choose of controlled parameter p.

5 Conclusion

We have discussed how posterior inference for individual texts in topic models can be done efficiently with our method. In theory, G-OPE remains the guarantee on quality and convergence rate of original OPE algorithm, which is the most important character among existing state-of-the-art inference methods. In practice, the parameter p of Bernoulli distribution in our method is a flexible way to deal with different datasets.

Besides, the spiritual idea in building approximation functions from G-OPE can be easily extended to a wide class of maximum a posteriori estimation or non-convex problems. By exploiting G-OPE carefully, we have derived an efficient method Online-GOPE for learning LDA from data streams or large corpora. As a result, it is the good candidate to help us to work with text streams and big data.

6 Predictive Probability

Predictive Probability shows the predictiveness and generalization of a model M on new data.

We followed the procedure in [¹²] to compute this measurement. For each document in a testing dataset, we divided randomly into two disjoint parts w _obs and w _ho with a ratio of 80:20. We next did inference for w _obs to get an estimate of 𝔼(θ ^obs ). Then we approximated the predictive probability as:

where ℳ is the model to be measured. We estimated E (β _k ) ∝ λ _k for the learning methods which maintain a variational distribution (λ) over topics. Log Predictive Probability was averaged from 5 random splits, each was on 1000 documents.

7 NPMI

NPMI measurement helps us to see the coherence or semantic quality of individual topics. According to [¹¹], NPMI agrees well with human evaluation on interpretability of topic models. For each topic t, we take the set {w ₁, w ₂, . . . , w _n } of top n terms with highest probabilities. We then computed:

where P (w _i , w _j ) is the probability that terms w _i and w _j appear together in a document. We estimated those probabilities from the training data. In our experiments, we chose top n = 10 terms for each topic. Overall, NPMI of a model with K topics is averaged as: