Feedback recurrent neural network-based embedded vector and its application in topic model

1.1.1 Related works

Statistical semantic models, such as latent semantic analysis (LSA) [3], probabilistic latent semantic analysis (PLSA) [4], and latent Dirichlet allocation (LDA) [5], are power tools for mining underlying topics in a document collection. PLSA is the probabilistic version of LSA, and both of them compute the similarities between documents with co-occurrences of terms, such as TF-IDF [6], but they ignore the order relationships between terms. For two sentences with synonyms, both LSA and PLSA consider them as unsimilar, but LDA can capture the semantics behind terms, and consider them as similar sentences. The LDA model assumes that a document collection is generated by distribution of topics, and each topic is a probabilistic distribution of terms. Based on LDA model, researchers also propose HDP-LDA [7], ADM-LDA [8], Tri-LDA [9], MB-LDA [10], and so on. Although the LDA and its extended models take the semantics between terms into consideration, they ignore the term orders in a document. For example, “Tom told Jim” and “Jim told Tom” have different semantics, but the LDA model can never differentiate them.

In recent years, embedded vector models, such as Word2vec [11], GloVe [12], and DSSM [13], are more and more popular, and deep learning for NLP has attracted more and more researchers. By embedding term frequency vectors into continuous multi-dimension vector space (dimension of continuous vector space is much smaller than that of one-hot vector), the similarities between documents can be accurately and quickly computed. During the generating of embedded vectors, Word2vec uses one of the neighbors of the target term, and GloVe uses the average of all neighbors of the target term to represent the target term. In addition, DSSM denotes a document with a set of triple characters, and this tri-character set can only capture the relationships between two adjacent terms.

In order to better capture the order relationships between terms, and thus improve the topic discovering effectiveness, this paper classifies the order relationships between terms into forward dependence and backward dependence and proposes a feedback recurrent neural network-based topic model.

1.1.2 Feedback recurrent neural network-based embedded vectors

In this section, we embed documents into a continuous vector space. Firstly, we generate an embedded vector for each document with simple recurrent neural network, then capture backward dependences of words by adding memory to each neural cell, and finally capture forward dependences by introducing feedback links.

Generating embedded vectors with recurrent neural network

With recurrent relationships, recurrent neural networks can model the relationship between the current word with its previous one. Theoretically, recurrent neural networks can be used to model sequences of any length.

Given a document collection D, after removing stop words, there are total T different words remained, so each word w _t (1 ≤ t ≤ T) can be denoted with a one-hot vector x _t . In this paper, we use a four-latent-layer recurrent neural network to compute the embedded vector for each document, and the structure of the proposed neural network is in Fig. 1. In Fig. 1, the arrows denote the dependences of data between neural cells; x _t and y _t are input and output of the t-th words respectively, and the dimension of latent layers is much smaller than the dimension of the input one-hot vector.

We apply Sigmoid function σ(x) = (1 + exp(−x))^− 1 as the activation function for each neural cell, where σ(x) is element-wise. While using the Sigmoid function, data of latent and output layers can be computed as follows.

$$ {\boldsymbol{h}}_t^1=\frac{1}{1+ \exp \left(-{\boldsymbol{U}}^1{\boldsymbol{h}}_{t-1}^1-{\boldsymbol{W}}^1{\boldsymbol{x}}_t\right)} $$

(1)

$$ {\boldsymbol{h}}_t^i=\frac{1}{1+ \exp \left(-{\boldsymbol{U}}^i{\boldsymbol{h}}_{t-1}^i-{\boldsymbol{W}}^i{\boldsymbol{h}}_t^{i-1}\right)},i=2,3,4 $$

(2)

$$ {\boldsymbol{y}}_t=\frac{1}{1+ \exp \left(-{\boldsymbol{U}}^4{\boldsymbol{h}}_{t-1}^4-{\boldsymbol{W}}^5{\boldsymbol{h}}_t^4\right)} $$

(3)

While learning parameters from documents, we use the output y _t of x _t as the prediction the next word x _t + 1, and the object function is given in Eq. 4.

$$ \min {\displaystyle \sum_{t=1}^{T-1}{\left({\boldsymbol{y}}_t-{\boldsymbol{x}}_{t+1}\right)}^2}+\lambda \left({\displaystyle \sum_{i=1}^5\left|\right|{\boldsymbol{W}}^i\left|\right|{}_2}+{\displaystyle \sum_{j=1}^4\left|\right|{\boldsymbol{U}}^j\left|\right|{}_2}\right) $$

(4)

Where \( {\displaystyle \sum_{t=1}^{T-1}{\left({\boldsymbol{y}}_t-{\boldsymbol{x}}_{t+1}\right)}^2} \) is the prediction loss, \( {\displaystyle \sum_{i=1}^5\left|\right|{\boldsymbol{W}}^i\left|\right|{}_2}+{\displaystyle \sum_{j=1}^4\left|\right|{\boldsymbol{U}}^j\left|\right|{}_2} \) is the regularization item, λ is the weight of the regularization item, and ||W||₂ is the sum of all squared elements in W.

With the document collection D, we can train the above model until stabilization and then get an embedded vector for each document. For each document d, we input x _t (t = 1, …, T) in turn, compute all latent vectors, and use \( {\boldsymbol{h}}_T^4 \) to represent the embedded vector of d, i.e., \( {\boldsymbol{v}}_d={\boldsymbol{h}}_T^4 \).

Adding memory to neural cell

For each neural cell of the above neural network, we apply long short-term memory (LSTM) [15] to represent the backward dependences between words. The structure of LSTM is in Fig. 2, and the computation for each unit is as follows.

$$ {\boldsymbol{i}}_t=\sigma \left({\boldsymbol{W}}_i{\boldsymbol{x}}_t+{\boldsymbol{U}}_i{\boldsymbol{h}}_{t-1}\right) $$

(5)

$$ {\boldsymbol{f}}_t=\sigma \left({\boldsymbol{W}}_f{\boldsymbol{x}}_t+{\boldsymbol{U}}_f{\boldsymbol{h}}_{t-1}\right) $$

(6)

$$ {\boldsymbol{o}}_t=\sigma \left({\boldsymbol{W}}_o{\boldsymbol{x}}_t+{\boldsymbol{U}}_o{\boldsymbol{h}}_{t-1}\right) $$

(7)

$$ {\boldsymbol{m}}_t={\boldsymbol{f}}_t\odot {\boldsymbol{m}}_{t-1}+{\boldsymbol{i}}_t\odot \sigma \left({\boldsymbol{W}}_m{\boldsymbol{x}}_t+{\boldsymbol{U}}_m{\boldsymbol{h}}_{t-1}\right) $$

(8)

$$ {\boldsymbol{h}}_t={\boldsymbol{o}}_t\odot \sigma \left({\boldsymbol{m}}_t\right) $$

(9)

In the above equations, the meaning of Sigmoid function σ(⋅) is the same as before and ⊙ is the dot product. Details of LSTM can be found in [15].

Adding feedback to recurrent neural network

LSTM-based recurrent neural networks can only capture backward dependences between words of a document. In order to capture forward dependences between words, we need to add feedback links between neural cells. Figure 3 illustrates our proposed feedback recurrent neural network, and this model can capture the relationships between x _t with x _t + 1 and x _t + 2.

In Fig. 3, each dotted line is a time slot, and the computation of \( {\boldsymbol{h}}_t^1 \) and \( {\boldsymbol{h}}_t^2 \) is the same as Fig. 1. However, the computation of \( {\boldsymbol{h}}_t^3 \) and \( {\boldsymbol{h}}_t^4 \) depends on three neural cells of the last time slot and can be computed as follows.

$$ {\boldsymbol{h}}_t^i=\frac{1}{1+ \exp \left(-{\boldsymbol{U}}^i{\boldsymbol{h}}_{t-1}^i-{\boldsymbol{W}}^i{\boldsymbol{h}}_t^{i-1}-{\boldsymbol{W}}^{i+3}{\boldsymbol{h}}_{t+1}^{i-2}\right)},i=3,4 $$

(10)

In the proposed four-latent-layer feedback recurrent neural network, we can capture the relationships between x _t with x _t + 1 and x _t + 2. If the number of latent layers is k, then we can capture the relationships between x _t with its following k − 2 words. So, the proposed feedback recurrent neural network can capture forward dependences between words.

1.1.3 Embedded vector-based topic model

From Section 1.1, we know that each document d ∈ D can be denoted with a vector v _d (\( {\boldsymbol{v}}_d={\boldsymbol{h}}_T^4 \)), where the dimension of v _d is p. Given d, we randomly choose a word w from d, substitute w with a random word w ' ≠ w and then get a muted document d '. For d and d ', the only difference is between w with w ', so we assume that they have similar topics. For choosing the muted word w ', we choose synonyms at the beginning, but the result is not very well. The reason is that by substituting a word with its synonym, the two documents are almost the same, so the generalization of the model is very weak.

After embedding d and d ' into continuous vector space, we get v _d and v '  _d . Then, we use the topic model described in Fig. 4 to analyze the underlying topics in a document collection. We assume that there are k topics in the document collection. Taking v _d and v '  _d as input, if the probability that both d and d ' belong to i-th topic is z _i , then we can estimate it as follows.

$$ {\widehat{z}}_i=\frac{1}{1+ \exp \left(-{\boldsymbol{W}}_i{\boldsymbol{v}}_d\boldsymbol{v}{\mathit{\hbox{'}}}_d\right)},i=1,\dots, k $$

(11)

where W _i is a p × p matrix, and the whole W is a p × p × k tensor.

1.1.4 Experiments

The experimental platform is a laptop with Intel Core i5-2450 2.5GHz CPU and 4GB memory. The operation system is Ubuntu 10.04, and all algorithms are implemented with Java.

1.1.5 Datasets

1.1.6 Baseline algorithms

We denote our proposed topic model with GRNN and compare it with PLSA [4], HDP-LDA [7], seTF-IDF [6], and DSSM [13]. PLSA is the basic probabilistic latent semantic analysis method, HDP-LDA extends LDA by modeling topics with hierarchical Dirichlet generation process, seTF-IDF is a term frequency—inverse document frequency method that takes semantics into consideration, and DSSM analyzes semantics by denoting each document with a set of triple characters.

1.1.7 Metrics

Comparison of efficiency We compared running time and maximum memory usage of algorithms under the two datasets, and the results are in Figs. 5 and 6 respectively. In the comparison of running time, GRNN is obviously better than other algorithms. Running time of PLSA, HDP-LDA, SeTF-IDF and DSSM under RML dataset is one half of that under NIPS, and running time of GRNN under RML is a quarter of that under NIPS. For maximum memory usage under the two datasets, GRNN use 520MB and 230MB respectively, and both of them are smaller than other algorithms.

1.1.8 Comparison of perplexity

By increasing the numbers of topics in both datasets, we compared the Perplexity of algorithms, and the results were in Figs. 7 and 8. In these two figures, Perplexities of PLSA and HDP-LDA decrease, Perplexity of SeTF-IDF does not have a trend, and Perplexities of DSSM and GRNN almost do not change. GRNN has the lowest Perplexity, DSSM has the second lowest Perplexity, and both of them are better than others.

1.1.9 Comparison of PMI

The same as comparison of Perplexity, we compared the PMI of algorithms by increasing the numbers of topics in both datasets, and the results were in Figs. 9 and 10. With the increasing of topic number, PMIs of all algorithms increase. In the NIPS dataset, when the number of topic is smaller than 45, PMI of DSSM is the biggest; and when the number of topic is bigger than 45, PMI of GRNN is the biggest. In the RML dataset, PMI of GRNN is bigger than others all the time.