Hierarchical Dirichlet process

In statistics and machine learning, the hierarchical Dirichlet process (HDP) is a nonparametric Bayesian approach to clustering grouped data. It uses a Dirichlet process for each group of data, with the Dirichlet processes for all groups sharing a base distribution which is itself drawn from a Dirichlet process. This method allows groups to share statistical strength via sharing of clusters across groups. The base distribution being drawn from a Dirichlet process is important, because draws from a Dirichlet process are atomic probability measures, and the atoms will appear in all group-level Dirichlet processes. Since each atom corresponds to a cluster, clusters are shared across all groups. It was developed by Yee Whye Teh, Michael I. Jordan, Matthew J. Beal and David Blei and published in the Journal of the American Statistical Association in 2006, as a formalization and generalization of the infinite hidden Markov model published in 2002. In statistics and machine learning, the hierarchical Dirichlet process (HDP) is a nonparametric Bayesian approach to clustering grouped data. It uses a Dirichlet process for each group of data, with the Dirichlet processes for all groups sharing a base distribution which is itself drawn from a Dirichlet process. This method allows groups to share statistical strength via sharing of clusters across groups. The base distribution being drawn from a Dirichlet process is important, because draws from a Dirichlet process are atomic probability measures, and the atoms will appear in all group-level Dirichlet processes. Since each atom corresponds to a cluster, clusters are shared across all groups. It was developed by Yee Whye Teh, Michael I. Jordan, Matthew J. Beal and David Blei and published in the Journal of the American Statistical Association in 2006, as a formalization and generalization of the infinite hidden Markov model published in 2002. This model description is sourced from. The HDP is a model for grouped data. What this means is that the data items come in multiple distinct groups. For example, in a topic model words are organized into documents, with each document formed by a bag (group) of words (data items). Indexing groups by j = 1 , . . . J {displaystyle j=1,...J} , suppose each group consist of data items x j 1 , . . . x j n {displaystyle x_{j1},...x_{jn}} . The HDP is parameterized by a base distribution H {displaystyle H} that governs the a priori distribution over data items, and a number of concentration parameters that govern the a priori number of clusters and amount of sharing across groups. The j {displaystyle j} th group is associated with a random probability measure G j {displaystyle G_{j}} which has distribution given by a Dirichlet process: where α j {displaystyle alpha _{j}} is the concentration parameter associated with the group, and G 0 {displaystyle G_{0}} is the base distribution shared across all groups. In turn, the common base distribution is Dirichlet process distributed: with concentration parameter α 0 {displaystyle alpha _{0}} and base distribution H {displaystyle H} . Finally, to relate the Dirichlet processes back with the observed data, each data item x j i {displaystyle x_{ji}} is associated with a latent parameter θ j i {displaystyle heta _{ji}} : The first line states that each parameter has a prior distribution given by G j {displaystyle G_{j}} , while the second line states that each data item has a distribution F ( θ j i ) {displaystyle F( heta _{ji})} parameterized by its associated parameter. The resulting model above is called a HDP mixture model, with the HDP referring to the hierarchically linked set of Dirichlet processes, and the mixture model referring to the way the Dirichlet processes are related to the data items. To understand how the HDP implements a clustering model, and how clusters become shared across groups, recall that draws from a Dirichlet process are atomic probability measures with probability one. This means that the common base distribution G 0 {displaystyle G_{0}} has a form which can be written as: where there are an infinite number of atoms, θ k ∗ , k = 1 , 2 , . . . {displaystyle heta _{k}^{*},k=1,2,...} , assuming that the overall base distribution H {displaystyle H} has infinite support. Each atom is associated with a mass π 0 k {displaystyle pi _{0k}} . The masses have to sum to one since G 0 {displaystyle G_{0}} is a probability measure. Since G 0 {displaystyle G_{0}} is itself the base distribution for the group specific Dirichlet processes, each G j {displaystyle G_{j}} will have atoms given by the atoms of G 0 {displaystyle G_{0}} , and can itself be written in the form: