Gaussian Mixture Models

In trying to learn about Gaussian mixture models, I've had some difficulty in simply reading about them, but I've found some videos that give a good explanation.  I will summarize my understanding below, and link to the videos at the end.

Essentially, a Gaussian mixture model is a way to combine several Gaussian PDFs (Probability Distribution Functions) of different shapes and sizes into a single distribution.  This is done by making a linear combination of the individual Gaussian PDFs.

Expectation-Maximization (EM) is a procedure that allows us to learn the parameters of the Gaussian mixture model.  These parameters are refined over several iterations.  The Expectation step (E-step) keeps fixed the mean μ_c, covariance Σ_c, and size π_c of the Gaussian, c.  The assignment probability, r_ic, that a each data point belongs to cluster c is then calculated.

In this example, the data point x is more likely to belong to distributyion 2 (66% chance) over distribution 1 (33% chance) 

In the Maximzation step (M-step) keeps the assignment probabilities the same while updating the parameters μ_c, Σ_c, and  π_c.  The assignment probabilities r_ic are used to calculate the new parameters.  Here, m represents the total data, whereas m_c represents the data corresponding to a particular cluster.

This way, the points with a large r_ic have more of an effect on the parameters than those with a small r_ic.  Each iteration of this EM model increases the log-likelihood of the model.  This can result in the model becoming stuck in local optima, so intialization is important.

Using Expectation-Maximization, the model can learn the parameters over time and refine a distribution given datapoints from overlapping distributions.

-----------------------------------
link to videos:
https://www.youtube.com/watch?v=Rkl30Fr2S38
https://www.youtube.com/watch?v=qMTuMa86NzU