Linear Discriminant Analysis

#machine-learning/classification

Linear Discrminant Analysis is a dimensionality reduction and classification technique. It is a basis method for many classification algorithms, with a clear geometric interpretation.

How it works?

Suppose that we have -clusters with means with the same covariance matrix . This means that each cluster varies along some directions in the space.

LDA first rotates the data and homogenizes the scale so that each cluster has a equi-variance. Then, assign each point to a cluster with the center closest to the point, adjusted by the class popularity.

Fisher's disciminant analysis

Fisher discrminant analysis aims to find a linear discrminant in a different way, which turns out to be the same discriminant as the LDA. An interesting point is that Fisher does not make the assumption of gaussian on data, as opposed to LDA.

Fisher's idea is that a good subspace should maximize the distance between classes while minimizing the distance within each class, which can be translated into the following problem:

where and are the between and within class variance matrix defined as

Logistic regression

Let us revisit the case of two classes and assume that the data follows a gaussian distribution with means and and variance . Finding the best dicision boundary with LDA and the logistic regression is essentially the same.

Reference

• LDA, Fisher discriminant analysis, and logistic regression
• LDA and Fisher discriminant analysis - geometric interpretation