Limitations and generalisations of PCA

Proportion of variance

We want to establish a technique to select the more relevant \(PC_k\), to minimize loss of information when we reduce the dimension. So our main question is how to measure the amount of structure captured by each \(PC_k\).

The proportion of variance is what we need to use here. Recall that the variance explained by \(\phi_k\) is [_k = _k^T _k,] which is exactly the { \(k\)-th eigenvalue of \(\mathbf{\Sigma}\)}.

The proportion of variance explained by \(\phi_k\) is then

[ .]

A good dimension reduction technique must retain at least 80% of the original variance.

Possible issues

The procedure always works well in theory, but presents some issues when looking at data in real life.

  • If the original variables are highly uncorrelated, we need too many principal components to explain at least 80% of variance

  • By changing units of measurement, the variances also change, for the very same data!

  • Even when the units are the same, the standard deviation of the original variables can vary a lot (e.g. Iris data): the variables with higher standard deviation will be more represented in the first PCs.

Issue 1 is structural, so no solution can be found. Issues 2 and 3 can be solved by scaling the data set.

Scaling the original data

Let \(\mathbf{X}\) be the matrix of a data set. Scaling \(\mathbf{X}\) means replacing each column \(\mathbf{X}\) of \(\mathbf{X}\) with

[ ,] where \(\mathbf{\mu} = \begin{pmatrix} \mu \\ \mu\\ \vdots \\ \mu \end{pmatrix}\) is the mean vector of \(\mathbf{X}\) and \(sd\) is the standard deviation of \(\mathbf{X}\).

The covariance matrix of the scaled data is called the correlation matrix of the original data.

Other relevant notions for PCA are:

  • Correlation between PC and original variables: these are obtained as dot products of the loadings against the PCs
  • Quality of representation (cos2):
  • Absolute contributions (contrib):

Factor Analysis

Case Study

Accurate measurement of observations is a central component of the scientific process. However, some quantities (ex. intelligence'',love’‘, ``quality of life’’, ) are very hard to measure. These quantities are called {latent variables}.\[.5cm]

Direct measurement being impossible, one looks for those measurable features, { manifest variables}, that can tell us something about a latent variable. But how can we uncover the relationship between manifest and latent variables?\[.5cm]

Consider the case of researchers that want to measure the quality of life in several countries of the world. Possible associated manifest variables might be { life expectancies}, { socio-economic metrics}, { vaccination rates}, etc. How shall we use the observations to make statements about quality of life?

###{Latent variable models} One possible solution to the problem above is to perform { PCA} on the life expectancy data. Then, one can interpret the principal components as being latent variables, and the relation with the manifest variables is clear.

But there is a problem: this method is too and rarely produces good results.

A possible solution is to use factor analysis, a method employed in psychology, sociology, economics, marketing, health sciences, and many other disciplines.

Factor analysis works as follows: let \(\mathbf{X}=(X_1, \dots, X_n)\) be the random variable representing our features.

The goal of factor analysis is to find a smaller number of { factors \(f_1, \dots, f_q\)} \(\in \mathbb{R}^n\) explaining the behaviour of the observable variables.

FA considers multiple linear regressions

[X_1 = 1 + {11}f_1+ {21}f_2 + + {q1}f_q + 1] [ ] [X_p = p + {1p}f_1+ {2p}f_2 + + _{qp}f_q +_p]

where \(\mu_j\) is the mean vector of \(X_j\), and searches for the ``best possible’’ \(f_1, \dots, f_q\).

The factor model

If we put all the { loadings} \(\ell_{ij}\) in a matrix

[ = \[\begin{pmatrix} \ell_{11} & \ell_{12} & \cdots & \ell_{1p} \\ \ell_{21} & \ell_{22} & \cdots & \ell_{2p} \\ \vdots & \vdots & \vdots & \vdots \\ \ell_{q1} & \ell_{q2} & \cdots & \ell_{qp} \\ \end{pmatrix}\]

] and we set [ = ( f_1, , f_q), =(_1, , _p), =(_1, , _p),]

we find the regression model in matricial form:

[ = + + .]

Assumptions

It is useful to assume:

  • \(\mathbf{\mu}=E(\mathbf{\epsilon})=E(\mathbf{F})=0\),
  • \(\mathrm{Cov}(\epsilon_i, \epsilon_j)=0\) for \(i\neq j\),
  • \(\mathrm{Cov}(\epsilon_i, f_j)=0\) for all \(i=1, \dots, p\), \(j=1, \dots, q\),
  • \(\mathrm{Cov}(f_i, f_j)=0\) for \(i\neq j\) and \(\mathrm{Cov}(f_i, f_i)=1\) for all \(i=1, \dots, q\).

We set \(\mathbf{\Sigma}_{\epsilon}:=\mathrm{Cov}{(\mathbf{\epsilon})}\) and \(\mathbf{\Sigma}_{\chi}:= \mathbf{L}\mathbf{L}^T\). Then, the covariance matrix of \(\mathbf{X}\) is: [ X = ^T = {} + _{}.]

Note that \(\mathbf{\Sigma}_{\epsilon}\) is a diagonal matrix.

FA: parameter estimation

With the assumptions made, the remaining tasks of FA are:

  • Estimating the parameters \(\widehat{\mathbf{L}}\) and \(\widehat{\mathbf{\epsilon}}\) that represent the data more closely,
  • Choose the most suitable number \(q\) of factors,
  • Use the previous points to estimate the factors \(\widehat{\mathbf{F}}\).

For step 1, we have two methods:

  • Principal factor analysis (parameter estimation using PCA)
  • Maximum likelihood factor analysis (parameter estimation using MLE)

Principal factor analysis

Recall from PCA: performed via diagonalization of the estimated covariance matrix \(\widehat{\Sigma}_X\).

A possible approach is to note that the matrix \(\mathbf{L}\mathbf{L}^\top = \mathbf{\Sigma_X} - \mathbf{\Sigma}_{\epsilon}\) is symmetric and hence easy to diagonalize. Perform eigencomposition and retain only the first \(q\) eigenvalues and eigenvectors.

In practice, this is done in two steps:

[Step 1] Estimate \(\mathbf{\Sigma_X} - \mathbf{\Sigma}_{\epsilon}\) (i.e. replace the 1 on the diagonal of \(\Sigma_X\) with suitable values called ) [Step 2] Obtain first \(q\) eigenvalues \(\widehat{\lambda}_1 > \dots > \widehat{\lambda}_q\) and eigenvectors \((\widehat{\phi}_1, \dots, \widehat{\phi}_q)\)

The formula for the estimated factor loadings is then \[\begin{align*} \widehat{\ell}_{ij} = \widehat{\phi}_{ji} \sqrt{\widehat{\lambda}_j}. \end{align*}\]

Maximum likelihood factor analysis

Let us assume that the variable \(\mathbf{X}\) has a multivariate normal distribution.

Under this assumption, we can find an explicit form for the { likelihood function}.

In this setting, we can pick the loadings \(\ell_{ij}\) and diagonal matrix \(\mathbf{\Sigma}_{\epsilon}\) that maximise the likelihood function.

Explicitly, this means maximising the value of the { log likelihood}

where \(\mathbf{S}\) is the sample covariance matrix.

Number of factors

We now need to choose a suitable number of factors for our analysis (highly challenging!)

Some good choices to account for enough correlation:

  • In PFA, the number of factors \(q\) is selected by considering the variance explained by the first \(q\) eigenvalues of the covariance matrix: \[\begin{equation*} \frac{\lambda_{1} + \lambda_{2} + \cdots +\lambda_{q}}{\sum_{j=1}^{p} \lambda_{j}}. \end{equation*}\] and using { elbow method} or { 80% rule of thumb} to choose \(q\).
  • In MLFA, one can apply a { likelihood ratio test}. For each \(k=1, \dots, p\) the test statistics provides an index of fit for the model with \(k\) factors. This can be used to set up hypothesis testing and choose an appropriate \(k\). More details in Chapter 3 of .

Factor estimation

Recall the factor model \[\begin{align*} \mathbf{X} = \mathbf{L} \mathbf{F} + \mathbf{\epsilon}, \end{align*}\]

After estimating \(\mathbf{L}\) and \(\mathbf{\Sigma}_{\epsilon}\) we now estimate the factor scores \(\mathbf{F}\). The aim is to get an error \(\mathbf{\epsilon}\) as little as possible.

Problem: Find \(\widehat{\mathbf{F}}\) that minimizes the sum of the squared residuals: \[\begin{align*} (\mathbf{X} - \mathbf{L} \mathbf{F})^\top (\mathbf{X} - \mathbf{L} \mathbf{F}), \end{align*}\]

The solution is computed mathematically as: [ = (){-1} ^.]

In summary, factor analysis identifies hidden variables (factors) using observable variables as input and a .

Out of all possible factor models, we select those that are more closely representing the data collected. This is {done in three steps}:

  • Estimation of loadings and errors
  • Selection of best number of factors
  • Factor scores estimation

The fitness of the model is assessed looking at {cumulative variance explained} and/or {likelihood ratio test} (where MLFA is appropriate).

FA vs PCA

Principal component analysis and factor analysis are two ways of achieving dimension reduction, but they are quite different.

Let us compare the decomposition of the covariance matrix:

\[\begin{align*} \text{PCA} : \mathbf{\Sigma_X} =& \mathbf{\Sigma_{X[q]}} +\mathbf{\Sigma_{e}} \\ \text{FA} : \mathbf{\Sigma_X} =& \mathbf{\Sigma_{\chi}} +\mathbf{\Sigma_{\epsilon}} \end{align*}\]

where \(\mathbf{\Sigma_{X[q]}}\) and \(\mathbf{\Sigma_{\chi}}\) have rank \(q\).

  • PCA is performed by maximising the variance of the first \(q\) components and \(\mathbf{\Sigma_{e}}\) is not necessarily diagonal.
  • In FA, \(\mathbf{\Sigma_{\epsilon}}\) is assumed to be diagonal, but \(Var(\epsilon_i)\) may still be large! In contrast to FA, all entries of \(\mathbf{\Sigma}_{e}\) are minimised in PCA.
  • PCA is an algebraic results without statistical modelling, while FA relies on underlying model with assumptions on the data generating process.

Factor Analysis with R

The standard command for factor analysis in `R’ is . This uses { maximum likelihood factor analysis}, hence one needs to check that the normality assumption is reasonable.