There is a variety of problems where one searches for a vector v\mathbf{v} such that the squared sum of the lengths of projections of points xi\mathbf{x}_i onto that vector is maximized or minimized:

minvE(v)=i=1nxi,v2=i=1nxivxiv=i=1nvxixiv=vXXvs.t.vv=1 \begin{aligned} \min_v E(\mathbf{v}) &= \sum_{i=1}^{n} \langle \mathbf{x}_i, \mathbf{v} \rangle ^2 = \sum_{i=1}^{n} \mathbf{x}_i^\intercal\mathbf{v} \cdot \mathbf{x}_i^\intercal\mathbf{v} \\ &= \sum_{i=1}^{n} \mathbf{v}^\intercal\mathbf{x}_i \cdot \mathbf{x}_i^\intercal\mathbf{v} = \mathbf{v}^\intercal\mathbf{X}^\intercal\mathbf{X}\mathbf{v} \\ \\ \text{s.t.}\quad\mathbf{v}^\intercal\mathbf{v} &= 1 \end{aligned}

This can be written as a Lagrange function by introducing a Lagrangian multiplier λ\lambda for the constraint:

L(v,λ)=vXXvλ(1vv) \begin{aligned} \mathcal{L}(\mathbf{v}, \lambda) = \mathbf{v}^\intercal\mathbf{X}^\intercal\mathbf{X}\mathbf{v} - \lambda (1 - \mathbf{v}^\intercal\mathbf{v}) \end{aligned}

Optimization of the original problem is equivalent to finding a solution to the following equation system:

 ⁣v,λL(v,λ)=0{XXv=λvvv=1 \nabla\!_{\mathbf{v},\lambda} \mathcal{L}(\mathbf{v}, \lambda) = 0 \Leftrightarrow \begin{cases} \mathbf{X}^\intercal\mathbf{X}\mathbf{v} = \lambda\mathbf{v} \\ \mathbf{v}^\intercal\mathbf{v} = 1 \end{cases}

The first line of this equation tells us that v\mathbf{v} must be an eigenvector of XX\mathbf{X}^\intercal\mathbf{X} with eigenvalue λ\lambda. Let’s call those ve\mathbf{v}_e and λe\lambda_e. Plugging this solution back into our original problem (and remembering that the eigenvector is not necessarily unit length) we obtain:

E(veve)=veXXveveve=veλveveve=λ E\left(\frac{\mathbf{v}_e}{|\mathbf{v}_e|}\right) = \frac{\mathbf{v}_e^\intercal\mathbf{X}^\intercal\mathbf{X}\mathbf{v}_e}{\mathbf{v}_e^\intercal\mathbf{v}_e} = \frac{\mathbf{v}_e^\intercal\lambda\mathbf{v}_e}{\mathbf{v}_e^\intercal\mathbf{v}_e} = \lambda

Depending on whether we want to obtain a minimum or a maximum we can choose the eigenvector with the smallest or largest eigenvalue respectively.