Variational principle

A good way to visualize a matrix is to use it to define a conic - that is, a second degree surface - of the corresponding dimension. Given a symmetric matrix $M$, the conic could be defined as the contour for the value $c$ of the function $f( X ) = (X, M X) = c$. Another representation woud be to chose vectors of unit length (for which $(X, X) = 1$), graphing the values of $f$. Taking this point of view, and looking for stationary values of $f$, $X$ could be modified slightly by adding $\varepsilon E$, to obtain

$$\begin{eqnarray*} & & \frac{(X + \varepsilon E, M (X + \varepsilon E)) } { (X + \varepsilon E, X + \varepsilon E)}. \end{eqnarray*}$$

The denominator compensates for the change in length of X; alternatively a Lagrange multiplier could have been used. Using the bilinearity and symmetry of of the inner product, we get

$$\begin{eqnarray*} {\rm numerator} & = & (X, M X) + 2 \varepsilon (X, M E) + \... ...} & = & (X, X) + 2 \varepsilon (X, E) + \varepsilon^2 (E, E). \end{eqnarray*}$$

Dividing numerator and denominator by $(X, X)$, approximating $1/(1 + \varepsilon)$ by $(1 - \varepsilon)$, and extracting the term proportional to $\varepsilon$, one gets

$$\begin{eqnarray*} & & (E, M X) - (E, (X, M X) X), \end{eqnarray*}$$

which is supposed to vanish irrespective of E. If the bilinear form is positive definite, that can only happen when

$$\begin{eqnarray*} M X & = & (X, M X) X, \end{eqnarray*}$$

which is to say, when $X$ is an eigenvector and $(X, M X)$ is its eigenvalue.

Figure: The Ellipsoid Defined by a Quadratic Form

These ideas have been expressed in a variety of ways. For example, the CourantCourant minimax principe Minimax Principle maximizes the quadratic form within a lower dimensional subspace, then minimizes for all such subspaces. To appreciate this in the figure at left, intersect the ellipsoid with planes and find the semimajor exis in each plane. Then choose the smallest of them all. That will get the one along the y-axis, not the x-axis, because the lattter is not the longest axis in any ellipse at all.

The symmetry of M is not really a requirement, and the form (X, M Y) could be examined relative to independent variations of X and Y. The result would be separate equations for left and right eigenvectors, with the practical difficulty that the normalization by (X, X) would change to (X, Y) without a guarantee that the product would not vanish even when X and Y were non-zero. That possibility is excluded for normal matrices, but can readily occur whenever the Jordan normal form describes M; in fact it characterizes nontrivial Jordan decomposition. Also note that we have been using calculus arguments in an algebraic environment, which may not always be such a good idea; for integer matrices, say.