Linear Algebra Essentials: Part 2

Themistoklis Haris

We conclude our chapter on the Adjoint of a linear operator.

Our main theorems

THEOREM:

Let V be an inner product space and T,U be linear operators on V. Then:

  • (T+U)^* = T^* + U^*
  • (cT)^* = \bar{c}T^*
  • (TU)^* = U^* T^*
  • (T^*)^* = T
  • I^* = I

The proof of this theorem is based on the definition of the adjoint: \langle T(x), y \rangle = \langle x, T^*(y) \rangle.

COROLLARY

The above properties hold for matrices too (through the L_A operator)

APPLICATION: Least Squares Approximation

The definition of the problem is classic: Given n points (t_1, y_1), (t_2, y_2), . . . , (t_n, y_n), we are trying to minimize \sum\limits_{i=1}^n (y_i-ct_i-d)^2, with c being the line of best-fit slope and d is its offset. If

A = \left [ \begin{array}{cc} t_1 & 1 \\ t_2 & 1 \\ . & .\\.&.\\.&.\\ \\ t_n & 1 \end{array} \right ], x = \left[\begin{array}{cc}c\\d\end{array}\right]   and y = \left[\begin{array}{cc}y_1\\y_2\\.\\.\\.\\y_n\end{array}\right]

then we are basically trying to minimize ||y-Ax||^2. We will study a general method of solving this minimization problem when A is an m\times n matrix. This allows for fitting of general polynomials of degree n-1 as well. We assume that m \geq n.

If we let x \in \mathbb{R}^n and y \in \mathbb{R}^m, then it is easy to see that \langle Ax, y \rangle = \langle x, A^*y \rangle from the properties of the adjoint studied thus far. We also know that rank(A^* A) = rank (A) by the dimension theorem, because the nullities of the two matrices are the same (this is straightforward to see). Thus, if A has rank n, then A^* A is invertible.

Using these observations, our norm minimization problem can be solved through the minimality of orthogonal projections: Since m \geq n, the range of L_A is a subspace of \mathbb{R}^n. The projection of y \in \mathbb{R}^m onto W is a vector u = Ax_0 and it is such that ||Ax_0 - y|| \leq ||Ax-y|| for all x \in \mathbb{R}^n. To find x_0, let us note that u-y lies in W^{\perp}, so \langle Ax, Ax_0 - y \rangle = 0 for all x \in \mathbb{R}^n. In other words, \langle x, A^*(Ax_0 - y) \rangle = 0 so x_0 solves the equation A^* A x_0 = A^* y. If additionally the rank of A is n, we get that:

x_0 = (A^*A)^{-1} A^* y

APPLICATION: Minimal solutions to Systems of Linear Equations

How do we find minimal norm solutions to systems of linear equations? We prove the following theorem:

Theorem

Let A \in M_{m \times n}(F), b \in F^m and suppose that the system Ax=b is consistent. Then:

  • There exists exactly one minimal solution s to Ax=b and s\in R(L_A^*).
  • The vector s is the only solution to Ax=b that lines in R(L_A^*). So, if u satisfies (A A^*)u = b, then s = A^* u.

In other words, finding minimal norm solutions to systems like Ax=b can be done by finding any solution to the system AA^* u = b and letting x = A^* u.

Proof

The theorem follows relatively easily from the minimality of the projection onto a subspace (like before). We do, however, need that N(L_A) = (R(L_A^*))^{\perp}, a fact which holds for all linear transformations. This fact can be easily established, so we leave it as an exercise.