\section{Orthonormal Bases }

The canonical/standard basis
\[
e_1=\colvec{1\\0\\ \vdots \\ 0}, \phantom{X}
e_2=\colvec{0\\1\\ \vdots \\ 0}, \phantom{X} \ldots, 
\phantom{X} e_n=\colvec{0\\0\\ \vdots \\ 1}
\]
has many useful properties.
\begin{itemize}
\item Each of the standard basis vectors has unit length:
\[
||e_i||=\sqrt{e_i\dotprod e_i}=\sqrt{e_i^Te_i}=1.
\]
\item The standard basis vectors are \emph{orthogonal}\index{Othogonal} (in other words, at right angles or perpendicular).  
\[
e_i\dotprod e_j = e_i^Te_j=0 \text{ when } i\neq j
\]
\end{itemize}

This is summarized by
\[
e_i^Te_j=\delta_{ij}=\left\{ \begin{array}{cc}
1 & \qquad i=j \\
0 & \qquad i\neq j \\
\end{array}\right. ,
\]
where $\delta_{ij}$ is the \emph{Kronecker delta}\index{Kronecker delta}.  Notice that the Kronecker delta gives the entries of the identity matrix.

Given column vectors $v$ and $w$, we have seen that the dot product $v\dotprod w$ is the same as the matrix multiplication $v^Tw$.  This is the \emph{inner product}\index{Inner product} on~$\Re^n$.  We can also form the \emph{outer product}\index{Outer product} $vw^T$, which gives a square matrix.

The outer product on the standard basis vectors is interesting.  Set 
\begin{eqnarray*}
\Pi_1 &=& e_1e_1^T \\
  &=& \colvec{1\\0\\ \vdots \\ 0} \rowvec{1&0& \ldots & 0}\\
  &=& \begin{pmatrix}
      1 & 0 & \ldots & 0 \\
      0 & 0 & \ldots & 0 \\
      \vdots & & & \vdots \\
      0 & 0 &\ldots & 0\\
      \end{pmatrix}\\
&\vdots& \\
\Pi_n &=& e_n e_n^T \\
  &=& \colvec{0\\0\\ \vdots \\ 1} \rowvec{0&0& \ldots & 1}\\
  &=& \begin{pmatrix}
      0 & 0 & \ldots & 0 \\
      0 & 0 & \ldots & 0 \\
      \vdots & & & \vdots \\
      0 & 0 &\ldots & 1\\
      \end{pmatrix}\\
\end{eqnarray*}
In short, $\Pi_i$ is the diagonal square matrix with a $1$ in the $i$th diagonal position and zeros everywhere else.
\footnote{This is reminiscent of an older notation, where vectors are written in juxtaposition.  This is called a \href{http://en.wikipedia.org/wiki/Dyadic_tensor}{``dyadic tensor''}\index{Dyad}, and is still used in some applications. }

Notice that 
$\Pi_i\Pi_j=e_ie_i^Te_je_j^T=e_i\delta_{ij}e_j^T$.  Then:
\[
\Pi_i\Pi_j = \left\{ \begin{array}{cc}
\Pi_i & \qquad i=j \\
0 & \qquad i\neq j \\
\end{array}\right. .
\]

Moreover, for a diagonal matrix $D$ with diagonal entries $\lambda_1,\ldots, \lambda_n$, we can write
\[
D= \lambda_1\Pi_1 + \ldots + \lambda_n\Pi_n.
\]

Other bases that share these properties should behave in many of the same ways as the standard basis.  As such, we will study:
\begin{itemize}
\item \emph{Orthogonal bases}\index{Othogonal basis} $\{v_1, \ldots, v_n \}$:
\[
v_i\dotprod v_j=0 \text{ if } i\neq j
\]
In other words, all vectors in the basis are perpendicular.

\item \emph{Orthonormal bases}\index{Orthonormal basis} $\{u_1, \ldots, u_n \}$:
\[
u_i\dotprod u_j = \delta_{ij}.
\]
In addition to being orthogonal, each vector has unit length.
\end{itemize}

Suppose $T=\{u_1, \ldots, u_n \}$ is an orthonormal basis for $\Re^n$.  Since $T$ is a basis, we can write any vector $v$ uniquely as a linear combination of the vectors in $T$:
\[
v=c^1u_1+\ldots c^nu_n.
\]
Since $T$ is orthonormal, there is a very easy way to find the coefficients of this linear combination.  By taking the dot product of $v$ with any of the vectors in $T$, we get:
\begin{eqnarray*}
v\dotprod u_i &=& c^1u_1\dotprod u_i + \ldots + c^iu_i\dotprod u_i + \ldots + c^nu_n\dotprod u_i \\
& = & c^1\cdot 0 + \ldots + c^i\cdot 1 + \ldots + c^n\cdot 0 \\
& = & c^i, \\
\Rightarrow c^i &=& v\dotprod u_i \\
\Rightarrow v &=& (v\dotprod u_1) u_1 + \ldots + (v\dotprod u_n)u_n\\
&=& \sum_i (v\dotprod u_i)u_i.
\end{eqnarray*}

This proves the theorem:
\begin{theorem}
For an orthonormal basis $\{u_1, \ldots, u_n \}$, any vector $v$ can be expressed
\[
v =\sum_i (v\dotprod u_i)u_i.
\]
\end{theorem}

\subsection{Relating Orthonormal Bases}

Suppose $T=\{u_1, \ldots, u_n \}$ and $R=\{w_1, \ldots, w_n \}$ are two orthonormal bases for $\Re^n$.  Then:\\[3mm]
\begin{eqnarray*}
w_1 &=& (w_1\dotprod u_1) u_1 + \ldots + (w_1\dotprod u_n)u_n\\
 & \vdots & \\
w_n &=& (w_n\dotprod u_1) u_1 + \ldots + (w_n\dotprod u_n)u_n\\
\Rightarrow w_i &=& \sum_j u_j(u_j\dotprod w_i) \\
\end{eqnarray*}
As such, the matrix for the change of basis from $T$ to $R$ is given by 
\[
P = (P^j_i) = (u_j\dotprod w_i).
\]

Consider the product $PP^T$ in this case.
\begin{eqnarray*}
(PP^T)^j_k&=&\sum_i(u_j\dotprod w_i)(w_i\dotprod u_k)\\
&=& \sum_i(u_j^T w_i) (w_i^T u_k)\\
&=& u_j^T \left[\sum_i (w_i w_i^T) \right] u_k \\
&=& u_j^T I_n u_k \qquad (*)\\
&=& u_j^T u_k = \delta_{jk}.
\end{eqnarray*}
In the equality $(*)$ is explained below.  So assuming $(*)$ holds, we have shown that $PP^T=I_n$, which implies that 
\[
P^T=P^{-1}.
\]

The equality in the line $(*)$ says that $\sum_i w_i w_i^T=I_n$.  To see this, we examine $(\sum_i w_i w_i^T)v$ for an arbitrary vector $v$.  We can find constants $c^j$ such that $v=\sum_j c^jw_j$, so that:
\begin{eqnarray*}
(\sum_i w_i w_i^T)v &=& (\sum_i w_i w_i^T)(\sum_j c^jw_j) \\
&=& \sum_j c^j \sum_i w_i w_i^T w_j \\
&=& \sum_j c^j \sum_i w_i \delta_{ij} \\
&=& \sum_j c^j w_j \text{ since all terms with $i\neq j$ vanish}\\
&=&v.
\end{eqnarray*}
Then as a linear transformation, $\sum_i w_i w_i^T=I_n$ fixes every vector, and thus must be the identity $I_n$.

\begin{definition}
A matrix $P$ is \emph{orthogonal}\index{Orthogonal matrix} if $P^{-1}=P^T$.
\end{definition}

Then to summarize,
\begin{theorem}
A change of basis matrix $P$ relating two orthonormal bases is an orthogonal matrix.  \textit{i.e.}
\[
P^{-1}=P^T.
\]
\end{theorem}

\begin{example}
Consider $\Re^3$ with the orthonormal basis 
\[
S=\left\{ 
u_1=\colvec{\frac{2}{\sqrt{6}}\\[1mm] \frac{1}{\sqrt{6}}\\[1mm] \frac{-1}{\sqrt{6}}},
u_2=\colvec{0\\[1mm] \frac{1}{\sqrt{2}}\\[1mm] \frac{1}{\sqrt{2}}},
u_3=\colvec{\frac{1}{\sqrt{3}}\\[1mm] \frac{-1}{\sqrt{3}}\\[1mm] \frac{1}{\sqrt{3}}}
\right\}.
\]
Let $R$ be the standard basis $\{e_1,e_2,e_3 \}$.  Since we are changing from the standard basis to a new basis, then the columns of the change of basis matrix are exactly the images of the standard basis vectors.  Then the change of basis matrix from $R$ to $S$ is given by:
\begin{eqnarray*}
P=(P^j_i)=(e_ju_i)&=&
\begin{pmatrix}
e_1\dotprod u_1 & e_1\dotprod u_2 & e_1\dotprod u_3 \\
e_2\dotprod u_1 & e_2\dotprod u_2 & e_2\dotprod u_3 \\
e_3\dotprod u_1 & e_3\dotprod u_2 & e_3\dotprod u_3 \\
\end{pmatrix} \\
= \begin{pmatrix}
u_1 & u_2 & u_3
\end{pmatrix} 
&=&
\begin{pmatrix}
\frac{2}{\sqrt{6}} & 0 & \frac{1}{\sqrt{3}} \\[1mm]
\frac{1}{\sqrt{6}}& \frac{1}{\sqrt{2}}&\frac{-1}{\sqrt{3}}\\[1mm]
\frac{-1}{\sqrt{6}}& \frac{1}{\sqrt{2}}&\frac{1}{\sqrt{3}}\\[1mm]
\end{pmatrix}. \\
\end{eqnarray*}

From our theorem, we observe that:
\begin{eqnarray*}
P^{-1}=P^T &=& \colvec{u_1^T\\u_2^T\\u_3^T} \\
&=& \begin{pmatrix}
\frac{2}{\sqrt{6}}& \frac{1}{\sqrt{6}}& \frac{-1}{\sqrt{6}}\\[1mm]
0 & \frac{1}{\sqrt{2}}&\frac{1}{\sqrt{2}} \\[1mm]
\frac{1}{\sqrt{3}}& \frac{-1}{\sqrt{3}}&\frac{1}{\sqrt{3}}\\[1mm]
\end{pmatrix}. \\
\end{eqnarray*}

We can check that $P^TP=I$ by a lengthy computation, or more simply, notice that
\begin{eqnarray*}
(P^TP)_{ij}&=& \colvec{u_1^T\\u_2^T\\u_3^T} \rowvec{u_1 & u_2& u_3} \\
&=& \begin{pmatrix}1&0&0\\0&1&0\\0&0&1\end{pmatrix}.
\end{eqnarray*}
We are using orthonormality of the $u_i$ for the matrix multiplication above.
\end{example}

\begin{remark}[Orthonormal Change of Basis and Diagonal Matrices.]
Suppose $D$ is a diagonal matrix, and we use an orthogonal matrix $P$ to change to a new basis.  Then the matrix $M$ of $D$ in the new basis is:
\[
M = PDP^{-1} = PDP^T.
\]
Now we calculate the transpose of $M$.
\begin{eqnarray*}
M^T &=& (PDP^T)^T\\
&=& (P^T)^TD^TP^T \\
&=& PDP^T\\
&=& M
\end{eqnarray*}
So we see the matrix $PDP^T$ is symmetric!
\end{remark}


\section*{References}
\begin{itemize}
\item Hefferon, Chapter Three, Section V: Change of Basis
\end{itemize}

Wikipedia:
\begin{itemize}
\item \href{http://en.wikipedia.org/wiki/Orthogonal_matrix}{Orthogonal Matrix}
\item \href{http://en.wikipedia.org/wiki/Diagonalizable_matrix}{Diagonalizable Matrix}
\item \href{http://en.wikipedia.org/wiki/Similar_matrix}{Similar Matrix}
\end{itemize}


\section*{Review Questions}

\begin{enumerate}
\item Let $D=\begin{pmatrix}
\lambda_1 & 0 \\
0 & \lambda_2 \\
\end{pmatrix}$.
\begin{itemize}
\item[$i.$] Write $D$ in terms of the vectors $e_1$ and $e_2$, and their transposes.
\item[$ii.$] Suppose $P=\begin{pmatrix}
a & b \\
c & d \\
\end{pmatrix}$ is invertible.  Show that $D$ is similar to
\[
M=\frac{1}{ad-bc}\begin{pmatrix}
\lambda_1ad-\lambda_2bc & (\lambda_1-\lambda_2)bd \\[1mm]
(\lambda_1-\lambda_2)ac & -\lambda_1bc + \lambda_2ad
\end{pmatrix}.
\]
\item[$iii.$] Suppose the vectors $\rowvec{a&b}$ and $\rowvec{c&d}$ are orthogonal.  What can you say about $M$ in this case?
\end{itemize}

\item Suppose $S=\{v_1, \ldots, v_n \}$ is an \emph{orthogonal} (not orthonormal) basis for $\Re^n$.  Then we can write any vector $v$ as $v=\sum_ic^iv_i$ for some constants $c^i$.  Find a formula for the constants $c^i$ in terms of $v$ and the vectors in $S$.

\item Let $u,v$ be independent vectors in $\Re^3$, and $P=\spa \{ u,v\}$ be the plane spanned by $u$ and $v$.  
\begin{itemize}
\item[$i.$] Is the vector $v^\bot = v-\frac{u\cdot v}{u\cdot u}u$ in the plane $P$?
\item[$ii.$] What is the angle between $v^\bot$ and $u$?
\item[$iii.$] Given your solution to the above, how can you find a third vector perpendicular to both $u$ and $v^\bot$?
\item[$iv.$] Construct an orthonormal basis for $\Re^3$ from $u$ and $v$.
\item[$v.$] Test your abstract formulae starting with 
\[
u=\rowvec{1 & 2 & 0} \text{ and } v=\rowvec{0 & 1 & 1}.
\]
\end{itemize}
\end{enumerate}

\newpage
