\section{Taylor Polynomials}
In this section, we review the notion of Taylor polynomials for functions of one real variable before generalizing this idea to functions of $n$-many variables.

\subsection{A review of Taylor polynomials for one variable functions} If $f(x)$ is a single-variable function defined in a neighbourhood of $x = a$, and if $f' (a)$, $f'' (a)$,...,$f^{(k)} (a)$ exist, we define its \underline{$k$-th order Taylor polynomial} at $a$ as
$$
p_k (x) = f(a) + f'(a) (x-a) + \frac{f'' (a)}{2!} (x - a)^2 + \cdots + \frac{f^{(k)} (a)}{k!} (x - a)^k
$$
Then $p_k (x)$ is the unique $k$-th order polynomial such that
$$
f(a) = p_k (a), f' (a) = p'_k (a), ..., f^{(k)} (a) = p_k^{(k)} (a).
$$

\medskip
Let $R_k (x) = f(x) - p_k (x)$ be the remainder of $f$. This remainder satisfies:
\begin{itemize}
\item If $f$ is $k$ times differentiable in a neighbourhood of $a$, then
\begin{equation}\label{eq:35eq1}
\lim_{x \rightarrow a} \frac{R_k (a)}{(x - a)^k} = 0.
\end{equation}
(For $k = 1$, this is the condition from the definition of differentiability.)

\medskip

\item If $f$ is $C^{k+1}$ in a neighbourhood of $a$,  then
\begin{equation}\label{eq:35eq2}
R_k (x) = \frac{f^{(k+1)}(u)}{(k+1)!} (x-a)^{k+1},
\end{equation}
for some $u$ between $a$ and $x$.
\end{itemize}

Geometrically, for functions of one variable:
\begin{itemize}
\item The linear equation $y = p_1 (x)$ describes the tangent line at $x = a$.

\medskip

\item The quadratic equation $y = p_2 (x)$ describes the parabola that provides the best (among parabolas) approximation to $y = f(x)$ near $x = a$.  Remember: this idea is used in the second derivative test for critical points.
\end{itemize}

\noindent
(See your single-variable calculus textbook for proofs and more details.) We now seek an analogue of these ideas for functions of $n$ variables. The general case is accomplished in most textbooks. Here, we work out the second order Taylor approximation to $f(x,y)$ near $(a,b)$.

\subsection{Second order Taylor approximation} 
Assume $f(x,y)$ is $C^2$ in a neighbourhood of $(a,b)$, and consider the line
$$
(x(t),y(t))=(a+tu_1,b+tu_2), \ \ t\in\RR,
$$
for some fixed unit vector $\mathbf{u} = \langle u_1, u_2 \rangle$.  For this part of the argument, we think of the direction $\mathbf{u}$ as fixed and $t$ as our variable.



\begin{figure}[ht]\label{35fig1}
\centering
\includegraphics[width= .5\textwidth]{35fig1.png}
\end{figure}


Consider the function
\begin{equation}\label{eq:35eq10}
g(t) =f(x(t),y(t))= f(a + tu_1, b + t u_2).
\end{equation}
Write its second order Taylor polynomial at $t = 0$:
$$
p_2 (t) = g(0) + g' (0) \cdot t + \frac{g'' (0)}{2} \cdot t^2.
$$
Now,  $g(0) = f(a,b)$, and
$$
g' (0) = \frac{d}{dt} \big[ f(a + tu_1, b + t u_2 ) \big]\Big|_{t = 0} = D_{\mathbf{u}} f (a,b) = \nabla f (a,b) \cdot \mathbf{u}.
$$
Also, using the Chain Rule and then setting $t=0$, we have
\begin{eqnarray*}
g'' (0) & = & \frac{d}{dt} \big[ D_{\mathbf{u}} f(a + t u_1, b + t u_2 ) \big] \Big|_{t = 0} \\[1ex]
\quad &=&  \frac{d}{dt} \big[f_1(a + t u_1, b + t u_2) u_1 + f_2(a + t u_1, b + t u_2) u_2 \big]\Big|_{t = 0} \\[1ex]
 & = & f_{11} (a,b) u_1^2 + f_{12} (a,b) u_1u_2 + f_{21} (a,b) u_2 u_1 + f_{22} (a,b) u_2^2 .
 \\
\end{eqnarray*}
You have probably noticed that $f_{21}=f_{12}$ in the above equation if $f$ is $C^2$, and we will go back to that in a moment. It is useful, though, to start by rewriting the last line in matrix form. 
Define the \underline{Hessian matrix} of $f(x,y)$ as
$$
\mathscr{H}f=
\begin{pmatrix}
f_{11} & f_{12} \\
f_{21} & f_{22}
\end{pmatrix}.
$$
With this notation, the last line in our computation of $g''(0)$ is equal to
$$
 \langle x - a, y - b \rangle \cdot \mathscr{H}f(a,b) \begin{pmatrix} x - a \\ y - b \end{pmatrix}
$$
This matrix formula has the advantage of generalizing to higher dimensions (we will do that below). Additionally, if 
$f$ is $C^2$ on some domain, then $f_{12} = f_{21}$ on that domain as noted above, so that
$\mathscr{H}f$ is symmetric.






We now return to our formula for $p_2 (t)$, and substitute the above expressions for $g(0)$, $g'(0)$, $g''(0)$. We get
\begin{align*}
p_2 (t) &= f(a,b) + \big[ f_1 (a,b) u_1 + f_2 (a,b) u_2 \big] t \\
&+ \frac{1}{2} \big[f_{11} (a,b) u_1^2 + f_{12} (a,b) u_1 u_2 + f_{21} (a,b) u_2 u_1 + f_{22} (a,b) u_2^2 \big] t^2.
\end{align*}
Finally, converting back to the $x,y$ variables using $u_1 t = x - a$ and $u_2 t = y - a$,  this becomes
\begin{align*}
p_2 (x,y) & =  f(a,b) + f_1 (a,b) (x - a) + f_2 (a,b) (y-b) \\[1ex]
& + \frac{1}{2} \bigg[f_{11} (a,b) (x-a)^2 
 + f_{12} (a,b)(y-b)(x-a) \\[1ex]
 &+ f_{21} (a,b) (x-a)(y-b) + f_{22} (a,b) (y-b)^2 \bigg]
\end{align*}

This is our Taylor polynomial. In matrix form,
$$
p_2 = f(a,b) + \nabla f(a,b) \cdot \langle x - a, y - b \rangle + \frac{1}{2} \langle x - a, y - b \rangle \cdot \mathscr{H}f(a,b) \begin{pmatrix} x - a \\ y - b \end{pmatrix}
$$
Then $p_2 (x,y)$ is the unique second order polynomial in $x$ and $y$ such that $p_2 (a,b) = f(a,b)$ and all partials of $p_2$ of order $\leq 2$ match those of $f$. With the equality of mixed partials taken into account, we have
\begin{align*}
p_2 (x,y) & =  f(a,b) + f_1 (a,b) (x - a) + f_2 (a,b) (y-b) \\[1ex]
& + \frac{1}{2} f_{11} (a,b) (x-a)^2 
 + f_{12} (a,b)(x-a)(y-b) + \frac{1}{2}f_{22} (a,b) (y-b)^2 .
\end{align*}





\subsection{Remainder estimates}
We would like $p_2(x,y)$ to be a good approximation to $f(x,y)$ near $(a,b)$. In fact, we would like $p_2(x,y)$ to be a \underline{better} approximation to $f(x,y)$ than $L(x,y)$ was. Geometrically, the linear approximation $z=L(x,y)$ represents the best fit to $z=f(x,y)$ by a plane. The hope is that, for a curved surface $z=f(x,y)$, we may be able to get a better fit using curved quadric surfaces. Quantitatively, we would like the second order error term 
$R_2(x,y):=f(x,y)-p_2 (x,y)$ to satisfy
\begin{equation}\label{eq:35eq3}
R_2(x,y)=o((x-a)^2+(y-b)^2),
\end{equation}
the analogue of (\ref{eq:35eq1}) in 2 dimensions. For comparison, recall that the error term in linear approximation, $R(x,y)=f(x,y)-L(x,y)$, only satisfied 
$$R_2(x,y)=o\left(\sqrt{(x-a)^2+(y-b)^2}\right),
$$
 which is worse than (\ref{eq:35eq3}).
 
The estimate (\ref{eq:35eq3}) is in fact true if $f(x,y)$ is $C^2$ on a neighbourhood of $(a,b)$, and can be proved
by calculating the one-dimensional remainder estimates for the intermediate function $g(t)$ defined in \eqref{eq:35eq1}. If $f(x,y)$ is $C^3$ on a neighbourhood of $(a,b)$, then there is also an estimate analogous to (\ref{eq:35eq2}). See your textbook for details.

As an example,  consider that the second order Taylor polynomial $p_2 (x,y)$ of
$$
f(x,y) = (x^2 + y^2) e^{-(x^2 + y^2)},
$$
at $(a,b) = (0,0)$ is given by $p_2 (x,y) = x^2 + y^2$.  This follows from the calculations:
$$
f_x (0,0) = f_y (0,0) = f_{xy} (0,0) = f_{yx} (0,0) = 0
$$
$$
f_{xx} (0,0) = f_{yy} (0,0) = 2.
$$
(You should check this for yourself!) In polar coordinates,
$$
f(r, \theta) = r^2 e^{-r^2}, \quad p_2 (r, \theta) = r^2,
$$
so that
$$
\vert R (r, \theta) \vert = \vert f(r,\theta) - p_2 (r, \theta) \vert = r^2 \vert 1 - e^{-r^2} \vert =o(r^2).
$$
To get the little-o estimate at the end, we used that $1-e^{-r^2}\to 0$ as $r\to 0$. 
This shows that $R_2 (x,y)$ satisfies \eqref{eq:35eq3}. In fact, in this case we can do even better: since
$$
\lim_{r\to 0} \frac{r^2(1-e^{-r^2})}{r^3}= \lim_{r\to 0} \frac{1-e^{-r^2}}{r}= 
\lim_{r\to 0} \frac{-2re^{-r^2}}{1}=0
$$
by L'H\^opital's Rule, we actually have $\vert R (r, \theta) \vert =o(r^3)$. This means that, close to $(0,0)$, the Taylor polynomial $x^2+y^2$ is an excellent approximation to $f(x,y)$. However, the approximation becomes worse as you move away from the origin. 
See Figures 3.6 and 3.7.


\begin{figure}[ht!]
\centering
  \includegraphics[width=.8\textwidth]{35fig2.png}
  \caption{We see that, near the point $(a,b) = (0,0)$,  the surface $p_2$ is a good approximation of $f$.  The above shows $f(r, \theta)$ and $p_2 (r, \theta)$ for values of $r$ satisfying $|r| \leq 1$.}\label{35fig2}
\end{figure}


\begin{figure}[ht!]
  \centering
  \includegraphics[width=.87\textwidth]{35fig3.png}
  \label{fig:sub2}
\caption{As $r \rightarrow \infty$, we see that $p_2$ ``grows'' quadratically, while $f(x,y)$ flattens out like $e^{-r^2}$.  Contour lines are added to make the flatness of the surface $f(x,y)$ more easily seen.}\label{35fig3}
\end{figure}



\subsection{Taylor approximation in many dimensions} 
Now suppose $f = f(x_1,...,x_n)$ is $C^2$ in a neighbourhood of $\mathbf{a} = (a_1,...,a_n)$. We then have the Taylor approximation $f(\mathbf{x}) \approx p_2 (\mathbf{x})$, where
\begin{equation}\label{eq:35secondorderapprox}
p_2 (\mathbf{x}) : = f (\mathbf{a}) + \nabla f (\mathbf{a}) \cdot (\mathbf{x} - \mathbf{a}) + \frac{1}{2} (\mathbf{x} - \mathbf{a}) \cdot \mathscr{H}f(\mathbf{a}) (\mathbf{x} - \mathbf{a})^T,
\end{equation}
where $T$ denotes the matrix transpose and $\mathscr{H}f(\mathbf{a})$ is the $n \times n$ Hessian matrix
$$
\mathscr{H}f =
\begin{pmatrix}
f_{11} & f_{12} & \cdots & f_{1n} \\
f_{21} & f_{22} & \cdots & f_{2n} \\
\cdots & \cdots & \cdots & \cdots \\
f_{n1} & f_{n2} & \cdots & f_{nn}
\end{pmatrix}
$$
with all derivatives evaluated at ${\bf a}$. 
The function $p_2 (\mathbf{x}) = p_2 (x_1,...,x_n)$ is a polynomial in $n$ variables of degree at most $2$, so that its graph is a quadric surface in $n+1$ dimensions.




\section*{Worked Problems}
\begin{enumerate}[1.]
\item {\it If $f(x,y) = x^4 y + 2xy^3 - y^4$, find its second order Taylor approximation $p_2 (x,y)$ at $(a,b) = (1,2)$. }

\medskip
At $(a,b) = (1,2)$, we have $f(1,2) = 2$.  Moreover
\begin{eqnarray*}
f_x (x,y) = 4 x^3 y + 2 y^3 & \Rightarrow & f_x (1,2) = 24 \\
f_y (x,y) = x^4 + 6x y^2 - 4 y^3 & \Rightarrow & f_y (1,2) = -7 \\
f_{xx} (x,y) = 12 x^2 y & \Rightarrow & f_{xx} (1,2) = 24 \\
f_{xy} (x,y) = f_{yx} (x,y) = 4x^3 + 6 y^2 & \Rightarrow & f_{xy} (1,2) = f_{yx} (1,2) = 28 \\
f_{yy} (x,y) = 12 xy - 12 y^2 & \Rightarrow & f_{yy} (1,2) = -24
\end{eqnarray*}
\end{enumerate}

Hence, utilizing the formula for $p_2 (x,y)$ given in terms of the first- and second-order partial derivatives of $f$, we have
\begin{align*}
p_2 (x,y) & =  2 + 24 (x - 1) - 7 (y-2) 
\\
&\ \ + \frac{1}{2} \big[24 (x-1)^2 + 2 \cdot 28 (x-1)(y-2) - 24 (y-2)^2 \big]
\end{align*}


\section*{Practice Problems}
\begin{enumerate}[1.]
\item {
Find the second order Taylor polynomial of the function 
$f(x,y)=\sin(x+y^2)$ at $(\pi,0)$.}

\bigskip

\item {
Suppose that the function $f(x,y)$ and all its partial derivatives are continuous for all
$x,y$, and that its second order Taylor polynomial  
at $(a,b)$ is 
$$
p_2(x,y) = -7 + 4(x-a) -13(x-a)^2 - 6 (x-a)(y-b) + 10(y-b)^2.
$$
Find the partial derivatives $f_1(a,b)$, $f_{11}(a,b)$, and $f_{12}(a,b)$.}

\bigskip


\item { Let $f(x,y)=Ax^2+Bxy+Cy^2$ for some constants $A,B,C$. Prove that the second order Taylor polynomial
of $f(x,y)$ at every point $(a,b)$ in the $xy$-plane is equal to $f(x,y)$.}


\bigskip

\item {
Let $F(x,y)=xf(x+y)+y g(x+y)$, where $f,g$ are real-valued functions of one variable whose all
derivatives are continuous.

\begin{itemize}

\item[(a)] Prove that $F(x,y)$ satisfies the equation $$F_{xx}(x,y)-2F_{xy}(x,y)+F_{yy}(x,y)=0$$
 for all $x,y$.

\medskip

\item[(b)]
Find the second order Taylor polynomial of $F$ at the point $(0,0)$ in terms of the derivatives of $f,g$ at 0.
\end{itemize}
}


\bigskip


\item {\it In Sections 3.2 and 3.3, we have seen examples of functions that are not differentiable at $(0,0)$ even though their 
first order partials exist at that point. What happens then is, we can try to define the linear approximation $L(x,y)$  using the partials we have, but this $L(x,y)$ is not going to be a good approximation to $f(x,y)$ near $(0,0)$ (so that $z=L(x,y)$ is not a tangent plane to the graph of $z=f(x,y)$ in any reasonable sense).

In the question below, we see something similar happen for the second order Taylor approximation. The function is differentiable at $(0,0)$ and the second order partials exist at $(0,0)$, so that we can still set up the second order Taylor polynomial $p_2(x,y)$ at $(0,0)$. However, at that level of approximation things start to go wrong, so that $p_2(x,y)$ is not a better approximation to $f(x,y)$ than $L(x,y)$ was.

Warning: this is the simplest example of this type with an explicit formula for $f(x,y)$, but the calculations are still rather long.}

\medskip

{ Let }
$$f(x,y)=\left\{\begin{matrix}
\displaystyle{\frac{x^4}{x^2+y^2}}&\hbox{ if }(x,y)\neq (0,0)\\
0&\hbox{ if }(x,y)=(0,0)\\
\end{matrix}
\right.
$$

\medskip

\noindent { (a) Compute $f_1,f_2,f_{11},f_{12}, f_{21}, f_{22}$ at $(0,0)$. 
Be careful and make sure to use the
definition of the derivative where necessary. As an intermediate step, it may also be necessary to find the general expressions for some of these partials at points other than the origin.}


\medskip

{(b) Based on your answer from (a), write the second order Taylor polynomial $p_2(x,y)$ for
$f(x,y)$ at $(0,0)$. Is it true that 
$$\lim_{(x,y)\to (0,0)} \frac{f(x,y)-p_2(x,y)}{x^2+y^2}=0?
$$
 Prove your answer.}
\end{enumerate}





