ODEs

The (matrix) exponential can be used to solve certain types of first-order linear systems of ordinary differential equations with non-constant coefficients: not only can we solve \begin{align}x’(t)&=x(t)+y(t) \\ y’(t)&=x(t)+y(t),\end{align} but we can also solve \begin{align}x’(t)&=tx(t)+y(t) \\ y’(t)&=x(t)+ty(t).\end{align}

Theorem 1. Let \(I\) be a connected open subset of \(\mathbb{R}\), let \(f:I\to E\) be differentiable and suppose that \(f’(t)=A(t)f(t)+b(t)\) for all \(t\in I\), where \(A:I\to L(E)\) and \(b:I\to E\) are continuous. Assume that \(A(s)A(t)=A(t)A(s)\) for all \(s,t\in I\). Choose any \(a\in I\). Then there exists some \(c\in E\) such that $$
f(t)=e^{\widehat{A}(t)} \left( c+\int_a^t e^{-\widehat{A}(s)}b(s)\,ds \right),
$$ where $$
\widehat{A}(t)=\int_a^t A(s)\,ds.
$$

Proof. Choose any \(a\in I\) and let $$
g(t)=e^{-\widehat{A}(t)}f(t)-\int_a^t e^{-\widehat{A}(s)}b(s)\,ds.
$$ It is easy to verify that \(A(t)=\widehat{A}’(t)\) commutes with \(\widehat{A}(t)\) for every \(t\in I\), so \begin{align}
g’(t) &= e^{-\widehat{A}(t)}(-\widehat{A}’(t))f(t)+e^{-\widehat{A}(t)}f’(t)-e^{-\widehat{A}(t)}b(t) \\
&= -e^{-\widehat{A}(t)}A(t)f(t)+e^{-\widehat{A}(t)}(A(t)f(t)+b(t))-e^{-\widehat{A}(t)}b(t) \\
&= 0
\end{align} and \(g\) is constant (see this theorem). \(\square\)

Suppose we have a (non-homogeneous) linear system of ODEs with constant coefficients. Then \(A(t)\) is constant and \(\widehat{A}(t)=tA\), so $$
f(t)=e^{tA} \left( c+\int_a^t e^{-sA}b(s)\,ds \right).
$$ If the system is homogeneous, then \(b=0\) and we simply have $$
f(t)=e^{tA}c.
$$ On the other hand, if \(E=\mathbb{R}\) then we just have the “integrating factor” method for solving ODEs of the form $$
x’(t)+P(t)x(t)=Q(t).
$$

Example. Consider the system \begin{align}
x’(t) &= 2tx(t)+y(t) \\
y’(t) &= y(t)+2tx(t),
\end{align} which can be written as $$
\begin{bmatrix}x’(t) \\ y’(t)\end{bmatrix} = \begin{bmatrix}2t & 1 \\ 1 & 2t\end{bmatrix}\begin{bmatrix}x(t) \\ y(t)\end{bmatrix} = A(t) \begin{bmatrix}x(t) \\ y(t)\end{bmatrix}.
$$ It is easy to see that \(A(t)A(s)=A(s)A(t)\) for all \(s,t\in\mathbb{R}\). We have $$\widehat{A}(t) = \begin{bmatrix}t^2 & t \\ t & t^2\end{bmatrix}.$$ Therefore the general solution is \begin{align}
\begin{bmatrix}x(t) \\ y(t)\end{bmatrix} &= e^{\widehat{A}(t)}\begin{bmatrix}C_1 \\ C_2\end{bmatrix} \\
&= \begin{bmatrix}
\frac{1}{2}C_{1}(e^{(t+1)t}+e^{(t-1)t})+\frac{1}{2}C_{2}(e^{(t+1)t}-e^{(t-1)t}) \\
\frac{1}{2}C_{1}(e^{(t+1)t}-e^{(t-1)t})+\frac{1}{2}C_{2}(e^{(t+1)t}+e^{(t-1)t})
\end{bmatrix}.
\end{align}

Determinant of the exponential

We now assume that \(V\) is a finite-dimensional real vector space.

Lemma 1. Let \(U\) be the open set of invertible operators in \(L(V)\). Then \(\det:U\to\mathbb{R}\) is differentiable, and $$
D\det(\tau)u=\det(\tau)\operatorname{tr}(\tau^{-1}u).
$$

Proof. Let \(\iota\) be the identity map on \(V\). It is easy to see that \(\det\) is differentiable, by choosing a basis for \(V\). Let \(f(s)=\det(\tau+su)=\det(\tau)\det(\iota+s\tau^{-1}u)\). Then $$
D\det(\tau)u=f’(0)=\det(\tau)\operatorname{tr}(\tau^{-1}u)
$$ since \(\det(\iota+s\tau^{-1}u)\) is a polynomial in \(s\) where the coefficient of \(s\) is \(\operatorname{tr}(\tau^{-1}u)\). \(\square\)

As a simple application of Theorem 1, we prove a well-known formula:

Theorem 2. For all \(\tau\in L(E)\), we have $$
\det(\exp(\tau))=\exp(\operatorname{tr}(\tau)).
$$

Proof. The exponential function is a map \(\exp:L(V)\to U\). Define \(\gamma:\mathbb{R}\to L(V)\) by \(\gamma(s)=s\tau\). Then \begin{align}
(\det\circ\exp\circ\gamma)’(s) &= D\det(\exp(s\tau))(D\exp(s\tau)\tau) \\
&= \det(\exp(s\tau))\operatorname{tr}(\exp(s\tau)^{-1}\exp(s\tau)\tau) \\
&= (\det\circ\exp\circ\gamma)(s)\operatorname{tr}(\tau).
\end{align} Therefore $$
(\det\circ\exp\circ\gamma)(s)=\exp(s\operatorname{tr}(\tau))
$$ by Theorem 1. \(\square\)

We will assume that all Banach algebras are unital.

Definition. Let \(E\) be a Banach algebra. If \(x\in E\), the exponential of \(x\) is $$
\exp(x)=e^x=\sum_{n=0}^\infty \frac{x^n}{n!},
$$ which converges absolutely for all \(x\). Thus we have a map \(\exp:E\to E\), called the exponential function.

The usual rules for power series apply. In particular, we can differentiate term by term inside the radius of convergence, which is infinite for the exponential function. Before doing this, we need a lemma (the proof is at the end of the post).

Lemma 1 (Power rule). Let \(E\) be a Banach algebra, let \(n\ge 0\), and let \(p_n:E\to E\) be the map defined by \(p_n(x)=x^n\). Then \(Dp_n(x)\) is the linear map given by $$
Dp_n(x)u=\sum_{k=0}^{n-1} x^kux^{n-k-1}.
$$ In particular, if \(E\) is commutative then \(Dp_n(x)\) is given by $$
Dp_n(x)u=nx^{n-1}u.
$$

Applying this lemma to the power series for \(\exp\) gives $$
D\exp(x)u=\sum_{n=1}^\infty \frac{1}{n!} \sum_{k=0}^{n-1}x^kux^{n-k-1}.\tag{*}
$$ Notice that when \(u\) commutes with \(x\), we have \(D\exp(x)u=\exp(x)u=u\exp(x)\). We also need another lemma (again, the proof is at the end of the post):

Lemma 2. For \(m,n\ge 0\), we have $$\int_0^1 s^m(1-s)^n\,ds=\frac{m!n!}{(m+n+1)!}.$$

Evaluation

Now we can evaluate the integral given at the beginning of the post. We have \begin{align}
\int_0^1 e^{sx}ue^{(1-s)x}\,ds &= \int_0^1 \sum_{m=0}^\infty\frac{s^m x^m}{m!}u\sum_{n=0}^\infty\frac{(1-s)^n x^n}{n!}\,ds \\
&= \sum_{m=0}^\infty\sum_{n=0}^\infty\frac{x^m u x^n}{m!n!}\int_0^1 s^m(1-s)^n\,ds \\
&= \sum_{m=0}^\infty\sum_{n=0}^\infty\frac{x^m u x^n}{(m+n+1)!},
\end{align} which is clearly equal to (*). (The rearrangements are valid because the infinite series are all absolutely convergent.) This proves our formula!

Proofs

Proof of Lemma 1. We use induction on \(n\). The case \(n=0\) is clear, so suppose that the result holds for \(n-1\). Since \(p_n(x)=xp_{n-1}(x)\), the product rule shows that \(Dp_n(x)\) maps \(u\) to \begin{align}
up_{n-1}(x)+xDp_{n-1}(x)u &= ux^{n-1}+x\sum_{k=0}^{n-2}x^kux^{n-k-2} \\
&= ux^{n-1}+\sum_{k=1}^{n-1}x^kux^{n-k-1} \\
&= \sum_{k=0}^{n-1} x^kux^{n-k-1}.
\end{align} \(\square\)

Proof of Lemma 2. We use induction on \(n\). The case \(n=0\) is obvious. Suppose the formula holds for \(n-1\). We have \begin{align}
\int_0^1 s^m(1-s)^n\,ds &= \int_0^1 s^m(1-s)^{n-1}(1-s)\,ds \\
&= \int_0^1 s^m(1-s)^{n-1}\,ds- \int_0^1 s^{m+1}(1-s)^{n-1}\,ds \\
&= \frac{m!(n-1)!}{(m+n)!} – \frac{(m+1)!(n-1)!}{(m+n+1)!} \\
&= \frac{m!(n-1)!(m+n+1)-(m+1)!(n-1)!}{(m+n+1)!} \\
&= \frac{m!n!}{(m+n+1)!}.
\end{align} \(\square\)

Theorem. Let \(A\subseteq\mathbb{R}^n\) be a convex open set and let \(f:A\to\mathbb{R}\). Suppose that \(f^{\prime\prime}\) exists on \(A\). Then \(f\) is convex if and only if \(f^{\prime\prime}(x)\) is positive semidefinite for all \(x\in A\).

Hessian matrices

Combining the previous theorem with the higher derivative test for Hessian matrices gives us the following result for functions defined on convex open subsets of \(\mathbb{R}^n\):

Let \(A\subseteq\mathbb{R}^n\) be a convex open set and let \(f:A\to\mathbb{R}\) be twice differentiable. Write \(H(x)\) for the Hessian matrix of \(A\) at \(x\in A\).

1. If \(f’(x)=0\) and \(H(x)\) is positive definite, then \(f\) has a strict local minimum at \(x\).
2. If \(f’(x)=0\) and \(H(x)\) is negative definite, then \(f\) has a strict local maximum at \(x\).
3. If \(f’(x)=0\) and \(H(x)\) has both positive and negative eigenvalues, then \(f\) does not have a local minimum or a local maximum at \(x\). That is, \(f\) has a saddle point at \(x\).
4. If \(H(x)\) is positive semidefinite for all \(x\in A\), then \(f\) is convex and has a strict global minimum at any \(x\) for which \(f’(x)=0\) and \(H(x)\) is positive definite.
5. If \(H(x)\) is negative semidefinite for all \(x\in A\), then \(f\) is concave and has a strict global maximum at any \(x\) for which \(f’(x)=0\) and \(H(x)\) is negative definite.

Since the determinant of a matrix is the product of its eigenvalues, we also have this special case:

Let \(A\subseteq\mathbb{R}^2\) be a convex open set and let \(f:A\to\mathbb{R}\) be twice differentiable. Write \(H(x)\) for the Hessian matrix of \(A\) at \(x\in A\): $$H(x)=\begin{bmatrix} a & b \\ b & d\end{bmatrix}.$$ (Note that \(a,b,d\) are functions of \(x\).)

1. If \(f’(x)=0\), \(ad-b^2 > 0\) and \(a > 0\), then \(f\) has a strict local minimum at \(x\).
2. If \(f’(x)=0\), \(ad-b^2 > 0\) and \(a < 0\), then \(f\) has a strict local maximum at \(x\).
3. If \(f’(x)=0\) and \(ad-b^2 < 0\), then \(f\) has a saddle point at \(x\).
4. If \(ad-b^2 \ge 0\) and \(a,d \ge 0\) for all \(x\in A\), then \(f\) is convex and has a strict global minimum at any \(x\) for which \(f’(x)=0\), \(ad-b^2 > 0\) and \(a > 0\).
5. If \(ad-b^2 \ge 0\) and \(a,d \le 0\) for all \(x\in A\), then \(f\) is concave and has a strict global maximum at any \(x\) for which \(f’(x)=0\), \(ad-b^2 > 0\) and \(a < 0\).

Proof of the theorem

We will prove a slightly more general statement.

Let \(E\) be a Banach space, let \(A\subseteq E\) be a convex open set, and let \(f:A\to\mathbb{R}\). We say that \(f\) is convex if $$
f(x+\lambda(y-x)) \le f(x)+\lambda(f(y)-f(x))
$$ for all \(x,y\in A\) and \(\lambda\in(0,1)\).

Theorem. Let \(A\subseteq E\) be a convex open set and let \(f:A\to\mathbb{R}\). Suppose that \(f^{\prime\prime}\) exists on \(A\). Then \(f\) is convex if and only if \(f^{\prime\prime}(x)\) is positive semidefinite for all \(x\in A\).

Here, \(f^{\prime\prime}(x):E\times E\to\mathbb{R}\) is a bilinear form and \(f^{\prime\prime}(x)\) is said to be positive semidefinite if \(f^{\prime\prime}(x)(h,h)\ge 0\) for all \(h\in E\).

The idea of the proof is quite simple: restrict \(f\) to line segments that lie in \(A\) and use the single variable case mentioned at the start of this post. In order to do this, we need a formula for the second derivative of a composition of maps between Banach spaces: \begin{align}
D^2(g\circ f)(x)(u,v) &= D^2 g(f(x))(Df(x)(u),Df(x)(v)) \\
&\qquad + Dg(f(x))(D^2 f(x)(u,v)).
\end{align} The proof, which is just a long computation, is included at the end of this post. Note that when \(f,g\) are maps from \(\mathbb{R}\) to \(\mathbb{R}\), we recover the formula $$
(g\circ f)^{\prime\prime}(x)=g^{\prime\prime}(f(x))[f'(x)]^2+g’(f(x))f^{\prime\prime}(x).
$$

Proof. Let \(x,y\in A\) and define \(\gamma:(-\varepsilon,1+\varepsilon)\to A\) by \(\gamma(\lambda)=x+\lambda(y-x)\), where \(\varepsilon > 0\) is chosen to be small enough. Then \(D\gamma(\lambda)(1)=y-x\) and \(D^2\gamma(\lambda)(1,1)=0\). Using the above formula, we get \begin{align}
(f\circ\gamma)^{\prime\prime}(\lambda) &= D^2(f\circ\gamma)(\lambda)(1,1) \\
&= f^{\prime\prime}(\gamma(\lambda))(y-x,y-x).\tag{*}
\end{align} If \(f^{\prime\prime}(z)\) is positive semidefinite for all \(z\in A\) then \((f\circ\gamma)^{\prime\prime}(\lambda)\ge 0\) for all \(\lambda\), so \(f\circ\gamma\) is convex. Then for all \(\lambda\in(0,1)\), $$
(f\circ\gamma)(0+\lambda(1-0)) \le (f\circ\gamma)(0)+\lambda[(f\circ\gamma)(1)-(f\circ\gamma)(0)]
$$ and $$
f(x+\lambda(y-x)) \le f(x)+\lambda(f(y)-f(x)),
$$ which proves that \(f\) is convex. Conversely, suppose that \(f\) is convex. Let \(z\in A\) and choose \(r > 0\) so that the open ball of radius \(r\) around \(z\) is contained in \(A\). Let \(|u| < r\) and set \(x=z-u/2\) and \(y=z+u/2\) in (*) so that \begin{align}
f^{\prime\prime}(z)(u,u) &= f^{\prime\prime}(\gamma(1/2))(y-x,y-x) \\
&= (f\circ\gamma)^{\prime\prime}(1/2) \\
&\ge 0
\end{align} since \(f\circ\gamma\) is convex. This shows that \(f^{\prime\prime}(z)\) is positive semidefinite for all \(z\in A\). \(\square\)

The second derivative formula

Lemma (Chain rule for the second derivative). Let \(E,F\) be Banach spaces. Let \(A\subseteq E\) and \(B\subseteq F\) be open sets. Let \(f:A\to F\) and \(g:B\to G\) with \(f(A)\subseteq B\). If \(D^2 f\) exists on \(A\) and \(D^2 g\) exists on \(B\), then \(D^2(g\circ f)\) exists on \(A\) and \begin{align}
D^2(g\circ f)(x)(u,v) &= D^2 g(f(x))(Df(x)(u),Df(x)(v)) \\
&\qquad + Dg(f(x))(D^2 f(x)(u,v)).
\end{align}

Proof. We can write \(D(g\circ f)=c\circ d\circ e\), where
\begin{align}
c:L(F,G)\times L(E,F) & \to L(E,G)\\
(\lambda,\mu) & \mapsto\lambda\circ\mu;\\
d:E\times E & \to L(F,G)\times L(E,F)\\
(x,y) & \mapsto((Dg\circ f)(x),Df(y));\\
e:E & \to E\times E\\
x & \mapsto(x,x).
\end{align} Note that \(c\) is a continuous bilinear map and \(e\) is a continuous
linear map. We compute \begin{align}
D^{2}(g\circ f)(x) & =Dc((d\circ e)(x))\circ Dd(e(x))\circ De(x)\\
& =Dc(Dg(f(x)),Df(x))\circ Dd(x,x)\circ e.
\end{align} Now $$
Dd(x,y)(u,v)=(D^{2}g(f(x))(Df(x)(u)),D^{2}f(y)(v)),
$$ so
\begin{align}
D^{2}(g\circ f)(x)(u) & =Dc(Dg(f(x)),Df(x))(Dd(x,x)(u,u))\\
& =Dc(Dg(f(x)),Df(x))(D^{2}g(f(x))(Df(x)(u)),D^{2}f(x)(u))\\
& =D^{2}g(f(x))(Df(x)(u))\circ Df(x)\\
& \qquad+Dg(f(x))\circ D^{2}f(x)(u)
\end{align} and \begin{align}
D^{2}(g\circ f)(x)(u)(v) & =D^{2}g(f(x))(Df(x)(u))(Df(x)(v))\\
& \qquad+Dg(f(x))(D^{2}f(x)(u)(v)).
\end{align} \(\square\)

In this final post, we are going to look at some applications of differentiation to locating maxima and minima of real valued functions. In order to do this, we will be using Taylor’s theorem (covered in part 2) to prove the higher derivative test for functions on Banach spaces, and the implicit function theorem (covered in part 4) to prove a special case of the method of Lagrange multipliers.

Consequences of Taylor’s theorem

Definition 35. Let \(f:X\to\mathbb{R}\) be a map defined on a topological space \(X\). If there is a neighborhood \(U\) of \(x\in X\) such that \(f(t)\le f(x)\) for all \(t\in U\), then we say that \(f\) has a local maximum at \(x\). Similarly, if \(f(t)\ge f(x)\) for all \(t\in U\) then we say that \(f\) has a local minimum at \(x\). If \(f\) has a local maximum or local minimum at \(x\), then we say that \(f\) has an extreme value at \(x\). If strict inequality holds, then we say that \(f\) has a strict local maximum or minimum.

In single variable calculus, a differentiable function \(f:\mathbb{R}\to\mathbb{R}\) has a local maximum or minimum at a point \(x\in\mathbb{R}\) only if \(f’(x)=0\). It is easy to extend this result to maps defined on Banach spaces.

Theorem 36. Let \(A\subseteq E\) be an open set and let \(f:A\to\mathbb{R}\). If \(f\) is differentiable at \(x\in A\) and has an extreme value at \(x\), then \(f’(x)=0\).

Proof. Let \(v\in E\) and let \(g(t)=x+tv\). Then \(f\circ g\) has an extreme value at \(0\), so \(0=(f\circ g)’(0)=f’(g(0))g’(0)=f’(x)v\). Therefore \(f’(x)=0\). \(\square\)

Also recall that if \(f:\mathbb{R}\to\mathbb{R}\) is of class \(C^1\) and there is a point \(x\in\mathbb{R}\) such that \(f’(x)=0\), then \(f(x)\) is a local minimum if \(f^{\prime\prime}(x) > 0\) and \(f(x)\) is a local maximum if \(f^{\prime\prime}(x) < 0\). There is a similar test for higher derivatives that follows from Taylor's theorem. Again, we can prove analogous statements for maps defined on Banach spaces.

If \(q\in L(E,\dots,E;\mathbb{R})\) is a multilinear map from \(E^p\) to \(\mathbb{R}\), then we say that \(q\) is a multilinear form.

Definition 37. Write \(h^{(p)}\) for the \(p\)-tuple \((h,\dots,h)\). We say that a form \(q\) is positive semidefinite if \(qh^{(p)} \ge 0\) for all \(h\) and positive definite if \(qh^{(p)} > 0\) for all \(h \ne 0\). The terms negative semidefinite and negative definite are defined similarly. If \(qh^{(p)}\) takes on both positive and negative values, then we say that \(q\) is indefinite.

Theorem 38 (Higher derivative test). Let \(A\subseteq E\) be an open set and let \(f:A\to\mathbb{R}\). Assume that \(f\) is \((p-1)\) times continuously differentiable and that \(D^p f(x)\) exists for some \(p\ge 2\) and \(x\in A\). Also assume that \(f’(x),\dots,f^{(p-1)}(x)=0\) and \(f^{(p)}(x)\ne 0\). Write \(h^{(p)}\) for the \(p\)-tuple \((h,\dots,h)\).

1. If \(f\) has an extreme value at \(x\), then \(p\) is even and the form \(f^{(p)}(x)h^{(p)}\) is semidefinite.
2. If there is a constant \(c\) such that \(f^{(p)}(x)h^{(p)}\ge c > 0\) for all \(|h|=1\), then \(f\) has a strict local minimum at \(x\) and (1) applies.
3. If there is a constant \(c\) such that \(f^{(p)}(x)h^{(p)}\le c < 0\) for all \(|h|=1\), then \(f\) has a strict local maximum at \(x\) and (1) applies.

Proof. By Corollary 24 and the given assumptions, we can write $$
f(x+h)-f(x)=\frac{1}{p!}f^{(p)}(x)h^{(p)}+\theta(h)|h|^p
$$ where \(\theta(h)\to 0\) as \(h\to 0\). First assume that \(f\) has an extreme value at \(x\). Choose a vector \(h_0\ne 0\) such that \(f^{(p)}(x)h_0^{(p)}\ne 0\). Then for sufficiently small \(t\in\mathbb{R}\) we have both $$
f(x+th_{0})-f(x)=\left(\frac{1}{p!}f^{(p)}(x)h_{0}^{(p)}\pm\theta(th_{0})\left|h_{0}\right|^{p}\right)t^{p}\tag{*}
$$ and $$
\left|\theta(th_{0})\right|\left|h_{0}\right|^{p}<\frac{1}{p!}f^{(p)}(x)h_{0}^{(p)}.
$$ For these \(t\), the sign of (*) is the same as the sign of \(f^{(p)}(x)h_0^{(p)}\). Since \(x\) is an extreme value, the sign of (*) must remain constant for small \(t\), which cannot happen unless \(p\) is even. Similarly, if \(f^{(p)}(x)h^{(p)}\) is not semidefinite then there is some vector \(h_1\ne 0\) such that \(f^{(p)}(x)h_1^{(p)}\) and \(f^{(p)}(x)h_0^{(p)}\) have opposite signs, which contradicts the fact that the sign of (*) is constant for small \(t\).

Now suppose that the condition in (2) holds. Then \begin{align}
f(x+h)-f(x) &= \frac{1}{p!}f^{(p)}(x)h^{(p)}+\theta(h)\left|h\right|^{p} \\
&= \left[\frac{1}{p!}f^{(p)}(x)\left(\frac{h}{\left|h\right|}\right)^{(p)}+\theta(h)\right]\left|h\right|^{p} \\
&\ge \left[\frac{c}{p!}+\theta(h)\right]\left|h\right|^{p}.
\end{align} Since \(\theta(h)\to 0\) as \(h\to 0\), the last term is positive for sufficiently small \(h\ne 0\). For these \(h\) we have \(f(x+h) > f(x)\), so \(f\) has a strict local minimum at \(x\). The proof for (3) is similar. \(\square\)

Corollary 39 (Higher derivative test, finite-dimensional case). In Theorem 38, further assume that \(E\) is finite-dimensional. Then \(h\mapsto f^{(p)}(x)h^{(p)}\) has both a minimum and maximum value on the set \(\{h\in E:|h|=1\}\), and:

1. If the form \(f^{(p)}(x)h^{(p)}\) is indefinite, then \(f\) does not have an extreme value at \(x\).
2. If the form \(f^{(p)}(x)h^{(p)}\) is positive definite, then \(f\) has a strict local minimum at \(x\).
3. If the form \(f^{(p)}(x)h^{(p)}\) is negative definite, then \(f\) has a strict local maximum at \(x\).

Proof. Since \(E\) is finite-dimensional, the set \(S=\{h\in E:|h|=1\}\) is compact. Therefore the continuous map \(h\mapsto f^{(p)}(x)h^{(p)}\) attains a minimum \(c\) and a maximum \(C\) on \(S\). Part (1) follows directly from part (1) of Theorem 38. If \(f^{(p)}(x)h^{(p)}\) is positive definite then \(c > 0\), so part (2) of Theorem 38 applies. If \(f^{(p)}(x)h^{(p)}\) is negative definite then \(C < 0\), so part (3) of Theorem 38 applies. \(\square\)

The simplest form of Corollary 39 occurs when \(p=2\). Let \(E\) be an \(n\)-dimensional real Banach space, let \(A\subseteq E\) be an open set, and let \(f:A\to\mathbb{R}\) be a class \(C^1\) map. Suppose that \(f^{\prime\prime}(x)\) exists at \(x\in A\). Let \(\{e_1,\dots,e_n\}\) be a basis for \(E\) so that \(E=E_1\times\cdots\times E_n\), where \(E_i\) is the subspace generated by \(e_i\).

Definition 40. The Hessian matrix of \(f\) at \(x\) is the real matrix $$
\begin{bmatrix}
D_1 D_1 f(x) & \cdots & D_1 D_n f(x) \\
\vdots & \ddots & \vdots \\
D_n D_1 f(x) & \cdots & D_n D_n f(x)
\end{bmatrix},
$$ where each element $$
D_i D_j f(x) \in L(E_i,L(E_j,\mathbb{R}))
$$ is identified with \(D_i D_j f(x)(e_i,e_j)\in\mathbb{R}\).

Theorem 29 shows that this matrix is symmetric. We can restate Corollary 39 in terms of the Hessian matrix.

Corollary 41. Suppose that \(f’(x)=0\) and \(f^{\prime\prime}(x)\) exists. Let \(H\) be the Hessian matrix of \(f\) at \(x\).

1. If \(H\) has both positive and negative eigenvalues, then \(f\) does not have an extreme value at \(x\).
2. If \(H\) is positive definite, then \(f\) has a strict local minimum at \(x\).
3. If \(H\) is negative definite, then \(f\) has a strict local maximum at \(x\).

Proof. It is clear that \(f^{\prime\prime}(x)(h,h)=\widetilde{h}^T H \widetilde{h}\), where \(\widetilde{h}\) is the column vector representing \(h\). \(\square\)

Lagrange multipliers

The method of Lagrange multipliers provides a necessary condition for a function \(f:A\to\mathbb{R}\) to be maximized or minimized subject to a constraint expressed as a function \(g:A\to\mathbb{R}\). We first need an elementary result from linear algebra.

Lemma 42. Let \(f,g:E\to\mathbb{R}\) be nonzero linear functionals. If \(\ker f\subseteq \ker g\), then \(f=\lambda g\) for some \(\lambda\in\mathbb{R}\).

Proof. \(\ker f\) cannot be a strict subset of \(\ker g\) since \(\dim(E/\ker f)=1\), so \(\ker f=\ker g\). Let \(v\notin \ker f\) and take \(\lambda=f(v)/g(v)\). Clearly \(f=\lambda g\) on \(\ker f=\ker g\). If \(x\notin \ker f\) then \(x=rv\) for some \(r\in\mathbb{R}\), so $$
f(x)=rf(v)=\lambda rg(v)=\lambda g(x).
$$ Therefore \(f=\lambda g\) on \(E\). \(\square\)

Theorem 43 (Method of Lagrange multipliers, single constraint). Let \(A\subseteq E\) be an open set. Let \(f:A\to\mathbb{R}\) and \(g:A\to\mathbb{R}\) be of class \(C^1\), and let \(S=g^{-1}(\{0\})\). If \(f|_S\) has an extreme value at \(x\in S\) and \(g’(x)\ne 0\), then there is a number \(\lambda\in\mathbb{R}\) such that \(f’(x)=\lambda g’(x)\).

Proof. If we can prove that \(\ker g’(x)\subseteq\ker f’(x)\) then the result follows from Lemma 42. Choose some \(w\notin\ker g’(x)\), let \(F=\ker g’(x)\) and let \(G=\langle w \rangle\); then \(E=F\oplus G\). Let \(B=A\cap F\) and let \(C=A\cap G\). Write \(x=(x_1,x_2)\) where \(x_1\in B\) and \(x_2\in C\). Since \(g’(x)\ne 0\), \(D_2 g(x)\) is invertible; we also have \(g(x_1,x_2)=0\). By the implicit function theorem, there exists a neighborhood \(U\subseteq B\) of \(x_1\) and a \(C^1\) map \(h:U\to C\) such that \(h(x_1)=x_2\) and \(g(x_1,h(x_1))=0\). Let \(\widetilde{h}:U\to A\) be given by \(t\mapsto(t,h(t))\) so that \(\widetilde{h}(U)\subseteq S\) and \(h’(x_1)|_F\) is the identity map on \(F\). Since \(f|_S\) has an extreme value at \(x\) we have \((f\circ\widetilde{h})’(x_1)=0\) by Theorem 36, so \(f’(x)\circ \widetilde{h}’(x_1)=0\) by the chain rule. In particular, if \(v\in\ker g’(x)=F\) then $$
0=[f'(x)\circ\widetilde{h}'(x_1)](v)=f’(x)v,
$$ so \(v\in\ker f’(x)\). \(\square\)

There is also a more general version for a constraint function that maps into an infinite-dimensional space. We omit the proof because it requires a few theorems from functional analysis.

Theorem 44 (Method of Lagrange multipliers, multiple constraints). Let \(A\subseteq E\) be an open set. Let \(f:A\to\mathbb{R}\) and \(g:A\to F\) be of class \(C^1\), and let \(S=g^{-1}(\{0\})\). If \(f|_S\) has an extreme value at \(x\in S\) and \(g’(x)\) is surjective, then there is a continuous linear map \(\lambda:F\to\mathbb{R}\) such that \(f’(x)=\lambda\circ g’(x)\).

Conclusion

We have seen how the theorems of multivariable calculus in \(\mathbb{R}^n\) generalize easily to more general Banach spaces. Because we can work coordinate-free, the proofs are often easier to understand than their \(\mathbb{R}^n\) counterparts. By constructing the derivative on Banach spaces, we gain a powerful tool that allows us to both do computations and prove things much more easily than before.

Navigation: 1. The derivative | 2. Higher derivatives | 3. Partial derivatives | 4. Inverse and implicit functions | 5. Maxima and minima

Now we’re going to prove the inverse function and implicit function theorems for Banach spaces.

Theorem 32 (Contraction principle). Let \((X,d)\) be a complete metric space and let \(\varphi:X\to X\) be a map satisfying $$
d(\varphi(x),\varphi(y)) \le cd(x,y)
$$ for all \(x,y\in X\) and some constant \(c < 1\). Then there is exactly one \(x\in X\) for which \(\varphi(x)=x\).

Proof. Choose any \(x_0\in X\) and define \(x_{n+1}=\varphi(x_n)\). For all \(n\ge 1\) we have $$
d(x_{n+1},x_n)=d(\varphi(x_n),\varphi(x_{n-1}))\le cd(x_n,x_{n-1}),
$$ so \(d(x_{n+1},x_n)\le c^n d(x_1,x_0)\) by induction. For all \(m > n\), \begin{align}
d(x_n,x_m) &\le d(x_n,x_{n+1})+\cdots+d(x_{m-1},x_m) \\
&\le (c^n+\cdots+c^{m-1})d(x_1,x_0) \\
&\le c^n(1-c)^{-1}d(x_1,x_0),
\end{align} which shows that \(\{x_n\}\) is a Cauchy sequence. Since \(X\) is complete, \(x_n\to x\) for some \(x\in X\). Furthermore, $$
x=\lim_{n\to\infty}x_{n+1}=\lim_{n\to\infty}\varphi(x_n)=\varphi(x)
$$ since \(\varphi\) is continuous. Uniqueness is obvious. \(\square\)

Theorem 33 (Inverse function theorem). Let \(A\subseteq E\) be an open set and let \(f:A\to F\) be of class \(C^p\) (with \(p\ge 1\)). Suppose that \(f’(p)\) is invertible for some \(p\in A\). Then there is a neighborhood \(U\subseteq A\) of \(p\) such that \(f(U)\) is open and \(f|_U:U\to f(U)\) is a \(C^p\) diffeomorphism.

Proof. Let \(\iota:E\to E\) be the identity map. By replacing \(f\) with \(f’(p)^{-1}\circ f\), we may assume that \(E=F\) and \(f’(p)=\iota\). Since \(f’\) is continuous at \(p\), there exists an open ball \(U\subseteq A\) around \(p\) such that \(|f’(x)-\iota| < \frac{1}{2}\) for all \(x\in U\). For \(y\in f(U)\), define the map \(\varphi_y(x)=x-f(x)+y\). Note that \(x\) is a fixed point of \(\varphi_y\) if and only if \(f(x)=y\). For \(y\in f(U)\) we have \(|\varphi'_y(x)|=|f'(x)-\iota|<\frac{1}{2}\) for all \(x\in U\), so by Corollary 16 we have $$
|\varphi_y(x_1)-\varphi_y(x_2)| \le \frac{1}{2}|x_1-x_2|\tag{*}
$$ for all \(x_1,x_2\in U\). Using the uniqueness argument in Theorem 32, we conclude that \(f|_U:U\to f(U)\) is a bijection.

Now let \(b\in f(U)\) so that \(b=f(a)\) for some \(a\in U\). Let \(B\) be an open ball with radius \(r\) around \(a\) such that \(\overline{B}\subseteq U\), and let \(B’\) be an open ball of radius \(r/2\) around \(b\). We want to show that \(B’\subseteq f(U)\), thus proving that \(f(U)\) is open. Let \(y\in B’\). If \(x\in\overline{B}\) then \begin{align}
|\varphi_y(x)-a| &\le |\varphi_y(x)-\varphi_y(a)|+|\varphi_y(a)-a| \\
&< \frac{1}{2}|x-a|+|y-b| \\
&< r,
\end{align} so \(\varphi_y(x)\in B\). This together with (*) shows that \(\varphi_y|_{\overline{B}}:\overline{B}\to\overline{B}\) is a contraction mapping, and since \(\overline{B}\) is complete we can apply Theorem 32 to obtain a fixed point \(x\in\overline{B}\) of \(\varphi_y|_{\overline{B}}\), which implies that \(f(x)=y\) and \(y\in f(U)\).

For the last part of the proof, we denote \(f|_U\) by \(f\) and \((f|_U)^{-1}\) by \(f^{-1}\) for convenience. Let \(y\in f(U)\) and \(y+k\in f(U)\) with \(k\ne 0\); there exist \(x\in U\) and \(x+h\in U\) with \(y=f(x)\) and \(y+k=f(x+h)\), noting that \(h\ne 0\). In fact we have \begin{align}
|h-k| &= |h-f(x+h)+f(x)| \\
&= |\varphi_y(x+h)-\varphi_y(x)| \\
&\le \frac{1}{2}|h|
\end{align} from (*), so \(|h|\le 2|k|\). Then \(h\to 0\) as \(k\to 0\) and \begin{align}
\frac{|f^{-1}(y+k)-f^{-1}(y)-f'(x)^{-1}k|}{|k|} &= \frac{|f'(x)^{-1}(f(x+h)-f(x))-h|}{|k|} \\
&\le |f'(x)^{-1}|\frac{|f(x+h)-f(x)-f'(x)h|}{|k|} \\
&\le 2|f'(x)^{-1}|\frac{|f(x+h)-f(x)-f'(x)h|}{|h|} \\
&\to 0
\end{align} as \(h\to 0\). (Note that \(f'(x)\) is invertible since \(|f'(x)-\iota|<\frac{1}{2}\).) This proves that $$
(f^{-1})'(y)=f'(x)^{-1}=f'(f^{-1}(y))^{-1},\tag{**}
$$ so \(f^{-1}\) is continuous and differentiable on \(f(U)\). Furthermore, (**) shows that \((f^{-1})'\) is of class \(C^p\) since the maps \(f^{-1}\), \(f'\) and \(\lambda\mapsto\lambda^{-1}\) (operator inversion) are all of class \(C^p\). \(\square\)

Theorem 34 (Implicit function theorem). Let \(A\subseteq E\) and \(B\subseteq F\) be open sets and let \(f:A\times B\to G\) be of class \(C^p\) (with \(p\ge 1\)). Suppose \((a,b)\in A\times B\) such that \(f(a,b)=0\) and \(D_2 f(a,b):F\to G\) is invertible. Then there exists a neighborhood \(U\) of \(a\) and a \(C^p\) map \(g:U\to B\) with the following properties:

1. \(g(a)=b\).
2. \(f(x,g(x))=0\) for all \(x\in A\).
3. \(g’(a)=-[D_2 f(a,b)]^{-1}\circ D_1 f(a,b)\).

Proof. Let \(\iota:E\to E\) be the identity map. Define \begin{align}
\widetilde{f}:A\times B &\to E\times G \\
(x,y) &\mapsto (x,f(x,y))
\end{align} and compute $$
\widetilde{f}’(a,b) = \begin{bmatrix}
\iota & 0 \\
D_1 f(a,b) & D_2 f(a,b)
\end{bmatrix}.
$$ Then \(\widetilde{f}’(a,b)\) is invertible, with $$
\widetilde{f}’(a,b)^{-1} = \begin{bmatrix}
\iota & 0 \\
-[D_2 f(a,b)]^{-1}\circ D_1 f(a,b) & [D_2 f(a,b)]^{-1}
\end{bmatrix}.\tag{*}
$$ By the inverse function theorem, there exist neighborhoods \(V\subseteq A\times B\) of \((a,b)\) and \(W\subseteq E\times G\) of \((a,0)\) such that \(\widetilde{f}|_V:V\to W\) is a \(C^p\) diffeomorphism. Let \(U=\{x\in E:(x,0)\in W\}\); it is clear that \(U\) is a neighborhood of \(a\). Define \(g:U\to B\) by \(g=\pi\circ(\widetilde{f}|_V)^{-1}\circ i\) where \(\pi:A\times B\to B\) is the canonical projection and \(i:A\to A\times B\) is given by \(i(x)=(x,0)\). To complete the proof, we check the three required properties. Firstly, $$
g(a)=\pi((\widetilde{f}|_V)^{-1}(a,0))=\pi(a,b)=b
$$ since \(\widetilde{f}(a,b)=(a,0)\). If \(x\in U\) then \((x,0)\in W\), so \((x,f(x,y))=\widetilde{f}(x,y)=(x,0)\) for a unique \(y\in B\) and $$
f(x,g(x))=f(x,\pi((\widetilde{f}|_V)^{-1}(x,0)))=f(x,\pi(x,y))=f(x,y)=0.
$$ Lastly, \(g’(b)\) is simply the bottom left entry of (*). \(\square\)

In the next and final post, we will look at some applications of Taylor’s theorem and the implicit function theorem to finding minima and maxima of maps from Banach spaces to \(\mathbb{R}\).

Navigation: 1. The derivative | 2. Higher derivatives | 3. Partial derivatives | 4. Inverse and implicit functions | 5. Maxima and minima

While we saw that differentiable maps may be naturally split into component functions when the codomain is a product of Banach spaces, the situation for the domain is more complicated. (This is partly due to the fact that as a topological space, there is no natural injection into a product of Banach spaces.) In this post, we will look at how the existence of partial derivatives relates to differentiability, how the symmetry of higher derivatives (covered in part 2) affects mixed partial derivatives, and finally a short proof of differentiation under the integral sign.

Let \(E_1,\dots,E_m\) be Banach spaces and let \(A_1\times\cdots\times A_m\subseteq E_1\times\cdots\times E_m\) be an open set where each \(A_j\) is open in \(E_j\). If \(f:A_1\times\cdots\times A_m\to F\) is any map and \(x=(x_1,\dots,x_m)\in A_1\times\cdots\times A_m\), we can consider the map $$t \mapsto f(x_1,\dots,t,\dots,x_m),$$ which can also be written as \(f\circ\iota\) where \(\iota:A_j\to A_1\times\cdots\times A_m\) is given by \(t\mapsto(x_1,\dots,t,\dots,x_m)\). If this map is differentiable at \(x_j\), we call its derivative the \(j\)th partial derivative of \(f\) at \(x\) and denote it by \(D_j f(x)\). Looking again at the definition of the derivative, we see that \(D_j f(x):E_j\to F\) is the unique continuous linear map such that $$
\lim_{h\to 0}\frac{f(x_1,\dots,x_j+h,\dots,x_m)-f(x_1,\dots,x_m)-D_j f(x)h}{|h|}=0.
$$ In practice, if we are working with functions defined on \(\mathbb{R}^m\) then we take \(E_j=\mathbb{R}\) for \(j=1,\dots,m\) so that we have a decomposition of \(\mathbb{R}^m\) into \(\mathbb{R}\times\cdots\times\mathbb{R}\). In this situation we often see the notation $$
\frac{\partial f}{\partial x_j}(x) = D_j f(x)(1),
$$ where we identify the linear map \(D_j f(x):\mathbb{R}\to F\) with the value \(D_j f(x)(1)\in F\).

It is not hard to see that all partial derivatives exist at \(x\) if \(f\) is differentiable at \(x\).

Theorem 25. Let \(A_1\times\cdots\times A_m\subseteq E_1\times\cdots\times E_m\) where each \(A_j\) is open in \(E_j\) and let \(f:A_1\times\cdots\times A_m\to F\). If \(f\) is differentiable at \(x=(x_1,\dots,x_m)\in A_1\times\cdots\times A_m\), then every \(D_j f(x)\) exists and we have \(D_j f(x)=Df(x)\circ\iota_j\) where \(\iota_j:E_j\to E_1\times\cdots\times E_m\) is the canonical injection, i.e. $$
Df(x)=\begin{bmatrix}D_1 f(x) & \cdots & D_m f(x)\end{bmatrix}.
$$

Proof. Apply the chain rule to \(f\circ\iota\) where \(\iota:A_j\to A_1\times\cdots\times A_m\) is given by \(t\mapsto(x_1,\dots,t,\dots,x_m)\). Alternatively, restrict \(h\) to elements of \(E_j\) in the definition of the derivative \(Df(x)\). \(\square\)

Definition 26. Let \(E_1,\dots,E_m\) and \(F_1,\dots,F_n\) be Banach spaces. Let \(A_1\times\cdots\times A_m\subseteq E_1\times\cdots\times E_m\) where each \(A_j\) is open in \(E_j\) and let \(f:A_1\times\cdots\times A_m\to F_1\times\cdots\times F_n\). The matrix $$
\begin{bmatrix}
D_1 f_1(x) & \cdots & D_m f_1(x) \\
\vdots & \ddots & \vdots \\
D_1 f_n(x) & \cdots & D_m f_n(x)
\end{bmatrix}
$$ is called the Jacobian matrix of \(f\) at \(x\in A\), where \(f_i=\pi_i\circ f\) and \(\pi_i:F_1\times\cdots\times F_n\to F_i\) is the canonical projection.

Theorem 27. If \(f:A_1\times\cdots\times A_m\to F_1\times\cdots\times F_n\) is differentiable at \(x\in A\), then the Jacobian matrix of \(f\) at \(x\) exists and represents \(Df(x)\).

Proof. Apply Theorem 8 followed by Theorem 25. \(\square\)

As in \(\mathbb{R}^n\), it may be the case that every partial derivative \(D_j f_i(x)\) exists but \(f\) is not differentiable at \(x\). The differentiability of \(f\) implies the existence of the Jacobian matrix, but the converse is not true. Thus we do not have a true analog of Theorem 8 for partial derivatives. We do however have an analog of Theorem 18.

Theorem 28. Let \(A_1\times\cdots\times A_m\subseteq E_1\times\cdots\times E_m\) where each \(A_j\) is open in \(E_j\) and let \(f:A_1\times\cdots\times A_m\to F\). Then \(f\) is of class \(C^p\) (with \(p\ge 1\)) if and only if every partial derivative $$
D_j f:A_1\times\cdots\times A_m\to L(E_j,F)
$$ is of class \(C^{p-1}\). In that case we have \(D_j f(x)=Df(x)\circ\iota_j\) where \(\iota_j:E_j\to E_1\times\cdots\times E_m\) is the canonical injection, i.e. $$
Df(x)=\begin{bmatrix}D_1 f(x) & \cdots & D_m f(x)\end{bmatrix}.
$$

Proof. It is clear from the proof of Theorem 25 that every partial derivative is of class \(C^{p-1}\) if \(f\) is of class \(C^p\). For the converse, we only need to prove that \(Df\) exists on \(A_1\times\cdots\times A_m\) since $$
Df(x)=\sum_{j=1}^m D_j f(x)\circ\pi_j
$$ implies that \(Df\) is of class \(C^{p-1}\) if every \(D_j f\) is of class \(C^{p-1}\), where \(\pi_j:E_1\times\cdots\times E_m\to E_j\) is the canonical projection. Let \(x\in A_1\times\cdots\times A_m\) and let \(\varepsilon > 0\). Since every \(D_j f\) is continuous, there exists a \(\delta > 0\) such that $$
|D_j f(y)-D_j f(x)| < \frac{\varepsilon}{m}
$$ for all \(j=1,\dots,m\) and \(y\in B_\delta(x)\subseteq A_1\times\cdots\times A_m\) where \(B_\delta(x)\) is the open ball of radius \(\delta\) around \(x\). Let \(h=(h_1,\dots,h_m)\in E_1\times\cdots\times E_m\) with \(|h|<\delta\). For \(j=0,\dots,m\), let \(p_j=h_1+\cdots+h_j\) so that \(p_0=0\), \(p_m=h\), and $$
f(x+h)-f(x)=\sum_{j=1}^m [f(x+p_j)-f(x+p_{j-1})].
$$ For each \(j=1,\dots,m\) the line segment from \(x+p_{j-1}\) to \(x+p_j=x+p_{j-1}+h_j\) is contained in \(B_\delta(x)\), so we have $$
f(x+p_j)-f(x+p_{j-1}) = \int_0^1 D_j f(x+p_{j-1}+th_j)h_j\,dt
$$ by the mean value theorem. Then \begin{align}
& \left\vert f(x+h)-f(x)-\sum_{j=1}^m D_j f(x)h_j \right\vert \\
&\le \left\vert \sum_{j=1}^m \left[ \int_0^1 D_j f(x+p_{j-1}+th_j)h_j\,dt - \int_0^1 D_j f(x)h_j\,dt \right] \right\vert \\
&\le \sum_{j=1}^m |h_j| \int_0^1 |D_j f(x+p_{j-1}+th_j)-D_j f(x)|\,dt \\
&\le \sum_{j=1}^m |h_j| \frac{\varepsilon}{m} \\
&\le |h|\varepsilon
\end{align} for all \(|h|<\delta\), which shows that $$
Df(x)=\sum_{j=1}^m D_j f(x)\circ\pi_j.
$$ \(\square\)

As in the case of the ordinary derivative \(Df\), we may take higher derivatives of partial derivatives: $$
D_{j_1}\cdots D_{j_r} f:A_1\times\cdots\times A_m\to L(E_{j_1},\dots,L(E_{j_r},F)\dots).
$$ These are sometimes known as mixed partial derivatives. Theorem 21 has an important interpretation in terms of the mixed partial derivatives of \(f\).

Theorem 29 (Equality of mixed partial derivatives). Let \(A_1\times\cdots\times A_m\subseteq E_1\times\cdots\times E_m\) where each \(A_j\) is open in \(E_j\) and let \(f:A_1\times\cdots\times A_m\to F\) be of class \(C^2\). Then $$
D_j D_k f(x)(u)(v) = D_k D_j f(x)(v)(u)
$$ for all \(1 \le j,k \le m\), \(x\in A_1\times\cdots\times A_m\), \(u\in A_j\) and \(v\in A_k\).

Proof. For \(j=1,\dots,m\), let \(\iota_j:E_j\to E_1\times\cdots\times E_m\) be the canonical injection. We have that \(D_k f(x)=Df(x)\circ \iota_k\), so \(D_k f=c\circ Df\) where \begin{align}
c:L(E_1\times\cdots\times E_m,F) &\to L(E_k,F) \\
\lambda &\mapsto \lambda \circ \iota_k.
\end{align} Similarly, \(D_j D_k f = d\circ D(D_k f)\) where \begin{align}
d:L(E_1\times\cdots\times E_m,L(E_k,F)) &\to L(E_j,L(E_k,F)) \\
\lambda &\mapsto \lambda \circ \iota_j.
\end{align} Note that \(c\) and \(d\) are both linear maps. Therefore \begin{align}
D_j D_k f(x) &= [d\circ D(D_k f)](x) \\
&= d(D(c\circ Df)(x)) \\
&= d(Dc(Df(x))\circ D^2 f(x)) \\
&= d(c\circ D^2 f(x)) \\
&= c\circ D^2 f(x)\circ\iota_j
\end{align} and \begin{align}
D_j D_k f(x)(u)(v) &= (c\circ D^2 f(x)\circ\iota_j)(u)(v) \\
&= c(D^2 f(x)(\iota_j(u)))(v) \\
&= D^2 f(x)(\iota_j(u))(\iota_k(v)).
\end{align} A similar calculation shows that \(D_k D_j f(x)(v)(u) = D^2 f(x)(\iota_k(v))(\iota_j(u))\). But $$
D^2 f(x)(\iota_k(v))(\iota_j(u)) = D^2 f(x)(\iota_j(u))(\iota_k(v))
$$ by Theorem 21, so the result follows. \(\square\)

In the special case \(E_j=\mathbb{R}\) for \(j=1,\dots,m\) and \(f:\mathbb{R}^m\to F\), we have $$
\frac{\partial f}{\partial x_j \partial x_k}(x) = D_j D_k f(x)(1)(1) = D_k D_j f(x)(1)(1) = \frac{\partial f}{\partial x_k \partial x_j}(x),
$$ which is sometimes known as the symmetry of second derivatives, or Clairaut’s theorem.

Often, \(p\) times continuous differentiability is defined in terms of the mixed partial derivatives of \(f\). The following theorem shows that this definition is equivalent to ours.

Theorem 30. Let \(A_1\times\cdots\times A_m\subseteq E_1\times\cdots\times E_m\) where each \(A_j\) is open in \(E_j\) and let \(f:A_1\times\cdots\times A_m\to F\). Then \(f\) is of class \(C^p\) (with \(p\ge 1\)) if and only if the partial derivative $$
D_{\tau(1)}\cdots D_{\tau(k)}f
$$ exists on \(A_1\times\cdots\times A_m\) and is continuous, for every \(k=1,\dots,p\) and every map \(\tau\) from \(\{1,\dots,k\}\) to \(\{1,\dots,m\}\). Furthermore, $$
D_{\tau(\sigma(1))}\cdots D_{\tau(\sigma(k))}f(x)(v_{\sigma(1)},\dots,v_{\sigma(k)}) = D_{\tau(1)}\cdots D_{\tau(k)}f(x)(v_1,\dots,v_k)
$$ for all \(x\in A_1\times\cdots\times A_m\) and any permutation \(\sigma\) of \(\{1,\dots,k\}\).

Proof. This follows directly from Theorem 28 and Theorem 29. \(\square\)

Differentiation under the integral sign

Theorem 31 (Differentiation under the integral sign). Let \(A\subseteq E\) be an open set and let \([a,b]\) be a closed interval with \(a < b\). Let \(f:[a,b]\times A\to F\) be a continuous map such that \(D_2 f\) exists on \([a,b]\times A\) and is continuous. Let \(g:A\to F\) be given by $$
g(x)=\int_a^b f(t,x)\,dt.
$$ Then \(g\) is differentiable on \(A\) and $$
Dg(x)=\int_a^b D_2 f(t,x)\,dt.
$$

Proof. Let \(x\in A\). Let $$
\lambda = \int_a^b D_2 f(t,x)\,dt.
$$ For sufficiently small \(h\) we have \begin{align}
g(x+h)-g(x)-\lambda h &= \int_a^b [f(t,x+h)-f(t,x)-D_2 f(t,x)h]\,dt \\
&= \int_a^b \left[ \int_0^1 D_2 f(t,x+sh)h\,ds-D_2 f(t,x)h \right]\,dt \\
&= \int_a^b \int_0^1 [D_2 f(t,x+sh)-D_2 f(t,x)]h\,ds\,dt
\end{align} so that \begin{align}
\frac{|g(x+h)-f(x)-\lambda h|}{|h|} &\le \int_a^b \int_0^1 |D_2 f(t,x+sh)-D_2 f(t,x)|\,ds\,dt \\
&\le \sup_{s,t} |D_2 f(t,x+sh)-D_2 f(t,x)|
\end{align} where the \(\sup\) is taken over all \(0\le s\le 1\) and \(a\le t\le b\).

Let \(\varepsilon > 0\). For each \(t\in[a,b]\) there is a neighborhood \(B_t\times U_t\) of \((t,x)\) such that \(|D_2 f(u,y)-D_2 f(t,x)|<\varepsilon\) whenever \((u,y)\in B_t\times U_t\), and such that \(B_t\) and \(U_t\) are open balls around \(t\) and \(x\) respectively. Since \([a,b]\) is compact, there are finitely many balls \(B_{t_1},\dots,B_{t_n}\) that cover \([a,b]\). Then for sufficiently small \(h\) such that \(x+h\in\bigcap_{k=1}^n U_{t_k}\) and all \(0\le s\le 1\) and \(a\le t\le b\) we have \(t\in B_{t_k}\) for some \(k\), so \begin{align}
|D_2 f(t,x+sh)-D_2 f(t,x)| &\le |D_2 f(t,x+sh)-D_2 f(t_k,x)| \\
&\qquad + |D_2 f(t_k,x)-D_2 f(t,x)| \\
&< 2\varepsilon.
\end{align} \(\square\)

The next post will prove exactly three things: the Banach fixed-point theorem, the inverse function theorem, and the implicit function theorem.

Navigation: 1. The derivative | 2. Higher derivatives | 3. Partial derivatives | 4. Inverse and implicit functions | 5. Maxima and minima

Last time, we covered the definition of the derivative and its basic properties, which all turn out to be quite similar to their single variable counterparts. Now we are going to explore higher derivatives. In traditional multivariable calculus, true higher derivatives do not exist (except in a specific situation which will be discussed in part 5). Of course, we have so-called “mixed/higher partial derivatives”, which are coordinate-dependent and notationally tricky to work with. As a consequence, the usual statement of Taylor’s theorem in \(\mathbb{R}^n\) ends up being ugly and hard to remember. In reality, Taylor’s theorem for Banach spaces looks almost exactly the same as the single variable Taylor’s theorem!

Recall that for any Banach space \(F\), the space of continuous linear maps \(L(E,F)\) is also a Banach space. If \(A\subseteq E\) is an open set and \(f:A\to F\) is differentiable, then \(Df=f’:A\to L(E,F)\) is a map between Banach spaces. Therefore we may consider the second derivative $$D^2 f=f^{\prime\prime}:A\to L(E,L(E,F))$$ obtained by differentiating \(f’\). Continuing the process, we have higher order derivatives $$D^p f = f^{(p)} : A \to L^p(E,F),$$ where \(L^p(E,F)=L(E,L(E,\dots,L(E,F)\dots))\). It is clear that \(D^p(f+g)=D^p f + D^p g\) and \(D^p(cf)=cD^p f\) for all scalars \(c\). We say that \(f\) is of class \(C^p\) or is \(p\) times continuously differentiable if \(D^p f(x)\) exists for all \(x\in A\) and \(D^p f\) is continuous on \(A\). Note that if \(f\) is of class \(C^p\), then \(D^k f\) is automatically continuous for all \(0 \le k < p\) as well. We identify \(L^p(E,F)\) with the space of multilinear maps \(L(E,\dots,E;F)\), and write \(D^p f(x_1,\dots,x_p)\) for \(D^p f(x_1)\cdots(x_p)\).

Our definition of \(p\) times continuous differentiability is not the usual one that is stated in terms of mixed partial derivatives, but part 3 will make it clear that these definitions are equivalent.

We have the following extension of Theorem 8, which follows easily by induction.

Theorem 18. Let \(A\subseteq E\) be an open set, let \(F_1,\dots,F_n\) be Banach spaces, let \(f:A\to F_1\times\cdots\times F_n\) and let \(f_i=\pi_i\circ f\) be the component functions of \(f\), where \(\pi_i:F_1\times\cdots\times F_n\to F_i\) is the canonical projection. Then \(f\) is of class \(C^p\) if and only if every \(f_i\) is of class \(C^p\). In that case we have \(D^p f_i(x)=\pi_i\circ D^p f(x)\), i.e. $$
D^p f(x) = \begin{bmatrix}D^p f_1(x) \\ \vdots \\ D^p f_n(x)\end{bmatrix}.
$$

Theorem 19. Let \(A\subseteq E\) and \(B\subseteq F\) be open sets. Let \(f:A\to F\) and \(g:B\to G\) be class \(C^p\) maps with \(f(A)\subseteq B\). Then \(g\circ f\) is of class \(C^p\).

Proof. We use induction on \(p\), with the chain rule proving the case \(p=1\) (the case \(p=0\) also holds because a composition of continuous maps is also continuous). Assume that the result holds for \(p-1\) and suppose \(f\) and \(g\) are of class \(C^p\). By the chain rule we have $$D(g\circ f)(x)=Dg(f(x))\circ Df(x).$$ As a function of \(x\), the right hand side is a composition of \(C^{p-1}\) maps, so the induction hypothesis shows that \(D(g\circ f)\) is of class \(C^{p-1}\) and therefore \(g\circ f\) is of class \(C^p\). \(\square\)

Symmetry

An important fact is that \(D^p f(x)\) is always symmetric (as a multilinear map) if \(f\) is of class \(C^p\). In part 3, we will show that the well-known equality of mixed partial derivatives is a special case of this. To prove this result, we start with the case \(p=2\).

Lemma 20. Let \(\varphi:E\times E\to F\) be a bilinear map. If there is a map \(\psi\) into \(F\) defined for sufficiently small \((v,w)\in E\times E\) such that $$
\lim_{(v,w)\to (0,0)} \psi(v,w) = 0
$$ and $$
|\varphi(v,w)| \le |\psi(v,w)||v||w|,
$$ then \(\varphi=0\).

Proof. Let \(v,w\in E\). For sufficiently small \(s>0\) we have $$
|\varphi(sv,sw)| \le |\psi(sv,sw)||sv||sw|,
$$ so $$
s^2|\varphi(v,w)| \le s^2|\psi(sv,sw)||v||w|.
$$ Dividing by \(s^2\) and taking \(s\to 0\) proves the result. \(\square\)

Theorem 21. Let \(A\subseteq E\) be an open set and let \(f:A\to F\) be a class \(C^2\) map. Then for every \(x\in A\), the bilinear map \(D^2 f(x)\) is symmetric. That is, $$
D^2 f(x)(v,w)=D^2 f(x)(w,v)
$$ for all \(v,w\in E\).

Proof. Let \(x\in A\) and choose \(r > 0\) so that the open ball of radius \(r\) around \(x\) is contained in \(A\). Let \(v,w\in E\) with \(|v|,|w| < r/2\). Define \(g(x)=f(x+v)-f(x)\). The mean value theorem then gives \begin{align}
g(x+w)-g(x) &= \int_0^1 g’(x+tw)w\,dt \\
&= \int_0^1 [Df(x+v+tw)-Df(x+tw)]w\,dt \\
&= \int_0^1 \left( \int_0^1 D^2 f(x+sv+tw)v\,ds \right) w\,dt \\
&= \int_0^1\int_0^1 D^2 f(x)(v,w)\,ds\,dt + \int_0^1\int_0^1 \psi(sv,tw)(v,w)\,ds\,dt \\
&= D^2 f(x)(v,w)+\varphi(v,w)
\end{align} where \(\psi(\alpha,\beta)=D^2 f(x+\alpha+\beta)-D^2 f(x)\) and $$
\varphi=\int_0^1 \int_0^1 \psi(sv,tw)\,ds\,dt.
$$ If we repeat the above process starting with \(g_1\) in place of \(g\), where \(g_1(x)=f(x+w)-f(x)\), we obtain that $$
g_1(x+v)-g_1(x)=D^2 f(x)(w,v)+\varphi(w,v).
$$ Since \(g(x+w)-g(x)=g_1(x+v)-g_1(x)\), we have $$
D^2 f(x)(w,v)-D^2 f(x)(v,w)=\varphi(v,w)-\varphi(w,v),
$$ and from the definitions of \(\varphi\) and \(\psi\) we see that $$
|D^2 f(x)(w,v)-D^2 f(x)(v,w)| \le 2 \sup_{0 \le s,t \le 1} |\psi(sv,tw)||v||w|.
$$ Since \(D^2 f\) is continuous, we can apply Lemma 20 to the bilinear map $$
(v,w) \mapsto D^2 f(x)(w,v)-D^2 f(x)(v,w)
$$ to obtain the result. \(\square\)

Theorem 22. Let \(A\subseteq E\) be an open set and let \(f:A\to F\) be a class \(C^p\) map. Then for every \(x\in A\), the multilinear map \(D^p f(x)\) is symmetric.

Proof. We use induction on \(p\), with Theorem 21 proving the case \(p=2\). Suppose the result holds for \(2,\dots,p-1\). If \(v_1,\dots,v_p\in E\), then \begin{align}
D^p f(x)(v_1,\dots,v_p) &= D^2 D^{p-2} f(x)(v_1,v_2)(v_3,\dots,v_p) \\
&= D^2 D^{p-2} f(x)(v_2,v_1)(v_3,\dots,v_p) \\
&= D^2 D^{p-2} f(x)(v_2,v_1,v_3,\dots,v_p)\tag{*}
\end{align} by applying Theorem 21 to the \(C^2\) map \(D^{p-2}f\). Also, the induction hypothesis shows that $$
D^{p-1}f(x)(v_{\sigma(2)},\dots,v_{\sigma(p)})=D^{p-1}f(x)(v_2,\dots,v_p)
$$ for any permutation \(\sigma\) of \(\{2,\dots,p\}\). If \(\varphi_\sigma:L^{p-1}(E,F)\to F\) is the linear map given by \(\lambda\mapsto\lambda(v_{\sigma(2)},\dots,v_{\sigma(p)})\) then \begin{align}
D^p f(x)(v_1,v_{\sigma(2)},\dots,v_{\sigma(p)}) &= \varphi_\sigma(D^p f(x)(v_1)) \\
&= D(\varphi_\sigma\circ D^{p-1}f)(x)(v_1) \\
&= D(\varphi_e\circ D^{p-1}f)(x)(v_1) \\
&= \varphi_e(D^p f(x)(v_1)) \\
&= D^p f(x)(v_1,\dots,v_p)\tag{**}
\end{align} where \(e\) is the identity permutation. Since any permutation of \(\{1,\dots,p\}\) can be expressed as a composition of the permutations considered in (*) and (**), \(D^p f(x)\) is symmetric. \(\square\)

Taylor’s theorem

Taylor’s theorem is an important tool for approximating a function based on its derivatives. It will be important in part 5, where we look at necessary and sufficient conditions for a point to be a local minimum or local maximum.

Theorem 23 (Taylor’s theorem). Let \(A\subseteq E\) be an open set and let \(f:A\to F\) be a class \(C^p\) map. Let \(x\in A\) and let \(v\in E\). Assume that the line segment \(x+tv\) with \(0\le t\le1\) is contained in \(A\). Write \(v^{(k)}\) for the \(k\)-tuple \((v,\dots,v)\). Then $$
f(x+v)=\sum_{k=0}^{p-1}\frac{D^k f(x)v^{(k)}}{k!} + R_p,
$$ where $$
R_p = \int_0^1 \frac{(1-t)^{p-1}}{(p-1)!} D^p f(x+tv)v^{(p)}\,dt.
$$

Proof. We use induction on \(p\), with the mean value theorem proving the case \(p=1\). Assume that the result holds for \(p-1\). Let $$
g(t)=\frac{(1-t)^{p-1}}{(p-1)!} \quad\mathrm{and}\quad h(t)=D^{p-1}f(x+tv)v^{(p-1)}
$$ so that $$
g’(t)=\frac{-(1-t)^{p-2}}{(p-2)!} \quad\mathrm{and}\quad h’(t)=D^p f(x+tv)v^{(p)}.
$$ (Note that for convenience, we are again identifying \(h’(t)\) with an element of \(F\).) Applying integration by parts with the vector space product \(\mathbb{R}\times F\to F\) gives \begin{align}
& \int_0^1 \frac{-(1-t)^{p-2}}{(p-2)!} D^{p-1}f(x+tv)v^{(p-1)}\,dt + \int_0^1 \frac{(1-t)^{p-1}}{(p-1)!} D^p f(x+tv)v^{(p)}\,dt \\
&= -\frac{1}{(p-1)!} D^{p-1}f(x)v^{(p-1)},
\end{align} and the result follows. \(\square\)

Corollary 24 (Taylor’s theorem with estimate). In Theorem 23, we also have $$
f(x+v)=\sum_{k=0}^p\frac{D^k f(x)v^{(k)}}{k!} + \theta(v)
$$ where $$
|\theta(v)| \le \sup_{0\le t\le 1} \frac{|D^p f(x+tv)-D^p f(x)|}{p!} |v|^p
$$ and $$
\lim_{v\to 0}\frac{\theta(v)}{|v|^p} = 0.
$$

Proof. Let \(\psi(\alpha) = D^p f(x+\alpha)-D^p f(x)\). We can write \(R_p\) as $$
\int_0^1 \frac{(1-t)^{p-1}}{(p-1)!} D^p f(x)v^{(p)}\,dt + \int_0^1 \frac{(1-t)^{p-1}}{(p-1)!} D^p \psi(tv)v^{(p)}\,dt.
$$ The first integral gives the \(p\)th term, and the second integral is bounded by $$
\sup_{0\le t\le 1} |\psi(tv)||v|^p \int_0^1 \frac{(1-t)^{(p-1)}}{(p-1)!}\,dt = \frac{1}{p!} \sup_{0\le t\le 1} |\psi(tv)||v|^p.
$$ The result follows from the continuity of \(D^p f\) at \(x\). \(\square\)

Next time, we will examine partial derivatives and how they relate to higher derivatives.

Navigation: 1. The derivative | 2. Higher derivatives | 3. Partial derivatives | 4. Inverse and implicit functions | 5. Maxima and minima

In multivariable calculus, one often encounters nonsensical equations such as the following:

(Chain rule.) If \(F=F(x,y)\) and \(x=x(t)\), \(y=y(t)\) then $$\frac{dF}{dt}=\frac{dF}{\partial x}\frac{dx}{dt} + \frac{dF}{\partial y}\frac{dy}{dt}.$$

(What is “\(F\)” on the left, and what is “\(F\)” on the right?) A partial derivative of a function \(f:\mathbb{R}^2\to\mathbb{R}\) is denoted by an assortment of vague expressions like $$
\frac{\partial f}{\partial x} \quad\mathrm{or}\quad \frac{\partial f(x,y)}{\partial x} \quad\mathrm{or}\quad \frac{\partial f}{\partial x}(x,y) \quad\mathrm{or}\quad f_x,
$$ which makes it hard to distinguish between the partial derivative as a function and the partial derivative at a point \((u,v)\):
$$
\frac{\partial f}{\partial x}(u,v) \quad\mathrm{or}\quad \left.\frac{\partial f(x,y)}{\partial x}\right|_{(x,y)=(u,v)} \quad\mathrm{or}\quad f_x(u,v) \quad\mathrm{?}
$$
Furthermore, it isn’t even clear what the difference between $$
\frac{df}{dx} \quad\mathrm{and}\quad \frac{\partial f}{\partial x}
$$ is!

Partial derivatives are usually introduced first because they provide an extension of differentiation in one variable that is convenient for computation. The so-called Jacobian matrix is relegated to a section on the change of variables theorem, which, as usual, is presented as a magic formula. Overall, one gets the impression that multivariable differentiation is far more complicated than single variable differentiation, and that the usual formulas do not apply.

This is all wrong. With the right definitions, the same old theorems (and even proofs) from single variable calculus still apply, with very few changes. I recently learned about this material from Lang’s Real and Functional Analysis, which has an elegantly written chapter on differentiation. I will be writing 5 posts covering the basics of differentiation, starting with this one.

Basic properties

Note: if you do not know what a Banach space or a Banach algebra is, replace “Banach space” with “something like \(\mathbb{R}^n\) or \(\operatorname{Mat}_{n,m}(\mathbb{R})\) but might be infinite-dimensional”, and replace “Banach algebra” with “Banach space where you can multiply vectors”.

Here, \(E\), \(F\) and \(G\) will denote real Banach spaces. We write \(L(E,F)\) for the space of continuous linear maps from \(E\) to \(F\). All Banach algebras are assumed to be unital. As usual, if \(f:E\to F\) is a linear map then we often write \(fx\) instead of \(f(x)\). Although we treat the general case here, I recommend visualizing \(E,F\) as \(\mathbb{R^n}\) for concreteness.

Definition 1. Let \(A \subseteq E\) be an open set and let \(f:A \to F\). A continuous linear map \(\lambda : E \to F\) is said to be the Fréchet derivative or just the derivative of \(f\) at a point \(x \in A\) if $$
\lim_{h \to 0} \frac{f(x+h)-f(x)- \lambda h}{|h|} = 0,
$$ and we write \(f’(x)=\lambda\) or \(Df(x)=\lambda\). If \(f\) has a derivative at a point \(x\), we say that \(f\) is differentiable at \(x\). If \(f\) is differentiable at all \(x \in A\), then we say that \(f\) is differentiable on \(A\) or simply differentiable.

Obviously there is the question of uniqueness.

Theorem 2. Let \(A \subseteq E\) be an open set. The derivative of a function \(f:A\to F\) at a point \(x \in A\), if it exists, is unique.

Proof. Suppose \(\lambda_1\) and \(\lambda_2\) are both derivatives of \(f\) at \(x\). Subtracting gives $$
\lim_{h\to 0} \frac{(\lambda_2-\lambda_1)h}{|h|} = 0.
$$ For a fixed nonzero \(u \in E\) we have $$
\lim_{t\to 0^+} \frac{(\lambda_2-\lambda_1)tu}{|tu|} = 0,
$$ and since the left hand side is independent of \(t\) we have \((\lambda_2-\lambda_1)u=0\) for all \(u \in E\). Therefore \(\lambda_1=\lambda_2\). \(\square\)

Theorem 3. Let \(A \subseteq E\) be an open set. If \(f:A\to F\) is differentiable at \(x \in A\) then \(f\) is continuous at \(x\).

Proof. For small \(h\) we can write $$
f(x+h)-f(x)=|h|\left(\frac{f(x+h)-f(x)-\lambda h}{|h|}\right)+\lambda h.
$$ The right hand side tends to \(0\) as \(h \to 0\), so \(f\) is continuous at \(x\). \(\square\)

Suppose \(f : A \to F\) is differentiable on \(A\). We have a map \(f’:A\to L(E,F)\) that sends each point in \(A\) to the derivative of \(f\) at that point. If \(f’\) is continuous then we say that \(f\) is continuously differentiable or of class \(C^1\). We will have more to say about this in a later post.

Theorem 4 (Chain rule). Let \(A\subseteq E\) and \(B\subseteq F\) be open sets. Let \(f:A\to F\) and \(g:B\to G\) with \(f(A)\subseteq B\). If \(f\) is differentiable at \(x\) and \(g\) is differentiable at \(f(x)\), then \(g\circ f\) is differentiable at \(x\) and $$(g\circ f)’(x)=g’(f(x))\circ f’(x).$$

Proof. To save space, let \(y=f(x)\) and define \begin{align}
\phi(s) &= f(x+s)-f(x)-f’(x)s, \\
\psi(t) &= g(y+t)-g(y)-g’(y)t, \\
\rho(h) &= g(f(x+h))-g(y)-g’(y)f’(x)h
\end{align} so that $$
\lim_{s\to 0}\frac{\phi(s)}{|s|} = \lim_{t\to 0}\frac{\psi(t)}{|t|}=0\tag{*}
$$ since \(f\) is differentiable at \(x\) and \(g\) is differentiable at \(y\). We want to show that $$
\lim_{h\to 0}\frac{\rho(h)}{|h|}=0.
$$ For all sufficiently small \(h\), \begin{align}
g(f(x+h))-g(y)&=g(y+f’(x)h+\phi(h))-g(y) \\
&=g’(y)(f’(x)h+\phi(h))+\psi(f’(x)h+\phi(h))
\end{align} and $$
\rho(h)=g’(y)\phi(h)+\psi(f’(x)h+\phi(h)).
$$ Since \(g’(y)\) is continuous, $$
\lim_{h\to 0}\frac{g’(y)\phi(h)}{|h|} = g’(y)\left[\lim_{h\to 0}\frac{\phi(h)}{|h|}\right]=0,
$$ and it remains to show that $$
\lim_{h\to 0}\frac{\psi(f’(x)h+\phi(h))}{|h|}=0.
$$ We can assume that \(|f’(x)| \ne 0\). Let \(\varepsilon>0\). By (*) there exist \(\delta,\gamma,\beta>0\) such that \(|\phi(s)|/|s|<|f'(x)|\) whenever \(|s|<\delta\), \(|\psi(t)|<\varepsilon(2|f'(x)|)^{-1}|t|\) whenever \(|t|<\gamma\), and \(|f'(x)h+\phi(h)|<\gamma\) whenever \(|h|<\beta\). Then for all \(|h|<\beta\), \begin{align}
\frac{|\psi(f'(x)h+\phi(h))|}{|h|} &\le \frac{\varepsilon}{2|f'(x)|} \left(\frac{|f'(x)h|}{|h|}+\frac{|\psi(h)|}{|h|}\right) \\
&< \varepsilon.
\end{align} \(\square\)

Theorem 5. Let \(F_1,F_2\) be Banach spaces, let \(A\subseteq E\) be an open set and let \(f:A\to F_1\) and \(g:A\to F_2\) be differentiable at \(x\in A\).

1. If \(f\) is constant then \(f’(x)=0\).
2. If \(f(x)=\lambda x\) for some continuous linear map \(\lambda\), then \(f’(x)=\lambda\).
3. If \(F_1=F_2\) then \((f+g)’(x)=f’(x)+g’(x)\).
4. \((cf)’(x)=cf’(x)\) for all scalars \(c\).
5. (Product rule). Suppose there is a continuous bilinear map \(\cdot:F_1\times F_2\to G\). Then $$
   (fg)’(x)=f’(x)g(x)+f(x)g’(x),
   $$ where \(f’(x)g(x)\) is the linear map that takes \(u\) to \(f’(x)u\cdot g(x)\).

Proof. The first 4 parts are obvious, so we only prove (5). We have \begin{align}
0 &= \lim_{h\to 0}\frac{[f(x+h)-f(x)-f'(x)h]g(x+h)+f(x)[g(x+h)-g(x)-g'(x)h]}{|h|} \\
&= \lim_{h\to 0}\frac{(fg)(x+h)-(fg)(x)-[f'(x)g(x+h)+f(x)g'(x)]h}{|h|}.\tag{*}
\end{align} Now $$
\frac{|f’(x)h[g(x+h)-g(x)]|}{|h|} \le |f’(x)||g(x+h)-g(x)| \to 0
$$ as \(h\to 0\) since \(\cdot\) is continuous and \(g\) is continuous at \(x\), so $$
\lim_{h\to 0}\frac{f’(x)h[g(x+h)-g(x)]}{|h|} = 0.
$$ Adding this to (*) gives $$
\lim_{h\to 0}\frac{(fg)(x+h)-(fg)(x)-[f'(x)g(x)+f(x)g'(x)]h}{|h|}=0.
$$ \(\square\)

Theorem 6. Let \(E\) be a Banach algebra and let \(U\) be the open set of its invertible elements. Then the map \(x\mapsto x^{-1}\) is differentiable on \(U\), and its derivative at a point \(x\) is given by $$u \mapsto -x^{-1}ux^{-1}.$$

Proof. We have \begin{align}
(x+h)^{-1}-x^{-1}+x^{-1}hx^{-1} &= (x(e+x^{-1}h))^{-1}-x^{-1}+x^{-1}hx^{-1} \\
&= (e+x^{-1}h)^{-1}x^{-1}-x^{-1}+x^{-1}hx^{-1} \\
&= [(e+x^{-1}h)^{-1}-(e-x^{-1})]x^{-1}.\tag{*}
\end{align} For sufficiently small \(h\) we have \(|e-(e+x^{-1}h)|=|x^{-1}h|<1/2\), so \begin{align}
|(e+x^{-1}h)^{-1}-(e-x^{-1}h)| &= \left\vert \sum_{k=0}^\infty (-x^{-1}h)^k - (e-x^{-1}h) \right\vert \\
&= \left\vert \sum_{k=2}^\infty (-x^{-1}h)^k \right\vert \\
&\le \frac{|x^{-1}h|^2}{1-|x^{-1}h|} \\
&= \frac{|x^{-1}h|}{|x^{-1}h|^{-1}-1} \\
&< |x^{-1}||h|.
\end{align} Combining this with (*) shows that $$
\frac{(x+h)^{-1}-x^{-1}+x^{-1}hx^{-1}}{|h|} \to 0
$$ as \(h\to 0\). \(\square\)

Corollary 7 (Quotient rule). Let \(F_1\) be a Banach space, let \(F_2\) be a Banach algebra, and let \(U\) be the open set of the invertible elements in \(F_2\). Let \(A\subseteq E\) be an open set and let \(f:A\to F_1\) and \(g:A\to U\) be differentiable at \(x\in A\). Suppose there is a continuous bilinear map \(\cdot:F_1\times F_2\to G\). Write \(fg^{-1}\) for the map \((fg^{-1})(x)=f(x)g(x)^{-1}\). Then \((fg^{-1})’(x)\) is given by $$
u \mapsto [f'(x)u]g(x)^{-1}-f(x)g(x)^{-1}[g'(x)u]g(x)^{-1}.
$$ In particular, if \(F_2\) is commutative then $$
(f/g)’(x)=[f'(x)g(x)-f(x)g'(x)]g(x)^{-2}.
$$

Linear maps and direct sums

Before continuing with the properties of the derivative, we describe a generalization of the concept of block matrices. Suppose \(E_1,\dots,E_m\) and \(F_1,\dots,F_n\) are vector spaces, and \(\lambda:E_1\times\cdots\times E_m\to F_1\times\cdots\times F_n\) is a linear map. We note that in the case of finitely many vector spaces, the notions of direct sum (coproduct) and direct product (product) coincide. Therefore, we have unique linear maps \(\lambda_{i,j}:E_j\to F_i\) such that \(\lambda_{i,j}=\pi_i\circ\lambda\circ\iota_j\), where \(\pi_i:F_1\times\cdots\times F_n\to F_i\) is the canonical projection and \(\iota_j:E_j\to E_1\times\cdots\times E_m\) is the canonical injection. In this case we write $$
\lambda=\begin{bmatrix}
\lambda_{1,1} & \cdots & \lambda_{1,m} \\
\vdots & \ddots & \vdots \\
\lambda_{n,1} & \cdots & \lambda_{n,m}
\end{bmatrix}$$ and say that the matrix represents \(\lambda\), and sometimes that the maps \(\lambda_{i,j}\) are the components of \(\lambda\). If \(\tau:F_1\times\cdots\times F_n\to G_1\times\cdots\times G_p\) is another linear map, then it is easy to verify that \(\tau\circ\lambda\) is represented by the product of the matrix representing \(\tau\) with the matrix representing \(\lambda\), with the usual matrix multiplication formula. On the other hand, if we start with linear maps \(\lambda_{i,j}\), then there is a unique linear map \(\lambda\) that has the components \(\lambda_{i,j}\). The situation is entirely analogous to block matrices in \(\mathbb{R}\) or \(\mathbb{C}\), so we do not go any further.

Theorem 8. Let \(A\subseteq E\) be an open set, let \(F_1,\dots,F_n\) be Banach spaces, let \(f:A\to F_1\times\cdots\times F_n\) and let \(f_i=\pi_i\circ f\) be the component functions of \(f\), where \(\pi_i:F_1\times\cdots\times F_n\to F_i\) is the canonical projection. Then \(f\) is differentiable at a point \(x\in A\) if and only if every \(f_i\) is differentiable at \(x\). In that case we have \(f’_i(x)=\pi_i\circ f’(x)\), i.e. $$
f’(x) = \begin{bmatrix}f’_1(x) \\ \vdots \\ f’_n(x)\end{bmatrix}.
$$

Proof. If \(\lambda\) is a linear map then the \(i\)th entry (with respect to the direct sum decomposition) of $$
T(h)=\frac{f(x+h)-f(x)-\lambda h}{|h|}
$$ is simply $$
T_i(h)=\frac{f_i(x+h)-f_i(x)-\pi_i\lambda h}{|h|}.
$$ Therefore \(T(h)\) approaches \(0\) as \(h\to 0\) if and only if every \(T_i(h)\) approaches \(0\) as \(h\to 0\). The second statement is clear from the above. \(\square\)

The fundamental theorem of calculus

The definition of the derivative extends naturally to closed intervals (with more than one point) in \(\mathbb{R}\). If \(f:[a,b]\to E\) is differentiable and \(x\in[a,b]\), then \(f’\) is a linear map from \(\mathbb{R}\) to \(E\). For convenience, we identify \(f’(x)\) with \(f’(x)(1)\) and write \(f’(x)=c\in E\), where \(c=f’(x)(1)\). This coincides with the elementary definition of the derivative as $$
f’(x)=\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}.
$$

Lemma 9. Let \(f:[a,b]\to E\) be differentiable. If \(f’(x)=0\) for all \(x\in[a,b]\), then \(f\) is constant.

Proof. Suppose \(f(t)\ne f(a)\) for some \(t\in[a,b]\), and choose a linear functional \(\lambda\) such that \(\lambda(f(t))\ne \lambda(f(a))\), e.g. by applying the Hahn-Banach theorem. Then \(\lambda\circ f\) is differentiable and \((\lambda\circ f)’(x)=0\) for all \(x\in[a,b]\), which implies that \(\lambda\circ f\) is constant. This is a contradiction. \(\square\)

Integration is essential to the study of differentiation. Here we assume the existence of an integral that can integrate Banach space valued continuous functions defined on closed intervals, e.g. the Bochner integral or the Regulated integral. However, we only use the most basic properties of the integral, such as linearity and the absolute value estimate.

Theorem 10 (Fundamental theorem of calculus). Let \(f:[a,b]\to E\) be an integrable function, and suppose that \(f\) is continuous at \(x\in[a,b]\). Then the map $$t\mapsto\int_a^t f$$ is differentiable at \(x\) and its derivative is \(f(x)\).

Proof. We have \begin{align}
\frac{1}{|h|}\left\vert \int_a^{x+h} f – \int_a^x f – f(x)h \right\vert &= \frac{1}{|h|}\left\vert \int_x^{x+h} [f(t)-f(x)]\,dt \right\vert \\
&\le \sup_t |f(t)-f(x)| \\
&\to 0
\end{align} as \(h\to 0\), where the \(\sup\) is taken over all \(t\) between \(x\) and \(x+h\) where \(f(t)\) is defined. \(\square\)

Corollary 11. Let \(f:[a,b]\to E\) be continuous, let \(F:[a,b]\to E\), and suppose that \(F’(x)=f(x)\) for all \(x\in[a,b]\). Then $$
\int_a^b f = F(b)-F(a).
$$

Proof. Apply Lemma 8 to the map $$x \mapsto F(x) – \int_a^x f.$$ \(\square\)

Corollary 12 (Integration by parts). Let \(E_1,E_2,F\) be Banach spaces and suppose there is a continuous bilinear map \(\cdot:E_1\times E_2\to F\). Let \(f:[a,b]\to E_1\) and \(g:[a,b]\to E_2\) be continuously differentiable functions. Then $$\int_a^b f’g + \int_a^b fg’ = f(b)g(b)-f(a)g(a).$$

Proof. We have \((fg)’=f’g+fg’\) by the product rule, so integrating both sides from \(a\) to \(b\) and applying Corollary 11 produces the result. \(\square\)

Mean value inequalities

Let \(\alpha:[a,b]\to L(E,F)\) be a continuous map into the space of linear maps from \(E\) to \(F\). If \(x\in[a,b]\) and \(y\in E\) then we write \(\alpha(x)y\) for the element \(\alpha(x)(y)\in F\).

Lemma 13. Let \(\alpha:[a,b]\to L(E,F)\) be a continuous map and let \(y\in E\). Then $$
\int_a^b \alpha(t)y\,dt = \left(\int_a^b \alpha(t)\,dt\right)y.
$$

Proof. The map \(\lambda\mapsto\lambda(y)\) is a continuous linear map from \(L(E,F)\) to \(F\), so the result follows from the general fact that $$\varphi \int_X f = \int_X \varphi \circ f$$ whenever \(\varphi\) is a continuous linear map between Banach spaces. \(\square\)

Theorem 14 (Mean value theorem). Let \(A\subseteq E\) be an open set, let \(f:A\to F\) be continuously differentiable, let \(x\in A\), and let \(v\in E\). If the line segment \(x+tv\) with \(0\le t\le 1\) is contained in \(A\), then $$
f(x+v)-f(x)=\int_0^1 f’(x+tv)v\,dt=\left(\int_0^1 f’(x+tv)\,dt\right)v.$$

Proof. Let \(g(t)=f(x+tv)\) so that \(g’(t)=f’(x+tv)v\). By the fundamental theorem of calculus, we have $$
g(1)-g(0)=\int_0^1 g’.
$$ Since \(g(0)=f(x)\) and \(g(1)=f(x+v)\), the result follows and we can apply Lemma 13. \(\square\)

The mean value theorem shows that the change in \(f(x)\) is determined by its derivative and the change in \(x\). This is made precise in the corollary below.

Corollary 15 (Mean value inequality). Let \(A\subseteq E\) be an open set, let \(f:A\to F\) be continuously differentiable, and let \(x,y\in A\). If the line segment between \(x\) and \(y\) is contained in \(A\), then $$
|f(y)-f(x)| \le |y-x| \sup_u |f’(u)|,
$$ where the \(\sup\) is taken over all \(u\) in the line segment. If \(z\in A\), then $$
|f(y)-f(x)-f’(z)(y-x)| \le |y-x| \sup_u |f’(u)-f’(z)|,
$$ with the \(\sup\) as above.

Proof. We have \begin{align}
|f(y)-f(x)| &= \left\vert \int_0^1 f’(x+t(y-x))(y-x)\,dt \right\vert \\
&\le |y-x|(1-0)\sup_{t\in[0,1]} |f’(x+t(y-x))|,
\end{align} which proves the first statement. Then apply this result to the map defined by \(g(v)=f(v)-f’(z)v\) to obtain the second statement. \(\square\)

Corollary 16 (Lipschitz estimate for \(C^1\) maps). Let \(A\subseteq E\) be a convex open set and let \(f:A\to F\) be continuously differentiable. If there is a constant \(M\) such that \(|f’(x)| \le M\) for all \(x\in A\), then $$
|f(x)-f(y)| \le M|x-y|
$$ for all \(x,y\in A\).

This allows us to generalize Lemma 9.

Corollary 17. Let \(A\subseteq E\) be a connected open set and suppose that the derivative of \(f:A\to F\) is zero on \(A\). Then \(f\) is constant.

Proof. If \(x\in A\) and \(B_r(x)\) is any open ball around \(x\) contained in \(A\) then Corollary 16 shows that \(f\) is constant on \(B_r(x)\). Since \(A\) is connected, \(f\) is constant on \(A\). \(\square\)

Next time, we will look at higher derivatives and the symmetries that arise.

Navigation: 1. The derivative | 2. Higher derivatives | 3. Partial derivatives | 4. Inverse and implicit functions | 5. Maxima and minima

Game Theory

So far the instructors have explained normal form games, Nash equilibrium, Pareto optimality, mixed strategies and maxmin strategies. The definitions are pretty clear and plenty of examples follow. The graded problem sets are usually fairly easy and sometimes include interesting examples not found in the videos. However, there are no proofs of any theorems or results. In fact, the theorems themselves are not even written down in any precise way – the instructors seem to be intent on avoiding mathematical notation here for some reason. Verdict: Enrol.

The Modern World: Global History since 1760

Starting near the end of the Commercial Revolution, the instructor explores global history all the way up to the present. There are usually one or two questions at the end of each video to determine whether you have been paying attention. The instructor (Philip Zelikow from the University of Virginia) really shows his enthusiasm for the subject in the videos. Verdict: Enrol!

Image and video processing: From Mars to Hollywood with a stop at the hospital

The instructor begins by giving many applications of image processing, and then gets straight into topics like Huffman coding and the discrete cosine transform (no proofs). I haven’t watched a lot of them yet. Verdict: ???

Introduction to Finance

The instructor is clearly excited about the subject, but keeps going off on tangents. Furthermore, he takes around an hour to explain the compound interest formula \(P(1+r/n)^{nt}\), something that should take 5 minutes at most. Use Khan Academy instead. Verdict: Avoid.

Analytic Combinatorics, Part I

Finally, a course in which the forums aren’t full of complaints about “too much math”. Sedgewick’s voice can be a little boring at times, but the content should be extremely interesting to anyone who knows about generating functions. Verdict: Enrol!

Determine convergence/divergence for the nonnegative series below. Prove your answers.

1. \(\displaystyle\sum_{n=2}^\infty\frac{1}{(\log n)^p}\) for any real \(p\)
2. \(\displaystyle\sum_{n=2}^\infty\frac{1}{(\log n)^n}\)
3. \(\displaystyle\sum_{n=2}^\infty\frac{1}{(\log n)^{\log n}}\)
4. \(\displaystyle\sum_{n=3}^\infty\frac{1}{(\log n)^{\log\log n}}\)
5. \(\displaystyle\sum_{n=2}^\infty\frac{1}{n \log n}\)
6. \(\displaystyle\sum_{n=3}^\infty\frac{1}{n (\log n) (\log\log n)^2}\)
7. \(\displaystyle\sum_{n=2}^\infty\frac{\log n}{e^{\sqrt{n}}}\)
8. \(\displaystyle\sum_{n=0}^\infty\frac{n!}{e^{n^2}}\)
9. \(\displaystyle\sum_{n=1}^\infty\log\left(1+\frac{1}{\sqrt{n}}\right)\)
10. \(\displaystyle\sum_{n=1}^\infty(\sqrt[n]{n}-1)^n\)
11. \(\displaystyle\sum_{n=1}^\infty(\sqrt[n]{n}-1)\)

Answers/Hints

1. No, compare with harmonic series.
2. Yes, compare with geometric series or use ratio/root test.
3. Yes, compare with \(\sum 1/n^2\).
4. No, compare with harmonic series.
5. No, use integral test.
6. Yes, use integral test.
7. Yes, compare with \(\sum 1/n^{3/2}\).
8. Yes, compare with geometric series.
9. No, compare with \(\log(1+1/n)\) and consider partial sums.
10. Yes, use root test.
11. No, compare with harmonic series. It may help to show that \((1+1/n)^n \le 3\) for all \(n\).

Very Big Hints

1. Obvious for \(p \le 0\). For \(p < 0\), we have \(\log n < n^{1/p}\) for sufficiently large \(n\).
2. \((\log n)^n > 2^n\) for sufficiently large \(n\).
3. \((\log n)^{\log n} = e^{(\log n)(\log \log n)} = n^{\log\log n} > n^2\) for sufficiently large \(n\).
4. \((\log n)^{\log\log n} = e^{(\log\log n)^2} < e^{(\sqrt{\log n})^2} = n\) for sufficiently large \(n\).
5. Integrate by taking \(u = \log x\).
6. Integrate by taking \(u = \log\log x\).
7. For sufficiently large \(n\) we have \(\log n < n^{1/4}\) and \(e^{\sqrt{n}} > e^{(\log n)^2} = n^{\log n} > n^2\).
8. \(n!/e^{n^2} < n^n/e^{n^2} = (n/e^n)^n < (1/2)^n\) for sufficiently large \(n\).
9. \(\log(1+1/\sqrt{n}) \ge \log(1+1/n) = \log(n+1) – \log n\), and \(\sum_{n=1}^N \log(1+1/n) = \log(N+1)\).
10. Use the root test since \(\sqrt[n]{n}-1 \rightarrow 0\). Alternatively, for \(n \ge 2\) we have $$
    n = (1+(\sqrt[n]{n}-1))^n \ge \frac{n(n-1)}{2} (\sqrt[n]{n}-1)^2
    $$ by the binomial theorem, so $$
    \sqrt[n]{n}-1 \le \sqrt{\frac{2}{n-1}}.
    $$ Now $$
    (\sqrt[n]{n}-1)^n \le \left(\frac{2}{n-1}\right)^{n/2},
    $$ and the comparison test can be applied.
11. For all \(n\) we have $$
    \left(1+\frac{1}{n}\right)^n = \sum_{k=0}^n \binom{n}{k} \frac{1}{n^k} = \sum_{k=0}^n \left(\frac{1}{k!}\right) \frac{n(n-1) \cdots (n-k+1)}{n^k} \le \sum_{k=0}^n \frac{1}{k!} \le 3,
    $$ so \((1+1/n)^n \le 3 \le n\) for all \(n \ge 3\). Therefore \(1+1/n \le \sqrt[n]{n}\), and the comparison test can be applied.