I’m willing to bet that most students who have used Rudin’s *Principles of Mathematical Analysis* have encountered this problem:

15. Call a mapping of \(X\) into \(Y\)

openif \(f(V)\) is an open set in \(Y\) whenever \(V\) is an open set in \(X\).Prove that every continuous open mapping of \(\mathbb{R}\) into \(\mathbb{R}\) is monotonic.

## Obvious?

Here is one “solution” that is fairly intuitive. It relies on finding a minimum or maximum and considering the image of a small neighborhood around that min/max:

**Lemma 1.** Let \(f:\mathbb{R}\to\mathbb{R}\) be a continuous open map. Then there cannot be three points \(p_1 < p_2 < p_3\) such that \(f(p_1),f(p_3) < f(p_2)\) (\(\wedge\) shape) or \(f(p_1),f(p_3) > f(p_2)\) (\(\vee\) shape).

**Proof.** Assume that the first case holds and let \(M=\sup f([p_1,p_3])\); since \(f([p_1,p_3])\) is compact, there is a point \(x\in[p_1,p_3]\) such that \(f(x)=M\). Since \(f(p_1),f(p_3) < f(p_2)\), we must have \(x \in (p_1,p_3)\). But then \(f((p_1,p_3))\) is not open, which is a contradiction. \(\square\)

Now we can prove the main theorem.

**“Proof” of main theorem.** If \(f\) is not monotonic, then there are three points as in Lemma 1. \(\square\)

The problem with this “proof” is the assumption: surely we must have a \(\wedge\) shape or a \(\vee\) shape, but how do we really prove its existence? Negating the definition of “monotonic” merely gives us the existence of four points \(a,b,c,d \in \mathbb{R}\) satisfying the following: \begin{align}

a < b &\quad\mathrm{and}\quad f(a) < f(b), \\
c < d &\quad\mathrm{and}\quad f(c) > f(d).

\end{align}

Then we use a purely combinatorial argument by a math.SE user:

If \(a=c\), use \(adb\) to form \(\vee\) or \(abd\) to form \(\wedge\).

If \(a=d\), use \(cdb\) to form \(\vee\).

If \(b=c\), use \(abd\) to form \(\wedge\).

If \(b=d\), use \(acd\) to form \(\wedge\) or \(cab\) to form \(\vee\).

Now we are left with the case when \(a,b,c,d\) are pairwise distinct.

If \(a < b < c < d\), use \(abd\) to form \(\wedge\) or \(acd\) to form \(\wedge\). If \(a < c < b < d\), use \(abd\) to from \(\wedge\) or \(acd\) to form \(\wedge\). If \(a < c < d < b\), use \(acd\) to form \(\wedge\) or \(cdb\) to form \(\vee\). If \(c < a < d < b\), use \(cdb\) to form \(\vee\) or \(cab\) to form \(\vee\). If \(c < d < a < b\), use \(cdb\) to form \(\vee\) or \(dab\) to form \(\vee\).

## A different approach

Such a case-by-case analysis is not particularly enlightening, so is there a different approach? This is the best I could come up with when I was attacking the problem:

We first start with a lemma. Call a function \(f:\mathbb{R}\to\mathbb{R}\) **locally monotonically increasing/decreasing** if for every \(x \in \mathbb{R}\), there is a neighborhood of \(x\) on which \(f\) is monotonically increasing/decreasing.

**Lemma 2.** A continuous open map \(f:\mathbb{R}\to\mathbb{R}\) is monotonically increasing if it is locally monotonically increasing. The same applies if “increasing” is replaced with “decreasing”.

**Proof.** If \(f\) is not monotonically increasing then we can find points \(x < y\) such that \(f(x) > f(y)\). Choose a neighborhood \(U \subseteq (-\infty,y)\) of \(x\) on which \(f\) is monotonically increasing. \(f\) cannot be constant on \(U\), for otherwise \(f(U)\) would not be open. So there is a point \(x’ \in U\) such that \(f(x’) \ne f(x)\). If \(x < x'\) then \(f(x) < f(x')\) and we can form a \(\wedge\) shape, and if \(x' < x\) then \(f(x') < f(x)\) and we can again form a \(\wedge\) shape. Both cases contradict Lemma 1. \(\square\)

Using this, we have:

**Proof of main theorem.** Let \(f:\mathbb{R}\to\mathbb{R}\) be a continuous open map. Let \(A\) be the set of all points \(x\) such that \(f\) is monotonically increasing in a neighborhood of \(x\). Define \(B\) similarly, but with “monotonically decreasing” instead. These two sets are clearly open. They are disjoint, for if \(x \in A \cap B\) then there is a neighborhood \(U\) of \(x\) on which \(f\) is both monotonically increasing and decreasing, i.e. constant; this is a contradiction since \(f(U)\) is not open. If we can show that \(A \cup B = \mathbb{R}\) then by the connectedness of \(\mathbb{R}\) either \(A=\mathbb{R}\) and \(B=\emptyset\) or \(A=\emptyset\) and \(B=\mathbb{R}\), in which case we can apply Lemma 2 to conclude that \(f\) is monotonically increasing or decreasing. Suppose \(x \in \mathbb{R} \setminus (A \cup B)\). If there exists a point \(y > x\) with \(f(y) > f(x)\) then since \(f\) is continuous, there is a neighborhood \(U\) of \(x\) such that \(f(U) \subseteq (-\infty,f(y))\). But \(f\) fails to be monotonically increasing on \(U\), so we have a \(\vee\) shape that contradicts Lemma 1. By a similar argument, there cannot be a point \(y > x\) with \(f(y) < f(x)\). Therefore \(f\) is constant on \((x,\infty)\), which contradicts the fact that \(f\) is open. \(\square\)

## An exercise

Prove Lemma 2 without assuming that \(f\) is continuous or open. *Hint: if \(x < y\) then \([x,y]\) is compact.*