WTF is a Hilbert Space?

An unexpectedly useful notion

Last week, I wrote extensively about the foundations of functional analysis—why it arose, what sort of questions it could be used to answer, and the most core constructions.

Why Functional Analysis?

We chased Fourier’s idea that you can decompose a function as an infinite linear combination of functions e^2πint, although we ran into some snags:

1. For most functions f: [0,1]→ℂ, if the Fourier series can be defined at all, it won’t converge pointwise to f.

2. Even if the Fourier series converges pointwise, it need not converge uniformly (that is, in the L^∞ sense). Moreover, there is no guarantee that it will behave nicely with respect to, say, differentiation.

3. Even though it is true that for any continuous, periodic function there is a sequence of finite linear combinations that will converge (in the L^∞ sense) to it, it is not always true that the Fourier series will converge to it (in the L^∞ sense). This is true even though the coefficients in those linear combinations will converge to the Fourier coefficients!

So, how do we resolve this? How do we get a theory of Fourier series that actually makes sense?

Easy: if you can’t get convergence with respect to one norm, redefine the problem and try to get convergence with respect to a different norm. Specifically, we are going to show that the natural setting in which to consider Fourier series isn’t L^∞ at all—it is L²!

Remember that last time, we talked about how each of the L^p spaces has a dual: if p,q>1 and 1/p+1/q=1, then every continuous, linear map L^p([0,1]) can be obtained from integrating against a function in L^q([0,1]). This means that L² is unique: it is the only L^p space that is self-dual (since 1/2+1/2=1).

So, if we take any two functions f, g in L²([0,1]) and integrate them against each other—

\(\int_0^1 f(t) g(t) \ \text{d}t\)

—then this operation is linear in both f and g, and it is continuous in both inputs as well.

This is almost the operation that we want to study, but not quite. Remember that the L² norm is

\(\left(\int_0^1 |f(t)|^2 \ \text{d}t\right)^{1/2}=\left(\int_0^1 \overline{f(t)}f(t) \ \text{d}t\right)^{1/2}.\)

So, to better match that, we are going to define

\(\langle f,g\rangle := \int_0^1 \overline{f(t)}g(t) \ \text{d}t.\)

Then ⟨f,f⟩=||f||₂², so it is always positive, unless f=0, in which case it is 0. This operation is now conjugate-linear in the first coordinate—that is,

\(\left\langle cf,g\right\rangle=\overline{c}\langle f,g\rangle\)

—but this is a comparatively light sacrifice.

Let me now give the general construction, and then we will talk about how to use this.

---

Definition: Let V be a complex vector space. An inner product on V is a map ⟨∗,∗⟩: V×V→ℂ that is

1. complex linear in the second variable¹ (that is, ⟨f,cg⟩=c⟨f,g⟩, and ⟨f,g+h⟩=⟨f,g⟩+⟨f,h⟩)

2. positive definite (that is, ⟨f,f⟩>0 unless f=0), and

3. swapping the inputs is equivalent to taking the conjugate—that is,

\(\langle f,g \rangle = \overline{\langle g,f\rangle}.\)

---

Observe that what we constructed above is precisely an inner product on L². But we aren’t done. The Cauchy-Schwartz inequality states that for any inner product,

\(\left|\langle f,g\rangle\right|^2 \leq \langle f,f \rangle \langle g,g \rangle.\)

This, in turn, allows you to show that the triangle inequality holds:

\(\sqrt{\langle f+g,f+g \rangle} \leq \sqrt{\langle f,f \rangle} + \sqrt{\langle g,g \rangle}.\)

What one deduces from this is that ||f||:=⟨f,f⟩^1/2 is a norm. Therefore, if a vector space has an inner product defined on it, then it is naturally a normed space. And we already discussed last time that any norm can be turned into a distance function, so the notion of completeness makes total sense. Ergo, the following definition is sensible.

---

Definition: A Hilbert space is a complete inner product space.

---

What are some examples? Well, we’ve already discussed one: L²([0,1]) is a Hilbert space, if we define

\(\langle f,g\rangle := \int_0^1 \overline{f(t)}g(t) \ \text{d}t.\)

But while that is the example that we will want to dig into more deeply, let’s back up and do something more down-to-earth first.

Specifically, let’s consider ℂⁿ with the inner product

\(\langle \vec{v},\vec{w}\rangle = \overline{\vec{v}}^T \vec{w}.\)

This is essentially just the dot product—we just need to add the complex conjugate to the first variable so that ⟨v,v⟩ will be the (square) of the Euclidean norm.

In the finite-dimensional setting, we know that the inner product has a rich geometric interpretation: we can use it to define orthogonality and the length of vectors, for instance. Specifically, two vectors are orthogonal to each other if ⟨v,w⟩=0, and the square of the length is ⟨v,v⟩.

We can also talk about orthogonal projection: take one vector w and project it onto a second vector v at a right angle.

Algebraically, this projection is

\(\frac{\langle \vec{v},\vec{w} \rangle}{\langle \vec{v},\vec{v} \rangle} \vec{v}.\)

This is especially easy when v is a unit vector: in that case, the projection is just ⟨v,w⟩v. This makes orthonormal bases very attractive.

An orthonormal basis for ℂⁿ is a collection of vectors v₁, v_2, v₃,… v_n such that:

1. It is a basis: every vector v in ℂⁿ can be written in the form c₁v₁+c₂v₂+…c_nv_n for some unique complex numbers c₁,… c_n.

2. Every vector in the basis is a unit vector.

3. Any two vectors in the basis are orthogonal to each other.

The last two points can be summarized as saying that ⟨v_i,v_j⟩=1 if i=j and 0 otherwise.

Why is an orthonormal basis so nice? As anyone who has taken a course in linear algebra will tell you, bases are nice in general—they give you convenient ways to calculate things, they give you a simple way to define linear operations (just specify what it does to the basis), and so on. But it can be a little troublesome to take a vector and write it down in terms of an arbitrary basis—it’s pretty fast, but ugly to do by hand and awful for abstract calculations.

Not so for an orthonormal basis! Finding the coefficient c_i is a one-step process: c_i=⟨v_i,v⟩, which follows immediately from linearity. So while an orthonormal basis isn’t always the right tool for the job, it is really convenient in many cases.

Now, here’s the really cool thing: although originally we defined the notions of length, orthogonality, and projection with finite-dimensional Euclidean space in mind, because everything is defined algebraically in terms of the inner product, there is absolutely no reason why we cannot take all of this theory and generalize it—without change—to any Hilbert space!

We can even lift the notion of an orthonormal basis, although there we have to do a little work. Not because the notion of orthonormality needs change—that is the same—but because the notion of basis needs to be tweaked.

The trouble is that what you might naively want to say is that a basis for a Hilbert space is a collection of vectors B such that for any vector v in our Hilbert space, there exists a unique set of complex numbers c_B such that

\(\vec{v}=\sum_{\vec{u}\in B}c_B \vec{u}.\)

But this is now an infinite sum! So we have to be ever so slightly careful and specify that v is the limit of the partial sums. We can do this because we know that the inner product induces a norm, which induces a distance function, which induces a notion of limits. This is almost the same as the finite-dimensional notion of basis that you are likely more familiar, but because limits are now in the mix, we actually have to worry about convergence.

Thankfully, that’s not too bad if we are working with an orthonormal basis. We want to determine when a sum like

\(\sum_{n=1}^\infty c_n \vec{u}_n\)

converges in our Hilbert space, given that the vectors are orthonormal. Consider taking the inner product of this sum with itself, assuming it converges:

\(\begin{align*} \left\langle \sum_{m=1}^\infty c_m\vec{u}_m,\sum_{n=1}^\infty c_n\vec{u}_n\right\rangle &= \sum_{m,n} \overline{c_m} c_n\langle \vec{u}_m,\vec{u}_n\rangle \\ &= \sum_{n=1}^\infty |c_n|^2. \end{align*}\)

(Exchanging the infinite sum with the inner product is kosher because the inner product is continuous, as we had previously deduced.)

Ah ha! So, a necessary condition is that the corresponding sum of the coefficients must converge. Less obvious, but nevertheless true, is that this is also sufficient.

Okay, with all of that out of the way, how does this apply to L²([0,1])? What would be an example of an orthonormal basis for this kind of infinite-dimensional Hilbert space?

…That is precisely what the Fourier series gives us!

Observe that if we define u_n to be the function t↦e^2πint, then

\(\langle \vec{u}_m, \vec{u}_n \rangle = \int_0^1 e^{2\pi i (n-m)t} \ \text{d}t = \begin{cases} 1 & \text{if $m=n$} \\ 0 & \text{if $m\neq n$}. \end{cases}\)

So, certainly, this collection of vectors is orthonormal. The only question is whether it is an orthonormal basis—is it true that any function in L²([0,1]) can be approximated arbitrarily closely by a linear combination of such vectors?

We’re actually most of the way to being able to prove that this is so, because we demonstrated last time that for any periodic, continuous function, there is a sequence of such linear combinations that will converge to it. That was in the L^∞ sense, but we also showed that it would converge in L^p for any p.

So, all that is left is to prove that the collection of periodic, continuous functions is dense in L²—for any f in L², there exists a sequence of periodic, continuous functions f_n such that f_n→f (in the L² sense).

Periodicity is really not an issue: since functions in L² are really equivalence classes, we can easily change the value at a point and force periodicity if we need to. Ergo, the crux of what we are asking is whether we can approximate functions in L² by continuous functions.

I think that this is certainly believable, but the details of the proof are a little technical and require digging into how integrable functions are defined. As such, I am going to skip going into detail.

(Here’s a very abridged overview: by the definition of the Lebesgue integral, one defines the integral of a function by considering sequences of sums of indicator functions—functions that are 1 on some chosen measurable set X, and 0 outside of it. Ergo, it is sufficient to prove that you can approximate any indicator function by a continuous function. By the regularity of Lebesgue measure, for any measurable set X there exists a compact set K and an open set U such that K is contained in X is contained in U, such that the measure of X is as close to the measures of K and U as we like. By Urysohn’s Lemma, there is a continuous function that is 1 inside K, 0 outside of U, and is between those two values in the middle. Because of how we constructed it, this continuous function will be as close to our original indicator function as we like.)

(Look, I warned you it was technical!)

In any case, we now see that we have proved that the functions t↦e^2πint form an orthonormal basis for L²([0,1]), from which we gather that for any function f in L²([0,1]), there exists a unique Fourier series decomposition:

\(f(t)=\sum_{n=-\infty}^\infty c_n e^{2\pi i nt},\)

where the convergence of the infinite sum must be understood to be in L², rather than, say, pointwise. Furthermore, for any such function, it is always true that

\(\sum_{n=-\infty}^\infty |c_n|^2 < \infty,\)

and conversely, if you have a sequence of complex numbers c_n that satisfy this condition, then the corresponding Fourier sum is a function in L². Finally, we can extract these coefficients using the fact that this is an orthonormal basis:

\(c_n=\left\langle t\mapsto e^{2\pi i nt},f\right\rangle.\)

And observe: all of this applies to all functions in L², not just continuous ones!

This is pretty much everything that we could wish for concerning Fourier series… with the one teeny sacrifice that we had to give up on L^∞ convergence and work with just L² convergence.

---

I want to end with a historical note. There is a weirdly common misconception that Hilbert spaces—or even all of functional analysis—grew out of quantum mechanics. I remember once being asked whether Hilbert spaces were an ad hoc mathematical construction to make sense of the Copenhagen interpretation.

David Hilbert and a diagram of the first few spherical harmonics, an orthonormal basis for the Hilbert space of square-integrable functions on the sphere. They also happen to correspond to atomic orbitals.

In reality, this is backwards. Frigyes Riesz introduced the general notion of an L^p space in 1910²; David Hilbert and Erhard Schmidt were already working with L² even before that, developing the idea of the orthonormal basis in that setting. In contrast, wave mechanics and matrix mechanics in quantum theory were only formulated in 1925—the abovementioned Copenhagen interpretation took form between 1925 and 1927.

While yes, functional analysis and Hilbert spaces are tremendously useful for quantum mechanics (and I will write about this in the future), you don’t need them to justify any of the theory—we saw above that Hilbert spaces are already very useful for entirely classical problems!

What is true is that in 1927, von Neumann showed that you could use Hilbert spaces to unify the disparate approaches of wave and matrix mechanics, giving a general theory of quantum mechanics. This revolutionary paper cemented the term “Hilbert space” and gave (with some small differences) the formal description I laid out above.³ So, certainly for physicists, this was the avenue through which Hilbert spaces entered their lexicon.

The two fields have grown together and have boosted one another. But they are, nevertheless, independent.

1. Many authors make it linear in the first variable. But I originally learned this in the context of quantum mechanics, and there it is always the second variable. That convention is the one that stuck with me.

2. Not under that name, though, if I am not mistaken.

3. Von Neumann required that Hilbert spaces be infinite-dimensional and separable—i.e., possessing a countably infinite orthonormal basis.