Posted by: matheuscmss | November 12, 2009

## Teichmüller curves with complementary series

Two weeks ago (October 29), Jean-Christophe Yoccoz, Artur Avila and I talked about the ${SL(2,\mathbb{R})}$ action on the moduli space of Abelian differentials and the spectral gap of the related ${SL(2,\mathbb{R})}$ unitary representations. The outcome of our conversation was the following result:

Theorem 1 There are Teichmüller curves (associated to square-tiled surfaces) with complementary series.

Since the short proof of this simple result depends on some facts from the theory of unitary ${SL(2,\mathbb{R})}$ representations (e.g., Ratner’s results about rates of mixing), we’ll divide this post into 5 sections:

• the first 4 sections covers some well-known useful facts about unitary ${SL(2,\mathbb{R})}$ representation such as Bargmann’s classification, examples of (regular) unitary ${SL(2,\mathbb{R})}$ representations coming from dynamical systems and Ratner’s estimates of rates of mixing; the basic reference for the facts stated in these sections is Ratner’s paper “The rate of mixing for geodesic and horocyclic flows” (besides the references therein).
• the last section contains the proof of theorem 1.

For the specialists, we can advance a few keywords of the proof of theorem 1: by a recent theorem of J. Ellenberg and D. McReynolds, given any finite index subgroup ${\Gamma}$ of the congruence subgroup ${\Gamma(2)}$, we can find a square-tiled surface whose Teichmüller curve is ${\Gamma\backslash\mathbb{H}}$; in particular, it suffices to find a subgroup ${\Gamma}$ such that the regular representation of ${SL(2,\mathbb{R})}$ on ${L^2(\Gamma\backslash\mathbb{H})}$ has complementary series; however, the existence of such subgroups can be easily derived from certain cyclic covering constructions plus a “reverse Ratner estimate” argument.

Remark 1 After we performed this cyclic covering construction, Pascal Hubert and Nicolas Bergeron informed us that this procedure was already known by A. Selberg. In fact, Selberg’s argument (see the subsection “Petites valeurs propres I. Critere geometrique d’existence” of this book project of N. Bergeron) and our argument are completely equivalent: Selberg uses that sufficiently small eigenvalues (${<1/4}$) of the Laplacian on ${L^2(\Gamma\backslash\mathbb{H})}$ lead to complementary series and we use that slow decay of correlations of the geodesic flow of ${\Gamma\backslash\mathbb{H}}$ lead to complementary series, which are two equivalent ways to state the same fact in view of Ratner’s paper.

Introduction

Let ${\rho:SL(2,\mathbb{R})\rightarrow U(\mathcal{H})}$ be an unitary representation of ${SL(2,\mathbb{R})}$, i.e., ${\rho}$ is a homomorphism from ${SL(2,\mathbb{R})}$ into the group ${U(\mathcal{H})}$ of unitary transformations of the complex separable Hilbert space ${\mathcal{H}}$. We say that a vector ${v\in\mathcal{H}}$ is a ${C^k}$-vector of ${\rho}$ if ${g\mapsto\rho(g)v}$ is ${C^k}$. Recall that the subset of ${C^{\infty}}$-vectors is dense in ${\mathcal{H}}$.

The Lie algebra ${sl(2,\mathbb{R})}$ of ${SL(2,\mathbb{R})}$ (i.e., the tangent space of ${SL(2,\mathbb{R})}$ at the identity) is the set of all ${2\times2}$ matrices with zero trace. Given a ${C^1}$-vector ${v}$ of ${\rho}$ and ${X\in sl(2,\mathbb{R})}$, the Lie derivative ${L_X v}$ is

$\displaystyle L_X v := \lim\limits_{t\rightarrow0}\frac{\rho(\exp(tX))\cdot v - v}{t}$

where ${\exp(X)}$ is the exponential map (of matrices).

Exercise 1 Show that ${\langle L_Xv,w\rangle = -\langle v,L_Xw\rangle}$ for any ${C^1}$-vectors ${v,w\in\mathcal{H}}$ of ${\rho}$ and ${X\in sl(2,\mathbb{R})}$.

An important basis of ${sl(2,\mathbb{R})}$ is

$\displaystyle W:=\left(\begin{array}{cc}0&1\\-1&0\end{array}\right), \quad Q:=\left(\begin{array}{cc}1 & 0 \\ 0&-1\end{array}\right), \quad V:=\left(\begin{array}{cc}0&1\\1&0\end{array}\right)$

Exercise 2 Show that ${\exp(tW)=\left(\begin{array}{cc}\cos t&\sin t\\-\sin t&\cos t\end{array}\right)}$, ${\exp(tQ) = \left(\begin{array}{cc}e^t & 0\\ 0 &e^{-t}\end{array}\right)}$ and ${\exp(tV) = \left(\begin{array}{cc}\cosh t&\sinh t\\-\sinh t&\cosh t\end{array}\right)}$. Furthermore, ${[Q,W]=2V}$, ${[Q,V]=2W}$ and ${[W,V]=2Q}$ where ${[.,.]}$ is the Lie bracket of ${sl(2,\mathbb{R})}$ (i.e., ${[A,B]:= AB-BA}$ is the commutator).

The Casimir operator ${\Omega_{\rho}}$ is ${\Omega_{\rho}:=(L_V^2+L_Q^2-L_W^2)/4}$ on the dense subspace of ${C^2}$-vectors of ${\rho}$. It is known that ${\langle \Omega_{\rho}v,w\rangle = \langle v,\Omega_{\rho}w\rangle}$ for any ${C^2}$-vectors ${v,w\in\mathcal{H}}$, the closure of ${\Omega_{\rho}}$ is self-adjoint, ${\Omega_{\rho}}$ commutes with ${L_X}$ on ${C^3}$-vectors for any ${X\in sl(2,\mathbb{R})}$ and ${\Omega_{\rho}}$ commutes with ${\rho(g)}$ for any ${g\in SL(2,\mathbb{R})}$.

Furthermore, when the representation ${\rho}$ is irreducible, ${\Omega_{\rho}}$ is a scalar multiple of the identity operator, i.e., ${\Omega_{\rho}v = \lambda(\rho)v}$ for some ${\lambda(\rho)\in\mathbb{R}}$ and for any ${C^2}$-vector ${v\in\mathcal{H}}$ of ${\rho}$.

In general, as we’re going to see below, the spectrum ${\sigma(\Omega_{\rho})}$ of the Casimir operator ${\Omega_{\rho}}$ is a fundamental object.

Bargmann’s classification

We introduce the following notation:

$r(\lambda):=\left\{\begin{array}{cc}-1 & \quad\quad\textrm{if } \lambda\leq -1/4,\\ -1+\sqrt{1+4\lambda} & \quad\quad\quad\,\,\,\,\textrm{ if } -1/4<\lambda<0\\-2 & \textrm{if } \lambda\geq 0\end{array}\right.$

Note that ${r(\lambda)}$ satisfies the quadratic equation ${x^2+2x-4\lambda=0}$ when ${-1/4<\lambda<0}$.

Bargmann’s classification of irreducible unitary ${SL(2,\mathbb{R})}$ says that the eigenvalue ${\lambda(\rho)}$ of the Casimir operator ${\Omega_{\rho}}$ has the form

$\displaystyle \lambda(\rho) = (s^2-1)/4$

where ${s\in\mathbb{C}}$ falls into one of the following three categories:

• Principal series: ${s}$ is purely imaginary, i.e., ${s\in\mathbb{R}i}$;
• Complementary series: ${s\in (0,1)}$ and ${\rho}$ is isomorphic to the representation ${\rho_s\left(\begin{array}{cc}a&b\\c&d\end{array}\right) f(x):= (cx+d)^{-1-s} f\left(\frac{ax+b}{cx+d}\right)}$, where ${f}$ belongs to the Hilbert space ${\mathcal{H}_s:=\left\{f:\mathbb{R}\rightarrow\mathbb{C}: \iint\frac{f(x)\overline{f(y)}}{|x-y|^{1-s}}dx\,dy<\infty\right\}}$;
• Discrete series: ${s\in\mathbb{N}-\{0\}}$.

In other words, ${\rho}$ belongs to the principal series when ${\lambda(\rho)\in(-\infty,-1/4]}$, ${\rho}$ belongs to the complementary series when ${\lambda(\rho)\in (-1/4,0)}$ and ${\rho}$ belongs to the discrete series when ${\lambda(\rho)=(n^2-1)/4}$ for some natural number ${n\geq 1}$.

Note that, when ${-1/4<\lambda(\rho)<0}$ (i.e., ${\rho}$ belongs to the complementary series), we have ${r(\lambda(\rho))=-1+s}$.

Some examples of ${SL(2,\mathbb{R})}$ unitary representations

Given a dynamical system consisting of a ${SL(2,\mathbb{R})}$ action (on a certain space ${X}$) preserving some probability measure (${\mu}$), we have a naturally associated unitary ${SL(2,\mathbb{R})}$ representation on the Hilbert space ${L^2(X,\mu)}$ of ${L^2}$ functions of the probability space ${(X,\mu)}$. More concretely, we’ll be interested in the following two examples.

Hyperbolic surfaces of finite volume. It is well-known that ${SL(2,\mathbb{R})}$ is naturally identified with the unit cotangent bundle of the upper half-plane ${\mathbb{H}}$. Indeed, the quotient ${SL(2,\mathbb{R})/SO(2,\mathbb{R})}$ is diffeomorphic to ${\mathbb{H}}$ via

$\displaystyle \left(\begin{array}{cc}a&b\\c&d\end{array}\right)\cdot SO(2,\mathbb{R})\mapsto \frac{ai+b}{ci+d}$

Let ${\Gamma}$ be a lattice of ${SL(2,\mathbb{R})}$, i.e., a discrete subgroup such that ${M:=\Gamma\backslash SL(2,\mathbb{R})}$ has finite volume with respect to the natural measure ${\mu}$ induced from the Haar measure of ${SL(2,\mathbb{R})}$. In this situation, our previous identification shows that ${M:=\Gamma\backslash SL(2,\mathbb{R})}$ is naturally identified with the unit cotangent bundle ${T_1 S}$ of the hyperbolic surface ${S:=\Gamma\backslash SL(2,\mathbb{R})\slash SO(2,\mathbb{R}) = \Gamma\backslash \mathbb{H}}$ of finite volume with respect to the natural measure ${\nu}$.

Since the action of ${SL(2,\mathbb{R})}$ on ${M:=\Gamma\backslash SL(2,\mathbb{R})}$ and ${S:=\Gamma\backslash\mathbb{H}}$ preserves the respective probability measures ${\mu}$ and ${\nu}$ (induced from the Haar measure of ${SL(2,\mathbb{R})}$), we obtain the following (regular) unitary ${SL(2,\mathbb{R})}$ representations:

$\displaystyle \rho_M(g)f(\Gamma z)=f(\Gamma z\cdot g) \quad \forall\, f\in L^2(M,\mu)$

and

$\displaystyle \rho_S(g)f(\Gamma z SO(2,\mathbb{R})) = f(\Gamma z\cdot g SO(2,\mathbb{R})) \quad \forall\, f\in L^2(S,\nu).$

Observe that ${\rho_S}$ is a subrepresentation of ${\rho_M}$ because the space ${L^2(S,\nu)}$ can be identified with the subspace $H_{\Gamma}:=\{f\in L^2(M,\mu): f \textrm{ is constant along } SO(2,\mathbb{R})-\textrm{orbits}\}$. Nevertheless, it is possible to show that the Casimir operator ${\Omega_{\rho_M}}$ restricted to ${C^2}$-vectors of ${\mathcal{H}_{\Gamma}}$ coincides with the Laplacian ${\Delta=\Delta_S}$ on ${L^2(S,\nu)}$. Also, we have that a number ${-1/4<\lambda<0}$ belongs to the spectrum of ${\Omega_{\rho_M}}$ (on ${L^2(M,\mu)}$) if and only if ${-1/4<\lambda<0}$ belongs to the spectrum of ${\Delta=\Delta_S}$ on ${L^2(S,\nu)}$.

Moduli spaces of Abelian differentials. An interesting space philosophically related to the hyperbolic surfaces of finite volumes are the moduli spaces of Abelian differentials: we consider the space ${\mathcal{Q}_g}$ of Riemann surfaces ${M}$ of genus ${g\geq 1}$ equipped with Abelian (holomorphic) 1-forms ${\omega}$ of unit area modulo the equivalence relation given by biholomorphisms of Riemann surfaces respecting the Abelian differentials. By writing ${\omega = \Re(\omega)+i\Im(\omega)}$, we can make ${SL(2,\mathbb{R})}$ act naturally on ${\mathcal{Q}_g}$ by

$\displaystyle \left(\begin{array}{cc}a&b\\c&d\end{array}\right)\cdot\omega:= (a\Re(\omega) + b\Im(\omega)) + i (c\Re(\omega)+d\Im(\omega)).$

See this excellent survey of Anton Zorich for more details about the ${SL(2,\mathbb{R})}$ action on the moduli spaces of Abelian differentials.

The case of ${\mathcal{Q}_1}$ is particularly clear: it is well-known that ${\mathcal{Q}_1}$ is isomorphic to the unit cotangent bundle ${SL(2,\mathbb{Z})\backslash SL(2,\mathbb{R})}$ of the modular curve. In this nice situation, the ${SL(2,\mathbb{R})}$ action has a natural absolutely continuous (w.r.t. Haar measure) invariant probability ${\mu_{(1)}}$, so that we have a natural unitary ${SL(2,\mathbb{R})}$ representation on ${L^2(\mathcal{Q}_1,\mu_{(1)})}$.

After the works of H. Masur and W. Veech, we know that the general case has some similarities with the genus 1 situation, in the sense that, after stratifying ${\mathcal{M}_g}$ by listing the multiplicities of the zeroes of ${\omega}$ and taking connected components ${\mathcal{C}}$ of these strata, this action has an absolutely continuous (with respect to a natural “Lesbegue” class induced by the period map) invariant probability ${\mu_{\mathcal{C}}}$. In particular, we get also an unitary ${SL(2,\mathbb{R})}$ representation on ${L^2(\mathcal{C},\mu_{\mathcal{C}})}$.

Rates of mixing and size of the spectral gap

Once we have introduced two examples (coming from Dynamical Systems) of unitary ${SL(2,\mathbb{R})}$ representations, what are the possible series (in the sense of Bargmann classification) appearing in the decomposition of our representation into its irreducible factors.

In the case of hyperbolic surfaces of finite volume, we understand precisely the global picture: the possible irreducible factors are described by the rates of mixing of the geodesic flow on our hyperbolic surface.

More precisely, let

$\displaystyle A:=\{a(t) := \textrm{diag}(e^t,e^{-1})\in SL(2,\mathbb{R})\}$

be the 1-parameter subgroup of diagonal matrices of ${SL(2,\mathbb{R})}$. It is not hard to check that the geodesic flow on a hyperbolic surface of finite volume ${\Gamma\backslash\mathbb{H}}$ is identified with the action of the diagonal subgroup ${A}$ on ${\Gamma\backslash SL(2,\mathbb{R})}$.

Ratner showed that the Bargmann’s series of the irreducible factors of the regular representation ${\rho_{\Gamma}}$ of ${SL(2,\mathbb{R})}$ on ${L^2(\Gamma\backslash SL(2,\mathbb{R}))}$ can be deduced from the rates of mixing of the geodesic flow ${a(t)}$ along a certain class of observables. In order to keep the exposition as elementary as possible, we will state a very particular case of Ratner’s results (referring the reader to Ratner’s paper for more general statements). We define $\mathcal{H}_{\Gamma}:=\{f\in H_{\Gamma}:\int f = 0\}$ equipped with the usual ${L^2}$ inner product ${\langle.,.\rangle}$. In the sequel, we denote by

$\displaystyle \mathcal{C}(\Gamma)=\sigma(\Delta_S)\cap (-1/4,0)$

the intersection of the spectrum of the Laplacian ${\Delta_S}$ with the open interval ${(-1/4,0)}$,

$\displaystyle \beta(\Gamma) = \sup\mathcal{C}(\Gamma)$

with the convention ${\beta(\mathcal{C}(\Gamma))=-1/4}$ when ${\mathcal{C}(\Gamma)=\emptyset}$) and

$\displaystyle \sigma(\Gamma)=r(\beta(\Gamma)):=-1+\sqrt{1+4\beta(\Gamma)}.$

We remember the reader that the subset ${\mathcal{C}(\Gamma)}$ detects the presence of complementary series in the decomposition of ${\rho_{\Gamma}}$ into irreducible representations. Also, since ${\Gamma}$ is a lattice, it is possible to show that ${\mathcal{C}(\Gamma)}$ is finite and, a fortiori, ${\beta(\Gamma)<0}$. Because ${\beta(\Gamma)}$ essentially measures the distance between zero and the first eigenvalue of ${\Delta_S}$ on ${\mathcal{H}_{\Gamma}}$, it is natural to call ${\beta(\Gamma)}$ the spectral gap.

Theorem 2 For any ${f,g\in\mathcal{H}_{\Gamma}}$ and ${|t|\geq 1}$, we have

• ${|\langle v, \rho_{\Gamma}(a(t))w\rangle|\leq C_{\beta(\Gamma)}\cdot e^{\sigma(\Gamma)t}\cdot \|v\|_{L^2}\|w\|_{L^2}}$ when ${\mathcal{C}(\Gamma)\neq\emptyset}$;
• ${|\langle v, \rho_{\Gamma}(a(t))w\rangle|\leq C_{\beta(\Gamma)}\cdot e^{-t}\cdot \|v\|_{L^2}\|w\|_{L^2}}$ when ${\mathcal{C}(\Gamma)=\emptyset}$, ${\sup(\sigma(\Delta_S)\cap(-\infty,-1/4))<-1/4}$ and ${-1/4}$ is not an eigenvalue of the Casimir operator ${\Omega_{\rho_{\Gamma}}}$;
• ${|\langle v, \rho_{\Gamma}(a(t))w\rangle|\leq C_{\beta(\Gamma)}\cdot t\cdot e^{-t}\cdot \|v\|_{L^2}\|w\|_{L^2}}$ otherwise, i.e., when ${\mathcal{C}(\Gamma)=\emptyset}$ and either ${\sup(\sigma(\Delta_S)\cap(-\infty,-1/4))=-1/4}$ or ${-1/4}$ is an eigenvalue of the Casimir operator ${\Omega_{\rho_{\Gamma}}}$.

Here ${C_{\beta(\Gamma)}>0}$ is a constant such that ${C_{\mu}}$ is uniformly bounded when ${\mu}$ varies on compact subsets of ${(-\infty,0)}$.

In other words, Ratner’s theorem relates the (exponential) rate of mixing of the geodesic flow ${a(t)}$ with the spectral gap: indeed, the quantity ${|\langle v,\rho_{\Gamma}(a(t))w\rangle|}$ roughly measures how fast the geodesic flow ${a(t)}$ mixes different places of phase space (actually, this is more clearly seen when ${v}$ and ${w}$ are characteristic functions of Borelian sets), so that Ratner’s result says that the exponential rate ${\sigma(\Gamma)}$ of mixing of ${a(t)}$ is an explicit function of the spectral gap ${\beta(\Gamma)}$ of ${\Delta_S}$.

In the case of moduli spaces of Abelian differentials, our knowledge is less complete than the previous situation: as far as I know, the sole result about the “spectral gap” of the ${SL(2,\mathbb{R})}$ representation ${\rho_{\mathcal{C}}}$ on the space ${L^2_0(\mathcal{C},\mu_{C})}$ of zero-mean ${L^2}$-functions (wrt to the natural measure ${\mu_C}$) on a connected component ${\mathcal{C}}$ of the moduli space ${\mathcal{Q}_g}$ is:

Theorem 3 (A. Avila, S. Gouezel, J.C. Yoccoz) The unitary ${SL(2,\mathbb{R})}$ representation ${\rho_{\mathcal{C}}}$ has spectral gap in the sense that it is isolated from the trivial representation, i.e., there exists some ${\varepsilon>0}$ such that all irreducible factors ${\rho_{\mathcal{C}}^{(s)}}$ of ${\rho_{\mathcal{C}}}$ in the complementary series are isomorphic to the representation ${\rho_s}$ with ${s<1-\varepsilon}$.

In the proof of this result, Avila, Gouezel and Yoccoz proves firstly that the Teichmüller geodesic flow (i.e., the action of the diagonal subgroup ${A=\{a(t):t\in\mathbb{R}\}}$ on the moduli space ${\mathcal{Q}_g}$ of Abelian differentials is exponentially mixing (indeed this is the main result of their paper) and they use a reverse Ratner estimate to derive the previous result from the exponential mixing.

Observe that, generally speaking, the result of Avila, Gouezel and Yoccoz says that ${\rho_{\mathcal{C}}}$ doesn’t contain all possible irreducible representations of the complementary series, but it is doesn’t give any hint about quantitative estimates of the “spectral gap”, i.e., how small ${\varepsilon>0}$ can be in general. In fact, at the present moment, it seems that the only situation where one can say something more precise is the case of the moduli space ${\mathcal{Q}_1}$:

Theorem 4 (Selberg/Ratner) The representation ${\rho_{\mathcal{Q}_1}}$ has no irreducible factor in the complementary series and it holds ${|\langle v,\rho_{\mathcal{Q}_1}w\rangle|\leq C \cdot t\cdot e^{-t}}$.

In fact, using the notation of Ratner’s theorem, Selberg proved that ${\mathcal{C}(SL(2,\mathbb{Z}))=\emptyset}$. Since we already saw that ${\mathcal{Q}_1 = SL(2,\mathbb{Z}) \backslash SL(2,\mathbb{R})}$, the first part of the theorem is a direct consequence of Selberg’s result, while the second part is a direct consequence of Ratner’s result.

In view of the previous theorem, it is natural to make the following conjecture:

Conjecture (J.-C. Yoccoz) The representations ${\rho_{\mathcal{C}}}$ don’t have complementary series.

While we are not going to discuss this conjecture here, we see that the goal of the theorem 1 is to show that this conjecture is false if we replace the invariant natural measure ${\nu_{\mathcal{C}}}$ by other invariant measures supported on smaller loci.

Teichmüller curves with complementary series

After this long revision of the basic facts around unitary ${SL(2,\mathbb{R})}$-representations, we focus on the proof of theorem 1. Before starting the argument, let us briefly recall the definition of Teichmüller curves of square-tiled surfaces and the definition of the associated representation. Firstly, we remind that a square-tiled surface is a Riemann surface ${M}$ obtained by gluing the sides of a finite collection of unit squares of the plane so that a left side (resp., bottom side) of one square is always glued with a right side (resp., top) of another square together with the Abelian differential ${\omega}$ induced by the quotient of ${dz}$ under these identifications. It is known that square-tiled surfaces are dense in the moduli space of Abelian differentials (because ${(M,\omega)}$ is square-tiled iff the periods of ${\omega}$ are rational) and the ${SL(2,\mathbb{R})}$-orbit of any square-tiled surface is a nice closed submanifold of ${\mathcal{Q}_g}$ which can be identified with ${\Gamma\backslash SL(2,\mathbb{R})}$ for an appropriate choice of a finite-index subgroup ${\Gamma}$ of ${SL(2,\mathbb{Z})}$ (in the literature ${\Gamma}$ is called the Veech group of our square-tiled surface). Furthermore, the Teichmüller geodesic flow is naturally identified with the geodesic flow on ${\Gamma\backslash\mathbb{H}}$. In other words, the Teichmüller flow of the ${SL(2,\mathbb{R})}$-orbit of a square-tiled surface corresponds to the geodesic flow of ${\Gamma\backslash\mathbb{H}}$ where ${\Gamma}$ is a finite-index subgroup of ${SL(2,\mathbb{Z})}$ (hence ${\Gamma}$ is a lattice of ${SL(2,\mathbb{R})}$). In the converse direction, J. Ellenberg and D. McReynolds recently proved that:

Theorem 5 (Ellenberg and McReynolds) Any finite-index subgroup ${\Gamma}$ of the congruence subgroup ${\Gamma(2)\subset SL(2,\mathbb{Z})}$ gives rise to a Teichmüller curve of a square-tiled surface.

For a nice introduction to the Teichmüller flow and square-tiled surfaces see this survey of A. Zorich. In any case, the fact that the ${SL(2,\mathbb{R})}$-orbits ${\mathcal{S}}$ of square-tiled surfaces are identified to the unit cotangent bundle of ${\Gamma\backslash\mathbb{H}}$ permit to introduce the regular unitary ${SL(2,\mathbb{R})}$-representation ${\rho_{\mathcal{S}}}$ on the space ${L^2_0(\Gamma\backslash\mathbb{H},\nu_{\Gamma})}$ of zero-mean ${L^2}$-functions (wrt the natural measure ${\nu_{\Gamma}}$ on ${S}$) of ${\mathcal{S}}$.

In this notation, the theorem 1 says that there are square-tiled surfaces such that the representation ${\rho_{\mathcal{S}}}$ associated to its ${SL(2,\mathbb{R})}$-orbit ${\mathcal{S}}$ has irreducible factors in the complementary series.

In view of Ellenberg and McReynolds theorem, it suffices to find a finite-index subgroup ${\Gamma\subset\Gamma(2)}$ such that ${\rho_{\Gamma}}$ has complementary series. As we promised, this will be achieved by a cyclic covering procedure. Firstly, we fix a congruence subgroup ${\Gamma(m)}$ such that the corresponding modular curve ${\Gamma(m)\backslash\mathbb{H}}$ has genus ${g\geq1}$, e.g., ${\Gamma(6)}$. Next, we fix a homotopically non-trivial closed geodesic ${\gamma}$ of ${\Gamma(m)\backslash\mathbb{H}}$ after the compactification of its cusps and we perform a cyclic covering of ${\Gamma(m)\backslash\mathbb{H}}$ (i.e., we choose a subgroup ${\Gamma\subset\Gamma(m)}$) of high degree ${N}$ such that a lift ${\gamma_N}$ of ${\gamma}$ satisfies ${\ell(\gamma_N)=N\cdot\ell(\gamma)}$. Now, we select two small open balls ${U}$ and ${V}$ of area ${1/N}$ whose respective centers are located at two points of ${\gamma_N}$ belonging to very far apart fundamental domains of the cyclic covering ${\Gamma\backslash\mathbb{H}}$, so that the distance between the centers of ${U}$ and ${V}$ is ${\sim N/2}$. Define ${u=\sqrt{N}\cdot\chi_U}$, ${v=\sqrt{N}\cdot\chi_V}$. Take ${f=u-\int u}$ and ${g=v-\int v}$ the zero mean parts of ${u}$ and ${v}$.

We claim that ${\rho_{\Gamma}}$ has complementary series, i.e., ${\sigma(\Gamma)>-1}$ (i.e., ${\beta(\Gamma)>-1/4}$) for a sufficiently large ${N}$. Actually, we will show a little bit more: ${\sigma(\Gamma)}$ is arbitrarily close to ${0}$ for large ${N}$ (i.e., the spectral gap of ${\Gamma}$ can be made arbitrarily small).

Indeed, suppose that there exists some ${\varepsilon_0>0}$ such that ${\sigma(\Gamma)<-\varepsilon_0}$ for every ${N}$. By Ratner’s theorem 2, it follows that

$\displaystyle |\langle f, \rho_{\Gamma}(a_t)g\rangle|\leq C(\varepsilon_0)\cdot e^{-\sigma(\Gamma)\cdot t}\|f\|_{L^2}\|g\|_{L^2}$

for any ${|t|\geq 1}$. On the other hand, since the distance between the centers of ${U}$ and ${V}$ is ${\sim N/2}$, the support of ${u}$ is disjoint from the image of the support of ${v}$ under the geodesic flow ${a(t_N)}$ for a time ${t_N\sim N/2}$. Thus, ${\langle u,\rho_{\Gamma}(a(t_N))v\rangle = \int u\cdot v\circ a(t_N) = 0}$, and, a fortiori,

$\displaystyle |\langle f, \rho_{\Gamma}(a_t)g\rangle| = \left|\int u\cdot v\circ a(t_N) - \int u\cdot\int v\right| = \int u\cdot\int v\sim 1/N.$

Putting these two estimates together and using the facts that ${\|f\|_{L^2}\leq\|u\|_{L^2}\leq 1}$ and ${\|g\|_{L^2}\leq\|v\|_{L^2}\leq 1}$, we derive the inequality

$\displaystyle 1/N\leq C(\varepsilon_0) e^{-\varepsilon_0\cdot N/2}.$

In particular, ${\varepsilon_0\leq C(\varepsilon_0)\cdot\frac{\ln N}{N}}$, a contradiction for a sufficiently large ${N}$.

Remark 2 As we pointed out in the introduction, the algebraic part of the proof of theorem 1 was already known by Selberg: in fact, as pointed out to us by N. Bergeron and P. Hubert, the same cyclic covering construction giving arbitrarily small first eigenvalue of ${S=\Gamma\backslash\mathbb{H}}$ (i.e., arbitrarily small spectral gap) was found by Selberg and the reader can find an exposition of this argument in the subsection 3.10.1 of Bergeron’s book project. In particular, although the “difference” between Selberg argument and the previous one is the fact that the former uses the first eigenvalue of the Laplacian ${\Delta_S}$ while we use the dynamical properties of the geodesic flow (more precisely the rates of mixing) and a reverse Ratner estimate, it is clear that both arguments are essentially the same.

Remark 3 We note that $\Gamma$ isn’t a congruence subgroup (i.e., it doesn’t contain $\Gamma(m)$ for any choice of $m$) for large $N$ because Selberg proved that the first eigenvalue of any congruence subgroup is uniformly bounded from below by $1/4$ (and hence there is no complementary series for the hyperbolic surfaces related to congruence subgroups).