at.algebraic topology - The space of Lie group homomorphisms

The general criterion is:

Given connected compact Lie groups $G,H$, $mathrm{Hom}(G,H)$ has finitely many components if and only if $H$ is semisimple or $G=1$.

Proof.
If $G=1$ there is nothing to prove.

Assume that H is not semisimple and $Gneq 1$. Then there exists a 1-dimensional torus $T$ in $H$ and a closed normal subgroup $N$ of codimension 1 such that $TN=H$. Define $T=Tcap N$, so that $Z$ is finite and define $T'=T/Z$. Then $mathrm{Hom}(T',G)$ can be identified to the set of elements in $mathfrak{g}$ whose exponential is 1; we have already seen that it has infinitely many components.

We have a continuous restriction map $mathrm{Hom}(H,G)to mathrm{Hom}(T,G)$. Its image contains $mathrm{Hom}(T',G)$ (identified to those homomorphisms trivial on $N$), and $mathrm{Hom}(T',G)$ is clopen in $mathrm{Hom}(T,G)$ (indeed, let $Z$ have exponent $n$; then 1 is an isolated point in the set $K$ of elements $gin G$ such that $g^n=1$ and $mathrm{Hom}(T',G)$ is the fiber of $(1,dots,1)$ for the mapping $mathrm{Hom}(T,G)to K^Z$ mapping $f$ to $zmapsto f(z)$). Thus $mathrm{Hom}(H,G)$ has infinitely many components (the proof even shows that it has a continuous map onto an infinite discrete set).

---

Conversely, suppose $H$ semisimple.

Consider the map $L:mathrm{Hom}(H,G)tomathcal{L}(mathfrak{h},mathfrak{g})$ mapping $f$ to its tangent map between Lie algebras ($mathcal{L}(mathfrak{h},mathfrak{g})$ denoting the whole space of linear maps). Then $L$ is injective. Since $L(f)$ is locally conjugate to $f$ by the exponential map, $L$ is continuous, and actually is a homeomorphism to its image (because if $L(f_n)$ tends to $L(f)$, then in a given compact neighborhood of 1 we have $f_n$ tending to $f$ uniformly, and this implies that $f_n$ tends to $f$ uniformly on all of $H$).

If $H$ is simply connected, then the image of $f$ is equal to the set of Lie algebra homomorphisms $mathrm{Hom}(mathfrak{h},mathfrak{g})$, which is Zariski closed, and hence has finitely many components in the ordinary topology. In general (still assuming $H$ semisimple), let $tilde{H}$ be its universal covering and $Z$ the (finite) kernel of $tilde{H}to H$. Recalling that the exponential is surjective, write $Z=exp(Z')$ for some finite subset $Z'$ of $mathfrak{h}$; then the image of $L$ is the set of elements $u$ in $mathrm{Hom}(mathfrak{h},mathfrak{g})$ such that for all $zin Z'$ we have $u(exp(z))=1$. Let $mathfrak{g}_1$ be the set of elements $x$ in $mathfrak{g}$ such that $exp(x)=1$: then we have the restatement: the image of $L$ is
$${fin mathrm{Hom}(mathfrak{h},mathfrak{g}): f(Z')subset mathfrak{g}_1}.$$

Fix a faithful continuous linear representation of $G$ (in $mathrm{SO}(n)$ for some $n$), so that we have a faithful representation of $mathfrak{g}$ (by antisymmetric matrices): then $mathfrak{g}_1$ is the set of elements in $mathfrak{g}$ whose eigenvalues are in $2ipimathbf{Z}$. Note that as soon as $Gneq 1$, it has infinitely many connected components (since all these eigenvalues can be achieved, using a 1-parameter subgroup whose image in $G$ is a 1-dimensional torus).

Nevertheless, observe that (still assuming $H$ compact semisimple) $mathrm{Hom}(mathfrak{h},mathfrak{g})$ is a compact subset of the vector space $mathcal{L}(mathfrak{h},mathfrak{g})$. Indeed, $mathfrak{h}$ admits a basis $(e_j)$ such that each $e_j$ belongs to some subalgebra $mathfrak{h}_j$ isomorphic to $mathfrak{so}(3)$, such that some 2-dimensional complex representation of $mathfrak{h}_i$ maps $e_j$ to the diagonal matrix $(i,-i)$ (where $i^2=-1$). It follows from representation theory of $mathfrak{sl}_2$ that for every $n$-dimensional complex representation $rho$ of $mathfrak{h}_j$ (and hence of $mathfrak{h}$) maps $e_j$ to an element with eigenvalues in ${-i(n-1),dots,i(n-1)}$. Thus the trace of $-rho(e_j)^2$ is $le (n-1)^2n$. Since $-mathrm{trace}(XY)$ is a definite positive symmetric bilinear form on antisymmetric matrices, this shows that $mathrm{Hom}(mathfrak{h},mathfrak{g})$ is bounded, hence compact.

In particular, there exists $n_0$ such that eigenvalues of $f(z)$ for all $zin Z'$ and $finmathrm{Hom}(mathfrak{h},mathfrak{g})$ are in the interval $[-2ipi n_0,2ipi n_0]subset imathbf{R}$. Thus the image by $L$ of $mathrm{Hom}(H,G)$ is the set of $finmathrm{Hom}(mathfrak{h},mathfrak{g})$ such that for every $zin Z'$ we have $prod_{k=-n_0}^{n_0} (f(z)-2ipi k)=0$. This is a Zariski-closed subset, and hence has finitely many connected components in the ordinary topology, and hence so does the homeomorphic space $mathrm{Hom}(H,G)$.