Simple cases for non-simplicity

A subgroup $H$ of a group $G$ is said to be normal if one has $gH=Hg$ for every $g\in G$. The French terminology is that $H$ be distingué, and I must say that I find the English terminology definitely not illuminating, for it is quite a peculiar property for a subgroup $H$ to be normal. The definition itself my seem not illuminating and I start here with two better ways of introducing the notion of a normal subgroup.

1. An equivalence relation $\sim$ on $G$ is compatible with its group structure (namely, $g\sim g'$ and $k\sim k'$ imply $gk\sim g'k'$ for all $g,g',k,k'\in G$), so that the group structure of $G$ induces a group structure on the quotient set $G/\mathord\sim$ if and only if the class $H$ of the neutral element is a normal subgroup of $G$ and the relation satisfies: $g\sim g'$ if and only if $g'g^{-1}\in H$. 
2. Another definition could start with saying that $H$ is a normal subgroup if and only if there exists a morphism of groups $f\colon G\to G'$ such that $H=\ker(f)$. This is reminiscent of MacLane's definition of an abelian category (monomorphisms are kernels).

As I just said, being a normal subgroup is quite an extraordinary property, and a given group needs not having any other normal subgroup than the obvious ones: the full group, and the group reduced to the neutral element. These groups are called simple, with the exception of the trivial group which, by convention, is not said to be simple.

In a commutative group, every subgroup is normal; this is at least one useful point for the initial definition, to make this obvious. From this, one sees that a commutative group is simple if and only if it is cyclic of prime order.

Many exercises of undergraduate algebra ask to prove that a given subgroup of a given group is normal, or that a given group is not simple, and I discuss here a recent paper (Geller, Sue. « Simply Not a Simple Group ». The American Mathematical Monthly 127, n^o 4 (20 avril 2020): 352‑53. https://doi.org/10.1080/00029890.2020.1704167), starting with a classical exercise.

Let $G$ be a group and let $H$ be a subgroup of index $2$. Then $H$ is a normal subgroup.

Let $g\in G$. If $g\in H$, then $gH=H=Hg$, because $H$ is a subgroup. Otherwise, the group $G$ splits as the disjoint union of $H$ and $gH$, as well as the disjoint union of $H$ and $Hg$, because $H$ has index $2$. Consequently, $gH=G\setminus H=Hg$.

Let $G$ be a finite group and let $H$ be a subgroup whose index $p$ is the least prime factor of $\mathop{\mathrm{Card}}(G)$. Then $H$ is a normal subgroup.

Here I wish to let $G$ act on the left on the quotient set $G/H$. This action is transitive, and this furnishes a nontrivial morphism of groups from $G$ to the symmetric group $\mathfrak S(G/H)\simeq \mathfrak S_p$. Let $G'$ be the image of this morphism and let $N'$ be its cardinality. Since $N'$ divides the cardinality of $G$, the prime factors of $N'$ are at least $p$. However, $N'$ divides the cardinality of $\mathfrak S_p$, which is $p!$, so that the prime factors of $N'$ are at most $p$. Two cases are possible, either $N'=1$ (contradicting the fact that the initial action is transitive), or $N'=p$: the image of $f$ has cardinality $p$ so that $\ker(f)$ is a subgroup of index $p$. If $g\in\ker(f)$, one has in particular $gH=H$, so that $\ker(f)\subset H$. Since $H$ has also index $p$, we have $H=\ker(f)$, and $H$ is a normal subgroup.

Remark that if $H$ is a subgroup of a group $G$, and $G$ acts on the left on $G/H$, the kernel of the corresponding group morphism from $G$ to $\mathfrak S(G/H)$ is the intersection of all $gHg^{-1}$, the largest normal subgroup of $G$ which is contained in $H$.

A class of groups which are most often not simple are $p$-groups, that is, finite groups whose cardinality is a power of a prime number $p$. Indeed, such a group is never simple, unless it is a cyclic group of order $p$. This follows from the consideration of the action of $G$ on itself by conjugation. Assuming $G\neq 1$, the conjugacy classes of central elements have cardinality $1$, while the cardinality of the conjugacy class of a non-central element is divisible by $p$; the class equation implies that $\mathop{\mathrm{Card}}(G)-\mathop{\mathrm{Card}}(Z)$ is divisible by $p$, where $Z$ is the centrum of $G$; then $\mathop{\mathrm{Card}}(Z)$ is divisible by $p$, so that it is nontrivial, unless $Z=G$, that is, $G$ is commutative, and then cyclic of order $p$.

I can now discuss Sue Geller's note, which gives a criterion for non-simplicity of some finite groups, based on the existence of natural group morphisms from $G$ to permutation groups, given by the action of $G$ in the set of its Sylow subgroups.

Let $G$ be a finite group and let $p$ be a prime number. Write $n=\mathop{\mathrm{Card}}(G)=p^s m$, where $s\geq0$ and $p$ does not divide $n$. A subgroup $S$ of $G$ is a $p$-Sylow subgroup if its cardinality is equal to $p^s$. The Sylow theorems state the following properties of the set $\Sigma_p$ of $p$-Sylow subgroups and its cardinality $\sigma_p$.

1. The set $\Sigma_p$ is not empty (1st Sylow theorem); more generally, $G$ admits subgroups of all cardinalities dividing $p^s$.
2. One has $\sigma_p \equiv 1\pmod p$ and $\sigma_p\mid m$ (3rd Sylow theorem).
3. The map $g\mapsto gSg^{-1}$ induces an action of $G$ on the set $\Sigma_p$, and this action is transitive (2nd Sylow theorem), so that $\sigma_p=(G:N_G(S))$.
   

In particular, if $\sigma_p=1$, then the unique $p$-Sylow subgroup of $G$ is normal and $G$ is not simple (unless its order is a power of $p$, in which case $G$ is simple if and only if $G$ is cyclic of order $p$.

The book (I. Martin Isaacs, Finite Group Theory, Graduate Studies in Mathematics, v. 92 (Providence, R.I: American Mathematical Society, 2008) refines the congruence $\sigma_p\equiv 1\pmod p$. I give it here as a corollary.

Corollary. — Let $t$ be an integer such that for every distinct $p$-Sylow subgroups $S$ and $T$ of $G$, one has $\mathop{\mathrm{Card}}(S\cap T) \leq p^t$. Then $\sigma_p \equiv 1 \pmod{s-t}$.

Let $S$ be a $p$-Sylow subgroup of $G$ and let us restrict the action of $G$ on $\Sigma_p$ to n action of $S$. The orbit of $S$ is $S$ itself; its cardinality is $1$. If $T$ is another $p$-Sylow subgroup of $G$, then its orbit has cardinality $(S:Q)$, where $Q=S\cap N_G(T)$ is the set of all $s\in S$ such that $sTs^{-1}=T$. Then $Q$ is a $p$-group, hence is contained in a Sylow subgroup of $N_G(T)$, but $T$ is a Sylow subgroup of $N_G(T)$ which is normal (by construction of the normalizer), so that $T$ is the unique $p$-Sylow subgroup of $N_G(T)$; this implies that $Q\subset T$, hence $Q=S\cap T$. Consequently, $\mathop{\mathrm{Card}}(T)$ divides $p^t$ and the cardinality of the orbit of $T$ is a multiple of $p^{s-t}$. The corollary follows by recalling that the sum of the cardinalities of the orbits is equal to $\sigma_p$.

The theorem stated by Sue Geller is the following. It is in fact a variant of corollary 1.3 from the book of Isaacs.

Theorem. — Assume that $\sigma_p>1$ and that $n$ does not divide $\sigma_p!$. Then $G$ is not simple.

The transitive action of $G$ on the set $\Sigma_p$ furnishes a non-trivial morphism of groups $f\colon G\to\mathfrak S(\Sigma_p)$. In particular, $\ker(f)\neq G$. If $f$ were injective, then $G$ would be isomorphic to a subgroup of $\mathfrak S(\Sigma_p)\simeq \mathfrak S_p$, hence its order would divide $\sigma_p!$, a contradiction. Consequently, $\ker(f)$ is a non-trivial normal subgroup of $G$, and $G$ is not simple.

Example 1. — A group $G$ of order $48$ is not simple. One has $48=2^4\cdot 3$. Then $\sigma_2=1$ or $\sigma_2=3$. If $\sigma_2=1$, then $G$ is not simple, as we saw above. Otherwise, $\sigma_2=3$, and since $48$ does not divide $3!=6$, the theorem shows that $G$ is not simple.

Example 2. — Any simple group $G$ of order $60$ is  isomorphic to the alternating group $\mathfrak A_5$. The proof will consist in constructing a subgroup $H$ of $G$ of index $5$; then the action of $G$ on $G/H$ induces a non-trivial morphism from $G$ to $\mathfrak S(G/H)\simeq\mathfrak S_5$; since $G$ is simple, this morphism is injective. The composition with the signature morphism must be injective as well, so that its image is contained in the alternating subgroup. Since $G$ and $\mathfrak A_5$ both have cardinality $60$, $f$ is an isomorphism from $G$ to $\mathfrak A(G/H)\simeq\mathfrak A_5$.
This subgroup $H$ will be constructed as the normalizer of a subgroup of order $2$ ou $4$. From $60=2^2\cdot3\cdot5$, one gets $\sigma_2\in\{1,3,5,15\}$. 

The case $\sigma_2=1$ contradicts the assumption of simplicity for $G$, and the case $\sigma_2=3$ contradicts the theorem above, for $60$ does not divide $6$. 

If $\sigma_2=5$, let $S$ be a $2$-Sylow subgroup of $G$ and let $H=N_G(S)$ be its normalizer; one has $(G:H)=\sigma_2=5$.

Assume finally that $\sigma_2=15$.  By the above corollary, since $\sigma_2=15\not\equiv 1\pmod 4$, there exist distinct $p$-Sylow subgroups $S$ and $T$ of $G$ such that $P=S\cap T\neq\{1\}$. Let $H=N_G(P)$ be the normalizer of $P$. One has $H\subset S$ and $H\subset T$ (so says Isaac, without saying why), so that the inclusion $S\subset H$ is strict. Moreover, the inclusion $H\subset G$ is strict because $S$ is normal in $H$, but not in $G$. If $(G:H)\neq 5$, then $(G:H)=3$, and the action of $G$ on $\mathfrak S(G/H)$ furnishes an embedding of $G$ into $\mathfrak S_3$, which is absurd since $G$ has cardinality $60$ and $\mathfrak S_3$ has cardinality 6.