The Rank 3 Permutation Representations of the Finite Classical Groups

Abstract

The permutation representations in the title are all determined, and no surprises are found to occur. Introduction.A group G has rank 3 in its permutation representation on the cosets of a subgroup K if there are exactly 3 (K, K)-double-cosets; that is, if K has exactly 3 orbits on the set G j K of K-cosets.Such permutation representations have been studied a great deal during the past 15 years; classical groups have been intensively studied for more than a century.The purpose of this paper is to relate these two areas, by proving the following results.THEOREM 1.1.Let M be one of the groups Sp(2m-2,q), g±(2m,q), g(2m-l,q) or SU (m,q) for m;;' 3 and q a prime power.Let M <l G with GjZ(M)";; Aut(MjZ(M)).Assume that G acts as a primitive rank 3 permutation group on the set X of cosets of a subgroup K of G. Then (at least) one of the following holds up to conjugacy under Aut(MjZ(M)).(i) X is an M-orbit of Singular ( or isotropic) points.(ii) X is an M-orbit of maximal totally singular (or isotropic) subs paces and M = Sp(4, q), SU(4, q), SU(5, q), g-(6, q), g+ (8, q) or g+ (10, q).(iii) X is any M-orbit of nonsingular points and M = SU(m,2), g± (2m, 2), g ± (2m, 3) or g(2m -1,3).(iv) Xis either orbit of nons in gular hyperplanes of M = g(2m -1,4) or g(2m -1,8) (where G = g(2m -1,8) .3 in the latter case).(viii) M = Sp(6,2), K = G 2 (2) (Edge [15], Frame [19]).(ix) M = g(7, 3), K n M = G 2 (3).(x) M = SU(6,2), K n M = 3 .PSU(4, 3) •2 (Fischer [17]).