Reflection Groups, Generalized Schur Functions, and the Geometry of Majorization

Abstract

Let $G$ be a closed subgroup of the orthogonal group $O(n)$ acting on $R^n$. A real-valued function $f$ on $R^n$ is called $G$-monotone (decreasing) if $f(y) \geqq f(x)$ whenever $y \precsim x$, i.e., whenever $y \in C(x)$, where $C(x)$ is the convex hull of the $G$-orbit of $x$. When $G$ is the permutation group $\mathscr{P}_n$ the ordering $\sim$ is the majorization ordering of Schur, and the $\mathscr{P}_n$-monotone functions are the Schur-concave functions. This paper contains a geometrical study of the convex polytopes $C(x)$ and the ordering $\precsim$ when $G$ is any closed subgroup of $O(n)$ that is generated by reflections, which includes $\mathscr{P}_n$ as a special case. The classical results of Schur (1923), Ostrowski (1952), Rado (1952), and Hardy, Littlewood and Polya (1952) concerning majorization and Schur functions are generalized to reflection groups. It is shown that a smooth $G$-invariant function $f$ is $G$-monotone iff $(r'x)(r'\nabla f(x))\leqq 0$ for all $x \in R^n$ and all $r \in R^n$ such that the reflection across the hyperplane $\{z\mid r'z = 0\}$ is in $G$. Furthermore, it is shown that the convolution (relative to Lebesgue measure) of two nonnegative $G$-monotone functions is again $G$-monotone. The latter extends a theorem of Marshall and Olkin (1974) concerning $\mathscr{P}_n$, and has applications to probability inequalities arising in multivariate statistical analysis.