A COMPARISON OF THREE INDIRECT METHODS FOR ESTIMATING AVERAGE LEVELS OF GENE FLOW

Montgomery Slatkin

Evolution · 1989 · 1.3K citations · 32 references

DOIFull text

Open access

Concepts

TL;DR

Three methods for estimating average gene flow in natural populations are discussed and compared, each yielding estimates of the combined parameters (number of migrants Nm in a demic model or neighborhood size 4πDσ² in a continuum model) that determine the relative importance of gene flow and genetic drift. The study reviews the theory of these methods and derives new analytic results for the expectation of FST in stepping‑stone and continuum models with small sample sets. The authors compare three methods—FST, rare‑allele, and maximum‑likelihood—by deriving analytic expectations for FST and evaluating their effectiveness with simulated data. The FST and rare‑allele methods produce comparable estimates across diverse conditions and are similarly sensitive to selection and population structure, while maximum‑likelihood methods are biased with few locations but more accurate with many; overall, FST is likely more useful under realistic conditions.

Abstract

Three methods for estimating the average level of gene flow in natural population are discussed and compared. The three methods are FST , rare alleles, and maximum likelihood. All three methods yield estimates of the combination of parameters (the number of migrants [Nm] in a demic model or the neighborhood size [4πDσ2 ] in a continuum model) that determines the relative importance of gene flow and genetic drift. We review the theory underlying these methods and derive new analytic results for the expectation of FST in stepping-stone and continuum models when small sets of samples are taken. We also compare the effectiveness of the different methods using a variety of simulated data. We found that the FST and rare-alleles methods yield comparable estimates under a wide variety of conditions when the population being sampled is demographically stable. They are roughly equally sensitive to selection and to variation in population structure, and they approach their equilibrium values at approximately the same rate. We found that two different maximum-likelihood methods tend to yield biased estimates when relatively small numbers of locations are sampled but more accurate estimates when larger numbers are sampled. Our conclusion is that, although FST and rare-alleles methods are expected to be equally effective in analyzing ideal data, practical problems in estimating the frequencies of rare alleles in electrophoretic studies suggest that FST is likely to be more useful under realistic conditions.

References

32