<i>OptM</i>: estimating the optimal number of migration edges on population trees using <i>Treemix</i>

Abstract

The software <i>Treemix</i> has become extensively used to estimate the number of migration events, or edges (<i>m</i>), on population trees from genome-wide allele frequency data. However, the appropriate number of edges to include remains unclear. Here, I show that an optimal value of <i>m</i> can be inferred from the second-order rate of change in likelihood (Δ<i>m</i>) across incremental values of <i>m</i>. Repurposed from its original use to estimate the number of population clusters in the software <i>Structure</i> (Δ<i>K</i>), I show using simulated populations that Δ<i>m</i> performs equally as well as current recommendations for <i>Treemix</i>. A demonstration of an empirical dataset from domestic dogs indicates that this method may be preferable in large, complex population histories and can prioritize migration events for subsequent investigation. The method has been implemented in a freely available R package called "OptM" and as a web application (https://rfitak.shinyapps.io/OptM/) to interface directly with the output files of <i>Treemix</i>.