Similarity to Molecules in the Training Set Is a Good Discriminator for Prediction Accuracy in QSAR

TLDR

QSAR models aim to predict activity of unseen molecules, but predictive accuracy depends on how similar the target compound is to training‑set compounds, which can be measured with various descriptors. The study proposes a method to estimate prediction reliability for any chemical structure based on its similarity to the training set of a QSAR model. The authors performed retrospective cross‑validation on 20 diverse activity sets, evaluating similarity to the nearest training molecule and the number of neighbors above a cutoff as potential accuracy discriminators. High similarity to or many neighbors in the training set consistently predict lower RMSE across narrow and diverse training sets, independent of QSAR method or descriptor, with stronger effects when activity–structure correlation is high.

Abstract

How well can a QSAR model predict the activity of a molecule not in the training set used to create the model? A set of retrospective cross-validation experiments using 20 diverse in-house activity sets were done to find a good discriminator of prediction accuracy as measured by root-mean-square difference between observed and predicted activity. Among the measures we tested, two seem useful: the similarity of the molecule to be predicted to the nearest molecule in the training set and/or the number of neighbors in the training set, where neighbors are those more similar than a user-chosen cutoff. The molecules with the highest similarity and/or the most neighbors are the best-predicted. This trend holds true for narrow training sets and, to a lesser degree, for many diverse training sets and does not depend on which QSAR method or descriptor is used. One may define the similarity using a different descriptor than that used for the QSAR model. The similarity dependence for diverse training sets is somewhat unexpected. It appears to be greater for those data sets where the association of similar activities vs similar structures (as encoded in the Patterson plot) is stronger. We propose a way to estimate the reliability of the prediction of an arbitrary chemical structure on a given QSAR model, given the training set from which the model was derived.