Beyond NDCG: Behavioral Testing of Recommender Systems with RecList

Patrick John Chia, Jacopo Tagliabue, Federico Bianchi, Chloe He, Brian Ko

Companion Proceedings of the Web Conference 2022 · 2022 · 14 citations · 25 references

DOIFull text

Open access

Concepts

Abstract

As with most Machine Learning systems, recommender systems are typically evaluated through performance metrics computed over held-out data points. However, real-world behavior is undoubtedly nuanced: ad hoc error analysis and tests must be employed to ensure the desired quality in actual deployments. We introduce RecList, a testing methodology providing a general plug-and-play framework to scale up behavioral testing. We demonstrate its capabilities by analyzing known algorithms and black-box APIs, and we release it as an open source, extensible package for the community.

References

25