Evaluating Item-Item Similarity Algorithms for Movies

Abstract

Recommender systems such as those used in e-commerce or Video-On-Demand systems generally show users a list of "similar items." Many algorithms exist to calculate item-item similarity and we wished to evaluate how users perceive these numerically expressed similarity. In our experiment, we performed a user study with four similarity algorithms to evaluate perceived correctness in item-item similarity as it relates to movies. We implemented three algorithms: collaborative filtering with Pearson, collaborative filtering with cosine, and content-filtering with TF-IDF. A pre-generated similarity list from TheMovieDB.org (TMDb) was used as the baseline. Our experiment showed that TMDb has the highest perceived similarity, followed by cosine and TF-IDF, while Pearson was practically unusable for users. A by-product of our experiment was a set of similar movie pairs, which we intend to use for offline evaluation.