Can Features for Phishing URL Detection Be Trusted Across Diverse Datasets? A Case Study with Explainable AI

Abstract

Phishing has been a prevalent cyber threat that manipulates users into\nrevealing sensitive private information through deceptive tactics, designed to\nmasquerade as trustworthy entities. Over the years, proactively detection of\nphishing URLs (or websites) has been established as an widely-accepted defense\napproach. In literature, we often find supervised Machine Learning (ML) models\nwith highly competitive performance for detecting phishing websites based on\nthe extracted features from both phishing and benign (i.e., legitimate)\nwebsites. However, it is still unclear if these features or indicators are\ndependent on a particular dataset or they are generalized for overall phishing\ndetection. In this paper, we delve deeper into this issue by analyzing two\npublicly available phishing URL datasets, where each dataset has its own set of\nunique and overlapping features related to URL string and website contents. We\nwant to investigate if overlapping features are similar in nature across\ndatasets and how does the model perform when trained on one dataset and tested\non the other. We conduct practical experiments and leverage explainable AI\n(XAI) methods such as SHAP plots to provide insights into different features'\ncontributions in case of phishing detection to answer our primary question,\n"Can features for phishing URL detection be trusted across diverse dataset?".\nOur case study experiment results show that features for phishing URL detection\ncan often be dataset-dependent and thus may not be trusted across different\ndatasets even though they share same set of feature behaviors.\n