Bot Detection Model using User Agent and User Behavior for Web Log Analysis

Abstract

In recent years, it has become a common function to automatically distribute content suitable for each user by letting AI learn the user’s behavior pattern from the user’s web access log. On the other hand, browsing information by a bot is included in the web access log. There are malicious bots for the purpose of DDos attacks and illegal mass extraction of content. Furthermore, it is not uncommon for bots to disguise themselves as if they were showing their attributes to the user. In this study, we propose a method to discriminate between the user and the bot’s web access log in order to exclude the bot’s web access log from the analysis target.