Exploitation and exploration in a performance based contextual advertising system

TLDR

Online advertising requires ranking systems that balance exploitation of known high‑performing ads with exploration to discover better options, a trade‑off well studied in reinforcement learning but only recently applied to performance‑based contextual advertising. The study introduces two new exploitation‑exploration strategies tailored for online advertising. The strategies learn an optimal trade‑off and incorporate confidence metrics of historical performance, and are evaluated within an offline simulation of an industry‑leading contextual advertising platform using real event‑log data. Experiments demonstrate that the proposed algorithms improve ad reach and click‑through‑rate compared to existing approaches.

Abstract

The dynamic marketplace in online advertising calls for ranking systems that are optimized to consistently promote and capitalize better performing ads. The streaming nature of online data inevitably makes an advertising system choose between maximizing its expected revenue according to its current knowledge in short term (exploitation) and trying to learn more about the unknown to improve its knowledge (exploration), since the latter might increase its revenue in the future. The exploitation and exploration (EE) tradeoff has been extensively studied in the reinforcement learning community, however, not been paid much attention in online advertising until recently. In this paper, we develop two novel EE strategies for online advertising. Specifically, our methods can adaptively balance the two aspects of EE by automatically learning the optimal tradeoff and incorporating confidence metrics of historical performance. Within a deliberately designed offline simulation framework we apply our algorithms to an industry leading performance based contextual advertising system and conduct extensive evaluations with real online event log data. The experimental results and detailed analysis reveal several important findings of EE behaviors in online advertising and demonstrate that our algorithms perform superiorly in terms of ad reach and click-through-rate (CTR).