Covariate models for bernoulli bandits

Abstract

Sequential selections are to be made from two stochastic processes, or "arms", each yielding Bernoulli responses. At each stage the arm selected depends on previous observations. The objective is to maximize the expected number of successes in the first n selections. The probability of success for a given selection depends on a covariate through an appropriate transformation. For one arm, this transformation is completely known; for the other, it depends on unknown parameters. Properties of optimal strategies are related to those for non-covariate models. Conditions are described under which it is optimal to observe the arm with unknown parameters if the covariate lies to one side of an index value.