arXiv (Cornell University) · 2017 · 16 citations · 0 references
ManagementDecision SciencesSupply Chain ManagementInventory ControlInventory ManagementMachine LearningDeep LearningInventory Control PoliciesInventory OptimizationInventory TheoryReinforcement LearningReplenishment PlanningBullwhip Effect ReductionDeep Reinforcement LearningReinforcement Learning In GamesReinforcement Learning TheoryReplenishmentSerial Supply Chain Inventory ControlStock ReplenishmentStock Replenishment Policies
The beer game is a widely used in-class game that is played in supply chain\nmanagement classes to demonstrate the bullwhip effect. The game is a\ndecentralized, multi-agent, cooperative problem that can be modeled as a serial\nsupply chain network in which agents cooperatively attempt to minimize the\ntotal cost of the network even though each agent can only observe its own local\ninformation. Each agent chooses order quantities to replenish its stock. Under\nsome conditions, a base-stock replenishment policy is known to be optimal.\nHowever, in a decentralized supply chain in which some agents (stages) may act\nirrationally (as they do in the beer game), there is no known optimal policy\nfor an agent wishing to act optimally.\n We propose a machine learning algorithm, based on deep Q-networks, to\noptimize the replenishment decisions at a given stage. When playing alongside\nagents who follow a base-stock policy, our algorithm obtains near-optimal order\nquantities. It performs much better than a base-stock policy when the other\nagents use a more realistic model of human ordering behavior. Unlike most other\nalgorithms in the literature, our algorithm does not have any limits on the\nbeer game parameter values. Like any deep learning algorithm, training the\nalgorithm can be computationally intensive, but this can be performed ahead of\ntime; the algorithm executes in real time when the game is played. Moreover, we\npropose a transfer learning approach so that the training performed for one\nagent and one set of cost coefficients can be adapted quickly for other agents\nand costs. Our algorithm can be extended to other decentralized multi-agent\ncooperative games with partially observed information, which is a common type\nof situation in real-world supply chain problems.\n