2010 · 75 citations · 17 references
BusinessDecision SciencesMathematicsDecision TheoryMachine LearningProbability TheoryBellman EquationsDecision Under RiskReinforcement LearningRisk-sensitive Decision MakingValue FunctionsBellman Expectation EquationCumulative Distribution FunctionsDecision RulesDecision-making RulesNonparametric Distribution EstimationRecursionRisk-sensitive Reinforcement Learning
Standard Reinforcement Learning (RL) aims to optimize decision-making rules in terms of the expected return. However, especially for risk-management purposes, other criteria such as the expected shortfall are sometimes preferred. Here, we describe a method of approximating the distribution of returns, which allows us to derive various kinds of information about the returns. We first show that the Bellman equation, which is a recursive formula for the expected return, can be extended to the cumulative return distribution. Then we derive a nonparametric return distribution estimator with particle smoothingbasedonthis extendedBellman equation. A key aspect of the proposed algorithm is to represent the recursion relation in the extendedBellmanequationbyasimplereplacement procedure of particles associated with a state by using those of the successor state. We show that our algorithm leads to a risksensitive RL paradigm. The usefulness of the proposed approach is demonstrated through numerical experiments. 1.
17
Optimization of conditional value-at-risk
R. T. Rockafellar, Stan Uryasev · The Journal of Risk · 2000 · 6.5K citations
Business, Finance, Risk Management +17
Andrew G. Barto · IFAC Proceedings Volumes · 1998 · 3K citations
Near-Optimal Reinforcement Learning in Polynomial Time
Michael Kearns, Satinder Singh · Machine Learning · 2002 · 858 citations · Full text