Reinforcement-Learning Based Portfolio Management with Augmented Asset Movement Prediction States

Yunan Ye, Hengzhi Pei, Boxin Wang, Pin‐Yu Chen, Yada Zhu, Xiao Ju

Proceedings of the AAAI Conference on Artificial Intelligence · 2020 · 159 citations · 16 references

DOIFull text

Open access

Concepts

TL;DR

Portfolio management seeks maximal profits or minimal risks by continuously extracting information from diverse data sources and optimizing decisions, making it a promising area for reinforcement learning. We propose SARL, a State-Augmented RL framework for portfolio management that tackles data heterogeneity and market uncertainty. SARL augments each asset’s state with a price‑movement prediction derived from financial data or news, enabling the RL agent to better handle heterogeneous inputs and non‑stationary markets. Experiments on Bitcoin and HighTech markets, using 7‑year Reuters news, show that SARL outperforms existing portfolio‑management methods in accumulated and risk‑adjusted profits, and simulations confirm that state augmentation significantly boosts performance.

Abstract

Portfolio management (PM) is a fundamental financial planning task that aims to achieve investment goals such as maximal profits or minimal risks. Its decision process involves continuous derivation of valuable information from various data sources and sequential decision optimization, which is a prospective research direction for reinforcement learning (RL). In this paper, we propose SARL, a novel State-Augmented RL framework for PM. Our framework aims to address two unique challenges in financial PM: (1) data heterogeneity – the collected information for each asset is usually diverse, noisy and imbalanced (e.g., news articles); and (2) environment uncertainty – the financial market is versatile and non-stationary. To incorporate heterogeneous data and enhance robustness against environment uncertainty, our SARL augments the asset information with their price movement prediction as additional states, where the prediction can be solely based on financial data (e.g., asset prices) or derived from alternative sources such as news. Experiments on two real-world datasets, (i) Bitcoin market and (ii) HighTech stock market with 7-year Reuters news articles, validate the effectiveness of SARL over existing PM approaches, both in terms of accumulated profits and risk-adjusted profits. Moreover, extensive simulations are conducted to demonstrate the importance of our proposed state augmentation, providing new insights and boosting performance significantly over standard RL-based PM method and other baselines.

References

16