Reinforcement Learning for Trading
BSc thesis: a PPO + RNN-LSTM trading agent in a custom OpenAI Gym environment, tuned via a 50-set technical-indicator search, that outperformed manual trading on the same indicators.
- Python
- PPO
- RNN-LSTM
- Reinforcement Learning
- Gym
- Optuna
Trading as sequential decision-making
Rather than forecasting next-day price (which supervised models are bad at, given how volatile the market is), this frames trading as sequential decision-making: an agent observes recent price history and account state, chooses to buy, sell, or hold, and is rewarded for the profit that choice compounds into. Training data is real historical OHLCV pulled from Yahoo Finance (Apple, Microsoft, IBM), inside a custom OpenAI Gym environment.
- State: the last 5 days of price data (scaled 0–1), plus account balance, shares held, total shares sold, and total sales value, plus whichever technical indicators that phase was testing.
- Action: number of shares to buy, sell, or hold at the current step.
- Reward: current balance scaled by a delay modifier (
timestep / max_timesteps), which biases the agent toward compounding long-run profit over grabbing an early gain and sitting out.
Three phases to a stable agent
- Preliminary: A2C first, dropped for inconsistent results; switched to an RNN-LSTM policy, which profited in 8/10 runs but still wasn't reliable. Optuna hyperparameter tuning on top of it didn't fix that.
- Secondary (indicator search): swept 50 random sets of technical indicators (RSI, MACD, Bollinger Bands, money-flow and volatility families, etc.), 10 runs per set, to find which combination gives stable output across companies. Best set (
volatility_dcl,volatility_bbm,trend_ema_fast,volume_fi,volatility_dchi,volatility_bbh, alongside the five base OHLCV features) returned $1,760.56 simulated profit, well ahead of the next-best set. - Final: Optuna-tuned PPO hyperparameters (16 epochs, minibatch 128, γ=0.91, GAE λ=0.94, learning rate 0.0021 vs. Stable-Baselines defaults) with the RNN-LSTM policy, plus 8 manually chosen indicator sets. Each was benchmarked against a human trader working the same indicators on TradingView.
Beating the manual benchmark
Across all 8 indicator sets in the final phase, the trained agent outperformed manual trading on the identical indicators. In the best case, it nearly doubled the human trader's profit ($968 manual vs. $1,958.40 agent).
| Algorithm | Policy network | Profit |
|---|---|---|
| PPO (proposed) | RNN-LSTM | $2,037.00 |
| PPO | DNN | $1,854.34 |
| PPO | RNN | $1,421.33 |
| A2C | DNN | $1,254.72 |
| A2C | RNN-LSTM | $854.14 |
| TRPO | DNN | $712.29 |
BSc thesis, BRAC University (2020), with Md Sultan Parvez and Partho Talukdar, supervised by Mahbubul Alam Majumdar, PhD.