Reinforcement learning for agents that act in physical or simulated worlds, from tabular methods to deep ones, with robots as the running example. I audited it.
Main ideas
- Markov decision processes and the value functions underneath everything.
- Temporal-difference control (Q-learning, SARSA, Expected SARSA) on small problems, where you can watch the table fill in.
- Function approximation, then deep Q-networks and Double DQN once the state comes from pixels.
- Policy gradients and actor-critic methods, with baselines and advantages to keep the variance down.
- Imitation learning, where the agent copies a teacher and then has to correct its own drift (DAgger).
- Self-play, where the opponent improves as the agent does.
The common thread is acting under uncertainty when the reward arrives late and noisy.
Project
In a team of four, Paddle Duel: five levels of a two-player paddle game.
| Level | What the agent sees | Methods |
|---|---|---|
| 1 | a small table of states | Q-learning, with SARSA for comparison |
| 2 to 3 | the same | Expected SARSA against a scripted opponent, then a zoo of them |
| 4 | pixels, reduced to features | Double DQN, DAgger imitation, predictive control |
| 5 | the same | self-play, DAgger, predictive control |
At the pixel levels, a simple controller that predicts where the ball will land and moves the paddle there beat the learned policies on held-out games. So the final player uses that controller, and a learned policy replaces it only after beating it on games it never trained on. The site has an arena where you can play the trained agents, and an explainer for each method.
Relevance to my research
Keeping a simple controller as the fallback, and letting a learned policy take over only once it has proved itself, is close to what runtime monitoring and safe control do for learned controllers (see Simplex). Opponents that change as you learn are a small version of long-term autonomy.