# 1011241_Siddhant: Paddle Duel RL Arena

Course project for **51.512 Reinforcement Learning for Embodied AI**, covering all five levels.

| Level | Observation | Method |
|------:|-------------|--------|
| 1 | Tabular | Q-learning (+ SARSA baseline) |
| 2 | Tabular | Expected SARSA vs scripted opponent |
| 3 | Tabular | Expected SARSA vs opponent zoo |
| 4 | Pixels to features | Double DQN, DAgger imitation, predictive / residual control |
| 5 | Pixels to features | Self-play, DAgger imitation, predictive / residual control |

The competition policy uses tabular tracking for Levels 1 to 3. For Levels 4
and 5 it defaults to predictive intercept control. Residual Double DQN or
DAgger + Double DQN can replace that pixel policy only after a held-out gate
against the predictive baseline.

## Quick start

```bash
uv sync

# Short test
bun run train:smoke
bun run dashboard

# Full training
bun run train
bun run eval
bun run dashboard
```

Competition agent:

```python
from student_agent import Agent
agent = Agent(side="left")  # or "right"
```

## GitHub Pages site

The project site is `index.html` in the repository root. It uses relative asset
paths and can be published directly with GitHub Pages.

```bash
# Rebuild the eight gameplay and method GIFs
bun run site:assets

# Preview at http://127.0.0.1:8088
bun run site:serve
```

The page includes Project, Play, and Explainer tabs, all five levels, feature
extraction, trajectory prediction, DQN and DAgger results, held-out metrics,
report links, and an interactive reward ledger.

The Play tab embeds the browser arena (`play/`). You control the left paddle
against course opponents (wall, random, weak, tracking, strong, zoo), the
submitted controllers (tabular tracking, predictive), and the trained
checkpoints (Expected SARSA, Feature Double DQN, self-play Double DQN,
Residual Double DQN, DAgger). Rewards follow `paddle_duel_env.py`. Export
JSON weights with `bun run play:policies` if you retrain.

The method explainer lives at `explainer/` and is also the Explainer tab.

```bash
# Project site (Project / Play / Explainer tabs)
bun run site:serve

# Or serve only the explainer at http://127.0.0.1:8089
bun run explainer:serve

# Or serve only the arena at http://127.0.0.1:8091
bun run play:serve
```

## Layout

```
1011241_Siddhant/
  paddle_duel_env.py          # official env (vendored)
  student_agent.py            # competition entrypoint
  paddle_duel/
    agents/                   # TabularTD + FeatureDQN
    train/                    # level trainers
    cli.py / eval.py
  artifacts/{checkpoints,metrics,figures,dashboard}
  tools/dashboard.ts          # bun HTML dashboard
  explainer/                  # interactive method explainer
  play/                       # real-time human vs approaches arena
  report/report.typ           # Typst report
  notebooks/                  # optional experiment notebooks
```

## Reproduce

| Command | Purpose |
|---------|---------|
| `bun run train:1` through `train:5` | Train one level |
| `bun run train:tabular` | Levels 1 to 3 |
| `bun run train:residual` | Predictive residual Double DQN (Level 4 curriculum) |
| `bun run eval:residual` | Compare residual vs predictive; write promotion gate |
| `bun run train:imitation` | DAgger + Double DQN for Levels 4 and 5 |
| `bun run train:imitation:4` | Level 4 imitation only |
| `bun run train:imitation:5` | Level 5 imitation (init from Level 4 if present) |
| `bun run eval:imitation` | Compare imitation vs predictive; write promotion gate |
| `bun run train` | All levels |
| `bun run eval` | Evaluation and baseline ranking |
| `bun run dashboard` | Build `artifacts/dashboard/index.html` |
| `bun run play:policies` | Export trained checkpoints to `play/policies/*.json` |
| `bun run play:smoke` | Browser-physics smoke plus Play policy file check |
| `bun run report:pdf` | Compile the Typst report |

Train/eval seeds are separated in code (training seeds start at `0`; evaluation uses disjoint offsets).

## Competition results

The held-out competition run uses 20 seeds per setting and a 15-second horizon.

| Opponent | Win rate |
|----------|---------:|
| Level 4 random | 100% |
| Level 4 weak | 70% |
| Level 4 tracking | 100% |
| Level 4 strong | 100% |
| Level 4 zoo | 95% |
| Level 5 tracking, noise 0.00 | 100% |
| Level 5 tracking, noise 0.05 | 100% |
| Level 5 tracking, noise 0.10 | 100% |

Raw results are in `artifacts/metrics/competition_eval.json`.

## Agent notes

- The tabular policy reads `(my_y, other_y, ball_x, ball_y, vx, vy, speed)`.
- The default pixel policy is predictive geometry on image features only.
- Residual Double DQN learns UP/DOWN/STAY on intercept features from that geometry.
- DAgger + Double DQN clones the predictive expert on stacked centroids, then fine-tunes with environment rewards. At test time it does not run the bounce model.
- `bun run eval:residual` writes `artifacts/metrics/residual_promoted.json`.
- `bun run eval:imitation` writes `artifacts/metrics/imitation_promoted.json`.
- `student_agent.Agent` uses residual or imitation only when the matching gate says `promote: true`.
- Feature Double DQN and Expected SARSA checkpoints remain available as experiments.

## Group

- Identifier: `1011241_Siddhant`
- Role: TA solution