Paddle Duel RL Arena | 1011241

The five-stage curriculum

From a discrete state to a two-agent pixel duel.

Each level adds one control problem. The observation changes, the opponent improves, or both paddles become learned agents. Every policy returns one action: up, down, or stay.

Recorded from the project environment

L1

Tabular tuple Wall practice

Learn the return

The agent sees seven discrete values for both paddles, ball position, direction, and speed. Q-learning and SARSA establish the first TD-control baselines.

L2

Level 2 gameplay against a weak scripted opponent

Tabular tuple Scripted opponent

Control against a player

Expected SARSA learns against a weak opponent. The state now needs to support defense, movement cost, and a scoring shot.

L3

Level 3 gameplay against the mixed opponent zoo

Tabular tuple Opponent zoo

Handle policy variation

Each rally draws from random, weak, tracking, and strong opponents. The policy must act before it knows which one it faces.

L4

Level 4 pixel gameplay using predictive control

RGB pixels Predictive + DAgger

Recover motion from frames

Color masks locate the paddles and ball. Two frames supply velocity. Court geometry supplies the future intercept.

L5

Level 5 duel between the submitted policy and a tracking agent

Mirrored RGB Two-agent duel

Play both sides

The right observation is mirrored, so one image policy works on either side. Self-play agents train with separate replay buffers and optimizers.

01 · Levels 1 to 3

Seven numbers describe the court.

Levels 1 to 3 bin paddle height, ball pose, heading, and speed into a tuple. That is a tabular MDP: one Q-value per (s, a). The catch is size, about 829k tuples, so most states are never visited.

my_y own paddle height bin
opp_y opponent paddle height bin
ball_x ball horizontal bin
ball_y ball vertical bin
ball_vx left or right
ball_vy up, flat, or down
ball_speed discrete speed class

Drag the paddles and ball. The tuple updates the same way the environment discretizes the court. Highlighted action is the tracking rule we actually submit for these levels: stay while the ball leaves, otherwise chase ball height.

(3, 4, 5, 2, -1, 1, 1)

02 · Lectures 7 and 8

Three targets, one Q-table.

SARSA, Q-learning, and Expected SARSA all store Q(s, a) and act ε-greedy. They differ only in the bootstrap target. We trained them on Levels 1 to 3. Held-out learned tables stayed near 20% win rate, so the submitted tabular policy is the tracking rule.

SARSA y = r + γ Q(s′, a′)

Q-learning y = r + γ max Q(s′, a′)

Expected SARSA y = r + γ Σ π(a′|s′) Q(s′, a′)

Method α 0.20 ε 0.10

Ready. Step update applies one backup to a toy Q-row.

Q(s,UP)0.05

Q(s,DOWN)0.05

Q(s,STAY)0.05

TD error0.00

03 · Levels 4 and 5

Pixels collapse to six coordinates.

The score bar is cropped out. Color masks recover left paddle, right paddle, and ball, each as a normalized (x, y). Feature DQN, DAgger, and predictive control all start from this map. Four frames stacked make a 24-D vector so velocity is implicit.

Move the objects. The vector below is what the pixel agents see.

[0.08, 0.50, 0.92, 0.42, 0.55, 0.30]

04 · Function approximation

Double DQN did not recover the bounce.

A 128×128 MLP trains on the 24-D stack with replay, a target network, Huber loss, and Double DQN targets y = r + γ Q_target(s′, argmax Q_online). Level 4 zoo: 6.7% (n = 30). Level 5 self-play vs tracking: 25% (n = 20). Residual Double DQN keeps intercept features at test time and still lost the gate: 10% zoo, 70% Level 5 (n = 10).

Toy trainer: loss and ε only. The real run is in the metrics JSON.

Episode0

Loss0.00

ε1.00

Replay0

05 · Submitted pixel policy

Predict the intercept, then aim.

Two frames give velocity. A triangle-wave fold handles wall bounces. The paddle moves to an impact offset aimed at open court, with a deadband that holds stay when it is already there.

Estimate velocity from consecutive ball positions
Compute frames until the ball reaches the paddle x
Fold y through wall reflections
Aim away from the opponent with offset 0.60

Aim offset 0.60

Held-out: 95% Level 4 zoo, 100% Level 5 tracking, n = 20. student_agent.py uses this for images unless a promotion gate is true.

vx-0.018

vy0.012

intercept y0.41

actionSTAY

Reward ledger

Every movement has a cost.

A rally can end with a point or a timeout. On timeout, cumulative reward selects the winner. The final controller stays still while the ball travels away, which saves movement cost without giving up the next interception.

Move up or down-0.001
Stay0.000
First ten paddle hits+0.060
Later paddle hits+0.020
Score a point+10.000

Left return

0.000

0 points · 0 hits · 0 moves

Right return

0.000

0 points · 0 hits · 0 moves

Rally ready. Select a side and add events.

Imitation learning · Ross et al. 2011

DAgger aggregates expert labels on the learner's states.

Predictive control already wins the gate. DAgger asks whether a 24-D Q-network can copy that expert from stacked centroids at test time. The expert is PredictivePixelAgent. Labels are expert actions.

The failure mode of cloning

Covariate shift

Let π_E be an expert policy and d_π the state distribution induced by a learner π. Behavioral cloning minimizes a classification loss on states drawn from d_{π_E}. At test time the agent acts from d_π. One wrong action puts the ball on a path the expert never labeled. The next state is then even farther from the training support. Ross, Gordon, and Bagnell (AISTATS 2011) bound the extra cost of this compounding as O(T²ε) for a horizon-T task when the per-step classification error is ε.

The DAgger fix

Query the expert on the learner's states

Dataset Aggregation trains π_i on the current labeled set D, rolls out π_i (or a mixture with the expert), asks the expert for the correct action on every visited state, and appends those pairs to D. The next clone is fit on a mixture of expert states and learner states. The same paper shows the extra cost then scales as O(Tε). The name is literal: the dataset grows by aggregation.

Behavioral cloning

Fit on d_{π_E}

L_BC = CE(Q(s), a_E), s ~ d_{π_E}

DAgger mix

β-greedy expert

a = a_E w.p. β_e, else ε-greedy Q

DQfD-style TD

Value + remaining labels

L = L_Huber + λ CE(Q(s), a_E)

Test time

Greedy on 24-D

a* = argmax_a Q_θ(s_{t-3:t}, a)

01

State the learner sees

Color masks recover left paddle, right paddle, and ball as normalized (x, y). Four successive 6-D frames are stacked into a 24-D vector, the same representation as Feature DQN. Velocity is implicit in the stack. The bounce fold is not an input.

02

Expert labels

PredictivePixelAgent is the expert: two-frame velocity, triangle-wave wall fold, aim offset 0.60, deadband 0.012. Every DAgger step queries that controller for a_E ∈ {UP, DOWN, STAY} on the current observation, even when the learner's action is the one executed in the env.

03

Clone, then aggregate

Q-values are treated as logits. 1,200 Adam steps of cross-entropy on 15,383 expert labels (50 Level 4 demo episodes). Then 220 DAgger episodes with β from 0.80 to 0.10. New states come from the mixture policy. Labels still come from the expert. The expert buffer is a deque of capacity 80,000.

04

TD with a classification regularizer

After DAgger, β = 0. The online net trains with Double DQN Huber targets and λ = 0.25 times CE on a minibatch from the expert buffer (Hester et al., DQfD). Level 5 loads those weights and repeats the four stages against tracking noise 0.00, 0.05, and 0.10.

Figure 1. In-training Level 4 zoo win rate during imitation. Behavioral cloning is one eval after 1,200 CE steps (n = 5). DAgger and TD markers are n = 10 greedy evals, no gradient steps. Opponent: Level 4 zoo. Horizon: 15 s. Render scale: 2.

Source: artifacts/metrics/level4_imitation_dqn.json, field eval. Final held-out zoo win rate on a fresh n = 20 seed set is 75%, plotted in Figure 3.

Figure 2. Behavioral cloning cross-entropy on Q-logits, 1,200 Adam updates, minibatch from the expert buffer. Values are subsampled every 52 steps from 1,200 recorded losses. First update 1.106, last 0.541.

Source: artifacts/metrics/level4_imitation_dqn.json, field bc_losses. Loss is cross-entropy on Q-logits. This stage fits expert actions.

Held-out evaluation

Measured across 20 seeds per setting.

Level 4 uses a 15-second horizon and pixel observations at render scale 2. Level 5 tests tracking opponents at three noise levels. Evaluation is deterministic and performs no parameter updates. Error bars are binomial standard error √(p(1−p)/n). A bar at 1.00 with n = 20 has SE = 0.

Figure 3. Submitted pixel policy (PredictivePixelAgent) held-out win rate. n = 20 independent seeds per bar. Horizon 15 s. Level 4 opponents as named. Level 5 is a duel against TrackingAgent at the stated action noise.

Source: artifacts/metrics/competition_eval.json. Weak is the only Level 4 setting below 95%. All 20 Level 5 episodes were wins at each noise level.

Figure 4. Pixel-policy comparison on the promotion gate. Left cluster: Level 4 zoo. Right cluster: Level 5 tracking noise 0.05. Predictive and DAgger + Double DQN: n = 20. Residual Double DQN: n = 10. Feature Double DQN: n = 30 (zoo), n = 20 (Level 5 self-play vs tracking).

Sources: imitation_vs_predictive.json (n = 20), residual_vs_predictive.json (n = 10), level4_dqn_eval.json (n = 30), level5_selfplay_eval.json (n = 20 vs tracking). Promotion requires matching or beating predictive on both clusters. Both learned gates remain promote: false.

Implementation

A small runtime path with a full training stack behind it.

The submission class dispatches by observation type. Tuple states use tabular tracking. Image states use predictive control unless residual_promoted.json or imitation_promoted.json sets promote: true.

01 · Observe RGB frame

The score bar is excluded from the agent state.

02 · Measure Six coordinates

Color masks find two paddles and the ball.

03 · Predict Folded intercept

Velocity and wall geometry locate the next hit.

04 · Act Up, down, stay

A deadband limits movement near the target.

Reproduce locally

uv for Python. bun for project tasks.

The repository includes locked dependencies, all training entry points, tests, evaluation, report source, the GIF generator used by this page, and a browser arena under Play.

Read setup notes

uv sync

# Fast integration check
bun run train:smoke

# Full training and evaluation
bun run train
bun run eval

# DAgger + Double DQN for Levels 4 and 5
bun run train:imitation
bun run eval:imitation

# Verify the submission agent
uv run python -m unittest discover -s tests -v

# Rebuild this site's GIFs
uv run python tools/generate_site_assets.py

Project artifacts

Inspect the evidence and the source.

The report explains the equations and design decisions. Raw JSON records the held-out results. The submission file contains the exact runtime policy.

PDF · 16 pages

Predict the bounce. Place the return.

From a discrete state to a two-agent pixel duel.

Learn the return

Control against a player

Handle policy variation

Recover motion from frames

Play both sides

Seven numbers describe the court.

Three targets, one Q-table.

Pixels collapse to six coordinates.

Double DQN did not recover the bounce.

Predict the intercept, then aim.

Every movement has a cost.

DAgger aggregates expert labels on the learner's states.

Covariate shift

Query the expert on the learner's states

Fit on d_{π_E}

β-greedy expert

Value + remaining labels

Greedy on 24-D

State the learner sees

Expert labels

Clone, then aggregate

TD with a classification regularizer

Measured across 20 seeds per setting.

A small runtime path with a full training stack behind it.

uv for Python. bun for project tasks.

Inspect the evidence and the source.

Technical report

Competition metrics

Imitation gate

Submission agent

Project handoff

Predict the bounce. Place the return.

From a discrete state to a two-agent pixel duel.

Learn the return

Control against a player

Handle policy variation

Recover motion from frames

Play both sides

Seven numbers describe the court.

Three targets, one Q-table.

Pixels collapse to six coordinates.

Double DQN did not recover the bounce.

Predict the intercept, then aim.

Every movement has a cost.

DAgger aggregates expert labels on the learner's states.

Covariate shift

Query the expert on the learner's states

Fit on dπE

β-greedy expert

Value + remaining labels

Greedy on 24-D

State the learner sees

Expert labels

Clone, then aggregate

TD with a classification regularizer

Measured across 20 seeds per setting.

A small runtime path with a full training stack behind it.

uv for Python. bun for project tasks.

Inspect the evidence and the source.

Technical report

Competition metrics

Imitation gate

Submission agent

Project handoff

Fit on d_{π_E}