The Rematch — Alina Kuznetsova

The board in the dark with projected pieces glowing

What was built

A way to play against an automated opponent on a real board.

As a serious chess player, I often find myself having to play online due to a lack of in-person opponents. But I find this defeats the experience: reduced to a tiny screen, the physical presence and atmosphere of a real game are completely lost.

I wanted to solve this by building an interface that combines computer vision and projection to let me play against AI on a physical board, with the engine's pieces projected directly onto it in real time. A webcam mounted above the board reads my moves as I make them. The system detects which piece I moved, sends it to the engine, and projects white's response as glowing pieces on the board itself.

Rather than a generic chess interface, I chose to frame it around a specific historical moment — which gave the project its conceptual weight, and became the basis for the live demonstration.

The Concept

Replaying the moment a machine beat a human for the first time in history.

In 1997, IBM's Deep Blue defeated Garry Kasparov — then the reigning world chess champion — in a six-game match. It was the first time a computer had beaten a world champion under standard tournament conditions, and it marked a turning point: the moment machines crossed a threshold in strategic reasoning that had previously been considered distinctly human.

The sixth and deciding game is the one I chose to restage. Kasparov's seventh move in that game is widely regarded in the chess community as the error that lost him the match. Rather than recreating the game as it happened, I start from the position immediately before that move — take Kasparov's seat — and play differently, seeing how the game might have unfolded from that moment.

My pieces are physical; the engine's are projected. This was a practical decision, but also a conceptual one: I wanted to be visibly present at the board, playing against an opponent that exists only as light and code.

I set the engine to 2750 Elo, matching Deep Blue's estimated 1997 strength, and gave it three seconds per move — long enough to feel like it was thinking, short enough to keep the game moving.

Deep Blue vs Kasparov (1997), Game 6 — immediately before Black's 7...h6. White has just played Nf3 (highlighted). Black to move.

The System

Seven modules. One live performance loop.

The system consists of seven Python modules: six handle one stage each of a pipeline from camera input to projected output, and the seventh — main.py — ties them together by running the live loop, keeping the clock, and projecting status messages such as check and checkmate.

main.pyrun loop, clock
and status text

overhead camera

apply board warpcorrect perspective from above

motion_detector.pycheck board is still

occupancy_detector.pydetect where my pieces are

game_state.pyinfer move, call Stockfish

renderer.pydraw white pieces

apply projector warpalign projection to board

projector to board

←

board_calibration.py

←

projector_calibration.py

Key Decision 01

A three-stage settling system to avoid reading half-finished moves

Since the camera reads the board continuously, I needed a reliable way to know when a move was actually complete. A simple motion check wasn't enough — a brief pause mid-move could be mistaken for a finished position. The solution was a three-stage state machine: MOVING while a piece was in motion, SETTLING once motion stopped, and STABLE only after the board had stayed still for a full second. The board was only read on reaching STABLE, ensuring the system never acted on an incomplete move.

Key Decision 02

Brightness comparison, not piece recognition

Rather than identifying pieces by shape, occupancy_detector.py checks which squares are darker than baseline using a proportional drop: (baseline - current) / baseline. This meant a single threshold worked across both light and dark squares — a black piece on a dark square barely changes absolute brightness, but against that square's low baseline the fractional drop is still clearly detectable.

Key Decision 03

Inferring moves, not reading them

game_state.py works backwards: it runs through every legal black move and checks which one produces the arrangement the webcam saw. Almost always only one legal move matches. The only exception was pawn promotion — where the detector cannot distinguish piece type — accepted as a known limitation since reaching promotion against a 2750 Elo engine was extremely unlikely.

board_calibration.py — warped top-down view with grid overlay. Green lines falling correctly between squares confirm the warp is accurate.

occupancy_detector.py — green rectangles mark detected squares. All black pieces in starting positions correctly identified.

The Demo

Playing the full game from start to checkmate.

Demonstrated as a live performance in front of an audience. Starting from the position immediately before Kasparov's seventh move in Game 6, I played Black against the engine at 2750 Elo.

The full game ran from first move to checkmate — each module completing its role without interruption. The engine put me in checkmate in around three minutes.

The loss was expected. The point was never to beat it, but to replay the game from that specific position and see how else it could unfold.

My pieces being physical while the engine's were projected drew a clear line between the two sides: I was present at the board; the machine existed only as light.

What I Learnt

Two problems. Two lessons worth keeping.

Problem 01

When to stop fixing code and fix the physical setup instead

My original board had dark brown squares that proved too dark for the occupancy detector. I spent time trying to solve it in software — attempting texture-based detection, trying to exclude projected squares from the read — but each fix introduced a new problem. Eventually I accepted that the right answer was to reprint the board with lighter dark squares.

Once I did, the proportional brightness threshold worked cleanly across the whole board. Knowing when to stop adjusting the code and change the physical setup instead turned out to be one of the more useful things I took from this project.

Problem 02

If you add something the night before, test every variation of it

Everything was working smoothly until the night before the first performance, when I decided to project illuminated status text onto the board announcing check and checkmate. I tested it and it seemed fine, so I included it.

What caught the bug was user testing — getting people to play multiple rounds across different board configurations. Testing enough variations revealed that when the text landed over a pawn specifically, the game froze. Pawns are the smallest piece, which makes them the most susceptible to a change in brightness: the projected text raised the square above the detection threshold, and the piece disappeared from the read. Solo testing hadn't caught it because the failure only appeared in specific positions that weren't obvious to anticipate in advance.

The fix was straightforward: make the text smaller and dimmer so it no longer overpowered the board. It was a reminder that getting something in front of users quickly is what surfaces the failures that matter.