1. Chess Vision: Computer Vision Chess Movement Detection
Ben Seaton
Pacific University
Email: seat3542@pacificu.edu
Abstract
The development of a computer vision system that recognizes chess piece movement
creates a variety of sub­problems such as pattern, object, and color recognition. Recognizing
chess piece movement is no trivial task especially when restrictions that makes the lives of
programmers easier, such as unique piece and board coloring, are eliminated. However it is
these restrictions that make this seemingly daunting problem approachable to those seeking a
project­based way to learn key components in computer vision. Therefore, a solution is
presented here that represents an intermediate compromise designed to teach fundamental
aspects of computer vision while still providing a challenge. It’s important to note that this
project serves as an introduction to the field of computer vision and is presented as such.
Prior to this project, I have had no experience in the field of computer vision. This project is
not a result of a formal “computer vision” course, nor did it involve the use of any “computer
vision” books. Instead, this project is approachable for anyone with an internet connection
willing to read a few research papers and sift through some man pages.
Introduction
The modern era has nearly destroyed board games with the advent of portable
screens containing literally millions of applications designed to do everything you’ve never
thought of. Yet despite this vendetta against board games, chess has proven adaptable. The
modern day chess player needs only a moment of time to input their move into a chess
engine and reproduce that move in an online match thus crushing their unsuspecting
opponent. But for those who cringe at the very thought of computer played chess and instead
long for gameplay on a live board, computer vision may offer a solution. Imagine being able to
play live chess (or any sort of live game for that matter) with people from all over the world in
the comfort of your own living room. While this technology may not exist in the
aforementioned form, the near future may present such a solution. For this solution to exist,
there must be a mechanism for recognizing chess piece movement on a live board. An
approach to recognizing chess piece movement is discussed in this paper which can be
broken down into three subproblems:
1.​ ​Board Detection – finding the board consists of identifying all 64 squares and their
coordinates.
2.​ ​Color Detection – piece and square color is used to determine the orientation of the
board.
3.​ ​Move Detection – successive frames are examined and compared to each other to
see where a move has been made.

2. (a) (b) (c)
Fig. 1. a,b,c) images show original setup. blank background is used to eliminate noise due to the carpet. camera is fixated at a
top down view.
A blank background was used for the purpose of reducing noise due to the dot like
patterns (see figure 2) of the carpet the board was placed on. To simulate reasonable
background noise that you might find on a table, whiteboards with drawn on dots were placed
on the sides of the chessboard. The whiteboards also created a glaring effect similar to what
one might find on a table.The program discussed was created using OpenCV 2.4.10 and
python 2.7.
Similar Work
Work done on the determination of chess piece movement is fairly limited though it
does exist. The most common approach seems to be coupling an edge detection algorithm or
corner detection algorithm with the hough line transform to first detect the board. Once the
board has been determined, successive frames are compared, and if there is a difference
between these frames it can be determined that a move has been made. Each solution to this
problem will differ in the flexibility that it affords the user. For example, some solutions may
restrict the board to a certain position or require some specialized equipment. To give some
perspective, a few similar works are discussed below:
● CVChess implemented by two students at Stanford University used a combination of
board image homography and heatmaps to determine moves made. The advantage of
this method is that it provides movement detection at a realistic angle [1].
● A project titled “Similar computer vision system…” presents a project based way to
learn image processing. The project described bares similarities with the solution
discussed in this paper, but greatly simplifies the task by having the user select the
region of interest by outlining the board before the start of the game. Furthermore, the
solution suggests marking the upper left corner of the board to distinguish the
orientation of the board. This paper proves extremely helpful for those looking to get
started on this problem [2].

3. (a) (b)
(c) (d)
(e)
Fig. 2. a.) original hough transform taken on image e. notice that no lines exist outside of the grid are detected , b) shows
filtered lines into two perpendicular sets, c) shows filtered sets of lines 9x9 strongest lines, d) all squares detected, e) original
preliminary corner detection algorithm leaves a number of undiscovered corners.

4. Restrictions and Assumptions
Before each module is discussed, a number of assumptions and restrictions have to
be made. Ideally, there would be no interaction between the computer and the player once
the program is executed and the camera and board would be free to move all over the place,
however, this can prove extremely challenging even for those who are experienced in the field
of computer vision. As a result, several restrictions and assumptions have been made (see
figure 2 for setup):
1. The camera is in a fixated position overlooking the board in a top down
manner.
2. Extreme angles between the board and camera do not exist. Essentially, the
board should be parallel to the camera, though there is room for variability.
3. Once the game has started, the board is not to be moved otherwise
recalibration of square coordinates will be necessary.
4. The board is assumed to be set up correctly once gameplay has started so that
the orientation of the board can be accurately determined by examining piece
and square coloring. This component will be examined in detail later in the
paper.
5. Frames are captured via the spacebar key, thus requiring user interaction after
each move is made signifying the end of that player’s turn. Think of the
spacebar as the chess clock.
6. Lighting conditions are optimal. “Normal” room lighting should be sufficient for
this project due to enhancement techniques later discussed.
7. The chessboard fills the at least half of the frame.
Board Detection
Board detection begins with the shi tomasi corner detection algorithm, which acts as a
preliminary filter and is the basis for the method described. The shi tomasi algorithm
accurately finds a large number of corners describing the chess board within the region of
interest. The region of interest or ROI is simply the 9x9 lines that make up the grid like pattern
forming all 64 squares of the chessboard. Corners discovered outside of the ROI present few
problems as most will form no coherent pattern that the hough line transform will take into
consideration with carefully chosen threshold values. Lines discovered outside the ROI can
easily be filtered out since these lines will likely contain a small number of votes and our
method involves taking those lines that have garnered a large number of votes from the
hough line algorithm. It’s important to note that those corners found using the Shi Tomasi
Corner Detection Algorithm will not be considered in the final determination of corners.
Instead, the intersection of the hough lines will be used to determine the final corners.

5. Found corners are overlaid onto a blank image (see figure 2e), and the Hough line
transform is used to find lines (see figure 2a). These lines are filtered into groups of lines with
similar theta values. The group with the largest rho difference between its smallest rho value
and largest rho value is selected as a group that represents a set of lines considered to be
part of the ROI. The second set of lines are determined by examining the remaining sets’
theta values to find a perpendicular set. At this point, two sets representing the grid have been
found (see figure 2b). Determination of vertical and horizontal lines can be arbitrarily chosen
since the orientation of the board is unknown at this point. The two sets will then be filtered
once more to eliminate redundant lines where lines with unique rho values and the most votes
are chosen resulting in 18 lines, 9 horizontal and 9 vertical, found (see figure 2d). The
intersection of each horizontal and vertical line is determined and thus all 81 points are found.
At this point, each square can be represented by a tuple of 4 points and the board is said to
be “detected”.
​(a) (b) (c)
(d) (e)

6.
Fig. 3. a,b) these images show masked isolated regions of an image using the “blobs” seen in d, c) shows results of the
canny edge detector, d) after a series of erosions and dilations, the pieces are “filled” in, e) contours are found and fitted with an
ellipse. green ellipses represent light colored pieces while red represent dark colored pieces
Color Detection
The user may choose to orient the board in any direction, thus leaving it up to the
programmer to determine where the light and dark colored pieces are positioned. Before the
orientation of the board can be determined the top left corner of the board must be arbitrarily
chosen (e.g. a1,h1,a8,h8). A1 was chosen for this solution and consequently, a static array of
strings representing the board with this orientation in mind was created.
Creating a mapping from the found squares to the static array is the goal of Color
Detection. The board is assumed to be set up correctly and so the expectation would be that
A1 is a dark square which holds a light colored piece. Square color is first determined on an
empty board, though this is not required, by examining each square’s average pixel value in
grayscale. This can easily be achieved by applying 64 separate masks, using the coordinates
found in the board detection phase, onto the original image transformed to grayscale thus
isolating each square. Each value returned can further be sorted in ascending order.
Determining square color is trivial at this point, since we know the dark squares will be
contained in the first half of the sorted array and the light squares will be contained in the
other half due to the symmetric nature of the board.
Piece color determination is a much more interesting problem since at any point and
time, the number of light and dark colored pieces may differ. This process was designed as
robustly as possible such that the color of the piece could be determined given a random
setup. This sort of robustness is not necessary due to the fact that the start position is
assumed at the beginning of the game. This paper focuses on the random setup simply
because it is a more challenging (and fun) problem.
The user is to set up the board once square colors have been determined. The
position of the board is then captured as the user pushes the spacebar key.This image is then
transformed into a binary image using the canny edge detector (see figure 3c).
Each square of this image is then isolated using the same process described previously. Each
square then undergoes a series of erosions and dilations, essentially filling in the chess piece
using the edges provided by the edge detector as a boundary forming a large blob (see figure
3d). The contours of the region of interest are then found and an ellipse is fitted to the blob
using these contours. This ellipse is used to isolate the chess piece by creating a mask over
the ellipsed region. Chess pieces can then be isolated individually and their pixel value can be
returned. A series of calculations follow which will evaluate the data collectively and determine
threshold values on the fly. These threshold values will be used to set a range of average
color values that a light colored and dark colored piece should expect to fall into. An
assumption is made that no dark colored piece or light colored piece are similar, within a
predefined threshold. Thus, chess pieces should be light and dark for optimal color detection.
Using this information, the color of each piece can be determined (see figure 3e).

7.
Fig. 4. This image shown represents a game with a number of moves made. Numbers show the number of white pixels
counted in each square. Notice unoccupied squares contain a value of 0, and those that are occupied have a value upwards of
100.
Movement Detection
Once the orientation of the board has been discovered, a correct mapping between
the two­dimensional array of strings representing the board and the live board should exist.
The user is to press the spacebar after each move has been made, capturing a frame in the
process. The frame undergoes histogram equalization to improve the contrast of the image
and normalize brightness. This will help to distinguish pieces and squares that are similar in
color, particularly dark colored pieces on dark squares. Gaussian blurring is then applied to
the image to eliminate noise that may later be picked up by the canny edge detector. Finally,
canny edge detection is applied to the image, thus producing a binary image. Once again,
each square is isolated using the previously found square coordinates. The region of interest
is then shrunk approximately twenty percent to eliminate overlapping between taller pieces
into adjacent squares, thus eliminating false positives. Furthermore, shrinking this region of
interest eliminates white pixels from edges from being considered which will help to reduce
any false positives. Finally, each square is evaluated and the number of white pixels present
in each masked square region is counted (see figure 4). If the number of white pixels in a
given square exceed a threshold value, the square is determined to be occupied. Repeat 63
more times and the occupancy of every square is known. Successive frames are then

8. compared via square occupancy, and if there exists a difference then a move has been made.
The move made is then outputted appropriately using the mapping previously described.
Testing and Results
Preliminary testing has been done at this point and the results are promising. The
accuracy of move detection seems to be fairly high. False positives have not been seen in
any testing, though no more than 20 moves have been performed at once. Further testing is
recommended to obtain definite quantitative results.
Color detection has been found to be mostly accurate. However, poor contours may
be found that represent a region of interest too large which leads to falsely identified piece
colors due to the masked region encompassing adjacent squares. More testing needs to be
done for definitive results.
Conclusion
The solution described has provided is able to accurately identify chess moves. Restrictions
can easily be lifted, such as the top down view, to provide the user with a greater range of
freedom, though some additional work would need to be done to allow for this method to work
accurately. It has also been shown that color detection and move detection can be combined
to identify the color of the piece and the occupancy of the square at once since the process is
very similar. Unfortunately, recognizing a randomly given position is extremely difficult to do
and is unrealistic using the described method. The recognition of chess pieces is an ongoing
problem that would require a number of additional considerations. Perhaps the incorporation
of a camera with depth sensors could be used to provide an easy way to get piece recognition
from a top down view. Finally, this code will be on a github for anyone interested.
Demonstrations of the program in action can be viewed via these two links:
1.) ​https://www.youtube.com/watch?v=I5U7PvcNMDI&feature=em­upload_owner
2.) https://www.youtube.com/watch?v=x4cd­UaXtBM&feature=em­upload_owner
Link one shows chess piece movement, while link two shows board recognition.
References
[1]J. Hack and P. Ramakrishan, 'CVChess: Computer Vision Chess Analytics', Stanford
University, 2015.

9. [2]E. Sokic and M. Ahic­Djokic, 'Simple computer vision system for chess playing robot
manipulator as a Project­based learning example', University of Sarajevo, 2008.