These slides address the problem of capturing, processing and analyzing images from the video stream of the Hearthstone game in order to obtain relevant information on the conduct of parties in this game. Since the information needs to be presented to the user in real-time, we needed to find the most suitable methods of extracting this information. Therefore, techniques such as background subtraction, histograms comparisons, key points matching, optical character recognition were investigated. Driven by the required processing speed, we ended up using optical character recognition on limited areas of interest from the captured image. After developing the application, we tested it in real-world context, while real games were played and presented the obtained results. In the end, we also provided two examples where the application would prove useful for better decision making during the game.

1. Autor Conducător științific
Universitatea
Politehnica
București
Facultatea de
Automatică și
Calculatoare
Catedra de
Calculatoare
Hearthstone Helper - Using Optical
Character Recognition Techniques For
Cards Detection
Costin-Gabriel CHIRU, Florin Oprea
costin.chiru@cs.pub.ro

2. Introduction
• Purpose: capture, process and analyze images
from the video stream of the Hearthstone
game in order to obtain relevant information
on the conduct of parties in this game
• Constraint: results should be offered in real-
time on a regular computer / tablet
• Methodology: apply different techniques for
extracting information from the images and
then use this information to identify and log
the cards that have been played.
09.09.2016 AIMSA 2016 1

3. Heartstone (Heroes of Warcraft)
• Free-to-play online game developed by Blizzard (2014)
• Turn-based game involving 2 players in a 1vs1 match
• Each player has his/her own deck of cards with 30 cards
• Each player start with 30 life points and the purpose of
the game is to play their cards so that to destroy the
opponent before they are themselves defeated
• The game features between 814 and 1003 different
cards used to cast spells, summon minions to help in the
battle or equip weapons
– They are classified in 5 categories based on their rarity and
usefulness and in 10 different classes according to their
availability to all the heroes, or to a specific one.
09.09.2016 AIMSA 2016 2

4. Application Usefulness
• By monitoring the game, the application is able to
memorize the cards that have been played and
thus:
– the player knows what remaining cards he/she has and
can compute the likelihood of extracting a particular
card
– he/she knows which cards will no longer be available in
the opponent deck (the decks may contain at most two
cards of the same type, or only one if it is a legendary
card)
– the player can predict the most probable deck of the
opponent based on the cards played so far, previous
history, opponent's rating, game style and current
meta-game.
09.09.2016 AIMSA 2016 3

5. Theoretical Background (1)
• To determine real time in-game events,
methods that are fast, efficient and accurate
are needed both for image capturing and for
cards recognition.
• Image capturing: acquire sufficient game
snapshots per seconds to detect changes
– Using API provided by Windows that allows direct
access to the opened windows from the
operating system
– up to 15 frames per second without
overwhelming the CPU
09.09.2016 AIMSA 2016 4

6. Theoretical Background (2)
• Cards recognition: needed to
– detect when a new card appeared (to avoid
continuous processing of video stream)
• Using Background Subtraction
– recognize different pictures based on the elements
from them. Can be done by:
• comparing their histograms (colors distribution)
• key points matching: choose specific points of interest and
then find the association between them.
• using key points in combination with decision trees (to
accelerate the execution)
• using perceptual hashes
• using OCR on the text appearing in the pictures and
comparing the text
09.09.2016 AIMSA 2016 5

7. Implementation Details (1)
• First of all, the Windows API was used to extract
the snapshots  extract information from the
snapshots
• Resolution might be large involving a large
processing time  only analyze specific regions
in the images to determine the start and end of a
game (analyzed areas are about 10 times smaller
than the original image)
• Next step: identify the moments of time when
these areas contain images of cards using
Background Subtraction and analyze the frames
sufficiently highlighted
09.09.2016 AIMSA 2016 6

8. Implementation Details (2)
09.09.2016 AIMSA 2016 7

9. Implementation Details (3)
• Next, we investigated the methods for image
recognition:
– Histograms: bad results due to the fact that all the
images were created from the same template
– Key points matching - invariant to rotation, scaling or
brightness: SURF (Speeded up robust features) and
SIFT (Scale-invariant feature transform) in
combination with Flann (Fast Library for approximate
nearest neighbor) for matching the key points
• Can be done offline for the cards from the DB, but the
matching algorithm might take minutes (~ 1000 cards *
tenths of a second for each card)  can’t be used for real-
time recognition
• For heroes recognition, SURF and Flann takes ~ 3 seconds
(only 9 possible heroes)  acceptable for this part  used
09.09.2016 AIMSA 2016 8

10. Implementation Details (4)
09.09.2016 AIMSA 2016 9

11. Implementation Details (5)
– Apply OCR techniques: used Tesseract and specified
specific text areas that help determining the playing
card (the card name and its attributes - cost, attack
and life points)
• Binarized the picture  better results for OCR and smaller
execution times
• Wavy text  problems for the recognition engine  had to
post-process the OCR output:
– compute the Levenshtein distance between the obtained text
and the cards name and
– use the attribute values returned by the OCR to filter out cards
• Very fast: took between 0.1 and 1.5 s
• Using OCR, the whole process (from capturing
the window image to the card classification) < 2 s
09.09.2016 AIMSA 2016 10

12. Application Output
• The GUI is to inform the player about relevant
changes in the application:
– the status of the application,
– the cards that have been played,
– the remaining cards
09.09.2016 AIMSA 2016 10
• The information
is updated in
real-time as the
cards are played

13. Cards Identification (1)
• Stormwind Knight:
– Cards from the DB with their
scores:
• "Silver Hand Knight": 2,
• "Stormwind Knight": 10,
• "Blood Knight": 10
• "The Black Knight": 2
– Use Levenshtein Distance:
• "Stormwind Knight": 9, 11, 11, 4, 8, 5
• "Blood Knight": 11, 12, 15, 8, 12, 10
– and cards’ attributes
• "Stormwind Knight": 4, 2, 5
• "Blood Knight": 3, 3, 3
• The card is correctly identified
09.09.2016 AIMSA 2016 12

14. • Kor'kron Elite :
– Do not contain any words from
the DB:
– Use cards’ attributes (4, 4, 3):
• "Kor’kron Elite"
• "Houndmaster”
• "Spellbreaker”
– And then Levenshtein Distance:
• "Kor’kron Elite": 4, 6, 2, 5, 3, 4
• "Houndmaster": 11, 13, 11, 10,
11, 12
• "Spellbreaker": 12, 14, 13, 11, 12
• The card is correctly
identified
09.09.2016 AIMSA 2016 12
Cards Identification (2)

15. Runtime and Errors
• Results obtained using an Intel(R) Core(TM) i7-4500U CPU @ 2.40 GHz
• Time analysis:
– Detection of a new game (SURF & FLANN): avg. 0.104s
– of the heroes of the two players (SURF & FLANN): avg. 0.346s
– Cards’ detection (OCR):
• Accuracy: system average accuracy was 99.1%.
– Errors due to partial occlusion of the card that needs to be detected.
– GUI offers the possibility of manually correcting the errors.
09.09.2016 AIMSA 2016 12
Type of cards OCR attributes OCR names Total
Cards extracted by the user 0.0146 0.7305 1.0159
Cards played by the opponent 0.0086 0.8234 0.9361
Type of cards Identified cards Errors
Cards extracted by the user 209 1
Cards played by the opponent 227 3
Total 436 4

16. Conclusions
• We presented an application whose purpose was to
monitor in-game events that occur in the Hearthstone
game
• Main requirement was to be quick enough to provide
answers in real time
• For the recognition of the analyzed images, we have
tried two categories of methods:
– One based on the picture recognition: using histograms or
using algorithms for key point matching  time
consuming
– one based on OCR use only the text from the pictures
 fast and accurate
• The application helps players in making the right
decisions or in taking calculated risks, thus improving
their performance and gaming experience
09.09.2016 AIMSA 2016 12

17. Questions
09.09.2016 AIMSA 2016 12
Thank you very much!