This document summarizes the development of a cheating detection system for the popular video game Call of Duty. It describes how Activision used predictive analytics and machine learning models to automatically detect cheating behaviors like boosting at scale. Initially, cheating detection was a manual process but the new system can analyze over 100 billion player records daily and identify 40,000 cheating instances, far more than the previous 200 detected manually each day, with a lower false positive rate. This proof of concept project demonstrated the value of analytics over heuristic rules and has led to greater adoption of analytics at Activision.

1. 1
Cheating Detection in Call of Duty®
Arthur Von Eschen, PhD
Sr. Director, Game Analytics
Activision
Predictive Analytics World
Berlin, 2014

2. 2
High Level… then Real Stuff
» Have you heard of Call of Duty?
» Do you play Call of Duty?
» High Level
» Provide context
» Real Stuff
» Actual analytics work

3. 3
Console Gaming New to Analytics
Game Analytics – Our Place @ Activision
Independent Studio Model (autonomy)

4. 4
Game Analytics – Our Place @ Activision
Central Studios
Started Aug 2012

5. 5
Game Analytics
» No customers
» Thought reporting was analytics
» Completely dismissed that ‘math’ or
a ‘computer’ could do something
better than a human
» No desire for analytics
» Needed an internal case study
» Didn’t try to sell accuracy, better
decision making, etc
» I didn’t talk about analytics
» Sold as ‘automation of manual work’
» They focus on more valuable work
» Built analytic models as a
comparison to heuristic rules
» Cheating (or Boosting) detection
» Was this first project / case study

6. 6
Call of Duty®
» A first person shooter
» Over 30 million people play
» For Black Ops 2
» 4.6 billion hours played
» 6.5 trillion shots fired
» 227 billion grenades thrown
» 386 billion kills made
» Multiplayer is the most
popular mode
» Video
» Bad player behavior can ruin
the experience

7. 7
Boosting (Cheating) Detection
Boosting
» Most frequent of the
unwanted behaviors
» Most diverse patterns
The problem
» Integrity of leaderboards
and competitions
» Players 20% more likely to
quit playing for day
» Players 40% more likely to
drop out of match early
We built an analytic service
» Similar to fraud detection
» Multiple models
» Decision centric
» Runs at scale
Boosting
» /api/boosting/get_top_offenders/<…>
» /api/boosting/get_offender_stats/<…>
» /api/boosting/get_counts/<…>
Other unwanted behavior
» /api/cheating/<method>
» /api/spawn_trapping/<method>

8. 8
Boosting Detection Service - Analytics
» Modeling - Classification
» Objective is to reduce false-positives
» Trained against known behaviors
» Tried logistic regression, decision trees, neural nets, rnd forest
» Settled on gradient boosting machine (GBM), AKA boosted trees
» Ensemble modeling technique
» Over 30 models in production
gbm.perf(m.deaths.gbm.final,plot.it = TRUE,oobag.curve = FALSE,overlay = TRUE,method="cv")
system.time(
test.pred_prob <- predict(m.deaths.gbm.final,data.deaths.test,type="response",n.trees=200,n.cores=4)
)
test.pred <- ifelse(test.pred_prob > 0.5,1,0)
confusionMatrix(data=test.pred,reference=data.deaths.test$boosted,positive = "1")
m.deaths.gbm.final <- gbm(boosted ~ death_space_killer_anomaly_x + death_space_victim_anomaly_y +
death_space_victim_anomaly_x + death_space_killer_anomaly_y + nemesis_deaths_ratio +
nemesis_deaths_total + hd_ratio + death_freq_anomaly + nemesis_distance_avg + bitch_kills_ratio,
data=rbind(data.deaths.train, data.deaths.test), shrinkage = 0.06, n.trees=200, cv.folds=10,
interaction.depth=29,distribution="bernoulli")
gbm.perf(m.deaths.gbm.final,plot.it = TRUE,oobag.curve = FALSE,overlay = TRUE,method="cv")
test.pred_prob <- predict(m.deaths.gbm,rbind(data.deaths.train, data.deaths.test), type="response",n.cores=4)
test.pred <- ifelse(test.pred_prob > 0.5,1,0)
confusionMatrix(data=test.pred,reference=rbind(data.deaths.train, data.deaths.test)$boosted,positive = "1")

9. 9
Boosting Detection Service - Analytics
» Modeling - Identify new methods
» Over 10 models in production
» Modeling work wasn’t the hardest part
» Each model processes over
100 billion records each day
» Applying data transformation
» Applying sophisticated algorithms
» Single model took 12 – 14 hours
to score our 30 million players
» On a large analytics cluster
» We needed to score over 40 models
every day

10. 10
Boosting Detection Service - Scaling
» Scaling was the most difficult step
» Database optimization
» Break apart queries, reuse datasets, caching,
flat-file import, chunk data
» Break process into multiple jobs and split
across server nodes
» Out of memory on DB, CPUs can’t handle
» Jobs compete for resources, not coordinated
» Query queue with query weights
» Query weights specify how “heavy” a query is
and the query manager does not send a
query for execution until there is “room”
» Run jobs in parallel (even for same model)
» Requires synchronization
» Use a msg queue (msg-oriented middleware)
» All 40+ models score < 3 hours
534
hrs
3 hrs
Before Scaling After Scaling

11. 11
Boosting Detection Service – Exposing the Service
» Build a Web Service & UI
» For integration
» For analysis
» For audit purposes
» For reporting
» Quantify problem
» See trends
» Effort
» Data Prep
» Modeling
» Scaling
» API / UI
» Boosting
» /api/boosting/get_top_offenders/<…>
» /api/boosting/get_offender_stats/<…>
» /api/boosting/get_counts/<…>
» Other unwanted behavior
» /api/cheating/<method>
» /api/spawn_trapping/<method>
35%
10%
10%
45%

12. 12
Boosting Detection Service – Results
» Quantitative Results
» The pre-analytics process
» 3 to 4 people
» Detecting roughly 200 people / day
» The analytic service
» <1 person to maintain
» Detecting over 40,000 people / day
» Lower false positive rate
» Organizational Results
» Proved that analytics (vs manual / heuristic approaches)
» Could be more precise
» Run at a greater scale
» And provide a stronger ROI
» Opened the door to greater adoption of analytics
» Increase in team size
» Increase of utilization in core product systems

13. 1313
We’re Hiring!
Arthur Von Eschen
arthur.voneschen@activision.com