1. Analysis on San Francisco Crime Rate
M.Sc. Student: Jiaying Li, Supervisor: Dr. Ian McLeod
Department of Statistical and Actuarial Sciences, University of Western Ontario
•Kaggle aspires to be "The Home of Data
Science". It curates many interesting chal-
lenges for modern data analytic meth-
ods. Some challenges offer large mone-
tary awards and others are offered for in-
terested students. There are nearly half a
million registered users on this website.
Kaggle
•This dataset is available from Kaggle and
comprises more than 1.7 million records
of crimes in the city during the period
2003-2015. The response variable to be
predicted is the crime type which has 39
categories as summarized in the barchart.
•The main predictors available were date
including time and GPS. This data is an
example of "long" data.
TREA
PORNOGRAPHY/OBSCENE MAT
GAMBLING
SEX OFFENSES NON FORCIBLE
EXTORTION
BRIBERY
BAD CHECKS
FAMILY OFFENSES
SUICIDE
LOITERING
EMBEZZLEMENT
ARSON
LIQUOR LAWS
RUNAWAY
DRIVING UNDER THE INFLUENCE
KIDNAPPING
RECOVERED VEHICLE
DISORDERLY CONDUCT
DRUNKENNESS
SEX OFFENSES FORCIBLE
STOLEN PROPERTY
PROSTITUTION
TRESPASS
WEAPON LAWS
SECONDARY CODES
FORGERY/COUNTERFEITING
FRAUD
ROBBERY
MISSING PERSON
SUSPICIOUS OCC
BURGLARY
WARRANTS
VANDALISM
VEHICLE THEFT
DRUG/NARCOTIC
ASSAULT
NON−CRIMINAL
OTHER OFFENSES
LARCENY/THEFT
Frequency of Crimes in San Francisco
Count
0
50000
100000
150000
Data Description
•The raw response “Category” is highly
imbalanced, with a lowest and highest fre-
quency to be 1 and 174,588 respectively.
In the preliminary analysis, “Category”
was then aggregated and reduced to 5 lev-
els based on a common classification of
crimes in law.
Preliminary analysis
•As shown in the map and the barchart
above, crime types differ in different re-
gion, but they share similar trend along
with hour.
Visualization
•Nearest neighbour classifier, naive Bayes,
quadratic discriminant analysis
•Multinomial regression
•One-vs.-rest and one-vs.one
•Tree methods, and random forest with
gradient boosting
•Neural nets, Support vector machines
•Two-stages model as a combination of
above.
All methods are already implemented in R
in well-established R packages on CRAN.
Methods Used
•For test case, I provided both a posterior
probability for each of the 39 categories,
and a final classification on each incident.
•Logloss = − 1
N
N
i=1
39
j=1 yij log(pij),
where N is the number of images in the
test set, yij = 1 if observation i is in class j,
and 0 otherwise, pij is the predicted prob-
ability that observation i belongs to class
j.
•Classification rate = nc
N , where nc is the
number of images correctly classified.
Evaluation
•Predict raw response with 39 classes di-
rectly.
Model Variables Classification rate Logloss
naive Bayes Pd,Hr,Yr,Week,DayOfWeek 22.85% 2.56
kNN PdDistrict,Hr,Yr,X,Y 22.33% 11.17
QDA X,Y 8.48% 3.47
NNet X,Y 19.97% 2.68
SVM PdDistrict,Hr,Yr,X,Y 22.41%
Not
Applicable
C5.0 X,Y 28.07% 2.50
CART X,Y 28.01% 2.87
RF PdDistrict,Hr,Yr,X,Y 23.78% 6.39
Boosting PdDistrict,X,Y 25.34% 5.12
•Predict 5 aggregated levels only.
Model Variables Classification rate
naive Bayes PdDistrict 38.78%
kNN Pd,X,Y,Hr,Yr 30.05%
QDA X,Y 30.48%
NNet DOW,Pd,X,Y,Hr,Yr 37.92%
ovo grpreg DOW,Pd,X,Y,Hr,Yr 38.81%
ova grpreg DOW,Pd,X,Y,Hr,Yr 38.78%
SVM DOW,Pd,X,Y,Hr,Yr 37.92%
C5.0 X,Y 44.02%
CART X,Y(stratified) 43.33%
RF PdDistrict,Hr,Yr,X,Y 39.72%
Boosting X,Y 41.61%
Model outputs
•One could group the crimes into 5 cate-
gories by making each class have approx-
imately equal numbers, as far as possi-
ble, to get something more balanced. The
model first classifies incidents into 5 ag-
gregated levels, followed by the second
stage to predict for each of the 39 crime
categories.
Model Variables Classification rate Logloss
C5.0+naiveBayes PdDistrict 26.08% 2.94
C5.0+C5.0 X,Y 27.11% 2.97
C5.0+Boosting PdDistrict,Hr,Yr,X,Y 27.45% 2.95
2-stage model
•My best predictor has a classification rate
of 28% which is much better than random
guessing! I have also submitted my best
predictions using C5.0 to the Kaggle web-
site, the log-loss or entropy is used to eval-
uate the model performance. For my pre-
dictor this was 2.50, which is close to 2.26,
which is the best so far on Kaggle.
•After some trials and error, location serves
as the most important factor in all mod-
els. Time of the day is useful to some ex-
tent, but models are more likely to suffer
from overfiting and a decrease in predic-
tion power if date and time are included
as predictor variables.
Conclusion
1.D. Kahle and H. Wickham. ggmap: Spatial Visualization with
ggplot2. The R Journal, 5(1), 144-161. URL http://journal.r-
project.org/archive/2013-1/kahle-wickham.pdf
2.Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical
learning (Vol. 1). Springer, Berlin: Springer series in statistics.
References
I would like to thank San Francisco OpenData for the data
source, and Kaggle for the platform.
Acknowledgement