Beyond the technicalities of data science, most well spread deadly mistakes are simple and could even be everyday life advices.

1. avier rudent

3. enough?
depends on what you want to achieve (train neural network, AB tes8ng…)
back to sta8s8cs (mathema8cal condi8ons) + rule of thumb
Many format can be easily read with R
Text, csv, excel, protobuffer, json, xml, html, SQL…
Many sources already available
Kaggle
Websites mining
Opendata
Government agencies
library(XML)
Web.page <- htmlTreeParse("hNp://lapresse.ca")
More details in coming lectures
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com
Mistake 10
Not Having (enough) Data

4. Where does your data come from?
stand up and get out, talk with people, read doc
library(ggplot2)
library(tabplot)
tableplot(diamonds)
Do not underes;mate
naïve tests…
Mistake 9
Do not check data quality
Package R Datacheck

5. What kind of data? What do I want to know?
Geographic? Time-series? Correla8on?
Which visualiza;on?
histogram, box plot, mosaic, heat map, hexbining, scaNer plot, line chart, 3D
Many R packages available:
ggplot2
leaflet
plot_ly
corrplot
Mistake 8
Do not look at your data
Look at your data
More details in coming lectures
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com

6. Choose the right color
Color blindness, prin8ng, meaning…
Level of interac;vity?

7. Set up deadline
Make to-do lists
Project management tool (Asana)
Plan & monitor your 8me
Mistake 7
Not having a plan
Have a plan and focus on it
Do not forget the big picture geng lost into technical tools
What is the question you want to answer?
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com

8. R package caret :
evaluate model, choice, es8mate performance (regression & classifica8on)
Sta;s;cal tests :
Goodness of fit, R2, Homer-Lemeshow test (MKmisc), Wald test, k-fold valida8on
Retrain “oMen”
Observe > Clean > Understand > Train > Predict
Mistake 6
Focus on training
Ques;on: How much snow will fall on Montréal during the 5 next years?
Data: Snowfall and temperature of the last 80 years
?
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com

9. Mistake 5
Keep it complex
Do not jump first on the fashion complicated method
Keep your method as simple as possible (focus on the ques8on)
Know the limits of this method
Compare the methods (caret, ROC)
Boosted Decision
Tree coupled to
neural network
Linear regression
Complexity comes at a price
(speed, error prone,
exper8se, amount of data)
Can you afford it?
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com

10. R standard func8on: p.adjust, Bonferroni, Benjamini-Hochberg
You are a miserable shooter, proba to hit 1%
You shoot 10,000 lasers, hit at 10,001st shot
Does that make you a shooHng genius?
Mistake 4
Do not correct for multiple tests
Mul8plica8on of sensors, data gathering protocols à Era of Big Data
The more data you analyze, the more weird cases will pop up regularly
Are they significant?
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com

11. document your work
R Markdown, Shiny
Create HTML, pdf, Word, slides, webpages, CV, journal, book
Automa8cally include & update the result of your analysis
More interac8ve? Dashboards, interac8ve maps…
hNp://rmarkdown.rstudio.com/gallery.html
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com
Mistake 3
Do not communicate or document
for the others as well as for yourself
More details in coming lectures

12. R Markdown
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com

13. Café, meetups, colleagues, board game jogging club
Publish online (blog)
Ask for external view of your work
Mistake 2
Stay alone
Do not stay alone, do not work alone
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com

14. DATA science associa8on: code of conduct
hNp://www.datascienceassn.org/code-of-conduct.html
Mistake 1
Ethics is a useless luxury
What are you doing? For whom?
What is the impact of your work?
- Company, society, yourself
- Short – long term
What type of data are you analyzing?
- Law & regula8on
- Privacy
Do you have any conflict of interest?
Montréal Big Data Meetup 22nd March 2017 – Xavier Prudent – www.xavierprudent.com
Tendency to focus on the technics, on the challenge
“Yes, but” answers?

15. CAST!
Xavier Prudent XAVIER PRUDENT
Organizer MICHAEL ALBO
The Audience ALL OF YOU
Technical Support OVH
Design-Photography CHRISTINE NAULLEAU
Special Thanks to George Lucas and to the
audience for their aNen8on
question? Comment?
Feel free to contact me:!
Xavier Prudent, prudentxavier@gmail.com!