Data Science at the Command Line Don't forget to think about this good old tool that is so handy, interactive and efficient, before rushing to Hadoop, Spark, Storm, etc. You can do streaming, you can make joins on csv, and even machine learning at the command line... and have a lot of fun.

1. January 21st, 2015
Data Science Consulting
Héloïse Nonne
@heloisenonne,
hnonne@quantmetry.com
Data Science at the Command Line
Paris Data Geek @ AXA
PLEASE DO NOT PRINT THE BLACK SLIDES.
THINK ABOUT THE ENVIRONMENT

2. Data Science at the Command Line
at Paris Data Geek
@ AXA

4. $ Let’s stop clicking
and
start typing
CLI has 40 years and counting
A modern tool!

5. • Agile and interactive
read–eval–print loop (REPL) vs edit-compile-run-debug cycle
• Close to the filesystem
• A good tool for the lazy
Automates repetitive tasks
• Good for integration with other technologies
C/C++, python, perl, R, ruby, etc.
Create your own tools
$ why use the command line?

6. Do we really need
Hadoop to process
a few GB of data?
1.75 GB – 2 million chess games
• Hadoop: 26 minutes (1.14MB/sec)
• Bash, local: 12 seconds (270MB/sec)
Source: Adam Drake, aadrake.com 2014

7. Streaming at the command line
Spout
<
Bolt
|
Sink
>

8. MapReduce at the command line
Word count
Mapper
grep –oE ‘[a-zA-Z]{2,}’
(Shuffle)
& Sort
sort
Reduce
uniq -c

9. $ Lets go parallel

10. We have many jobs to run and 4CPUs
Naive parallelization
GNU parallel
spawns a new process when one finishes
All CPUs remain active

11. $ Data Scientist Toolkit

12. csvcut
csvsort
csvstack
csvjoin
csvstat
gnuplot
lowercase
regex
sed
tr
csvcut
cut
awk
sort
uniq
curl
in2csv
sql2csv
scrape
jq

13. $ Machine learning at
the command line

14. mlpack
$ linear_regression --input_file dataset.csv --test_file predict.csv -v

15. dbacl
Don’t Be Afraid of the Command Line?

16. Online learning with
Vowpal Wabbit

17. Command line is good for
• Starting data project before going on Hadoop, Spark, …
• Data discovery
• Data cleaning
• Do some efficient machine learning (online, C)
• Model / Feature discovery

18. What next?
• Online learning
• Benchmark with bigger data
• Hadoop (Hive) vs CLI
• Benchmark of Machine learning at the CLI
• CLI tools vs Python / R

19. William Shotts
Jeroen Janssens
The man pages!
$ Bibliography

20. $ Questions?