There are plenty of public datasets out there available and the number is growing. Few recent and most useful of BigData ecosystem tools are showcased: Apache Zeppelin (incubating), Apache Spark and Juju.

1. Mining Public Datasets
Using Open Source Tools
At scale & on a budget
by Alexander

2. Software Engineer at NFLabs, Seoul,
South Korea
Co-organizer of SeoulTech Society
Committer and PPMC member of
Apache Zeppelin (Incubating)
@seoul_engineer
github.com/bzz
Alexander

3. IS DATA IMPORTANT?
Content Streaming Services
Taxi
Housing
Web Search
Kuaidi
OpenHouse
IoT

4. CONTEXT
Size of even Public Data is huge and growing
There could be more research, applications and
data products build using that data
Quality and number of free tools available to
public to crunch that data is constantly improving
Cloud brings affordable computations @ scale

5. PUBLIC DATA = OPPORTUNITY

6. AGENDA
Datasets
Tools
Scale and Budget

7. Datasets
Tools
Scale and Budget

8. • Internet archives
• Web applications logs (wikipedia activity, github activity)
• Genome
• AdClicks
• Webserver access logs
• Network traffic
• Scientific datasets
• Images, Songs (Million Song Dataset)
• Reviews,
• Social media
• Flight timetables
• Taxis
• n-gram language model
DATASETS

9. DATASETS
• 300Gb compressed
• Collaboration google and github engineers
• Events on PR, repo, issues, comments in JSON
http://githubarchive.org

10. http://www.commitlogsfromlastnight.com/

11. http://sideeffect.kr/popularconvention/

12. https://www.gitlive.net/

13. http://zoom.it/kCsU

14. Common Crawl
https://commoncrawl.org
Nonprofit, by Factual
On AWS S3 in WARC, WAT, formats
since 2013, monthly: ~150Tb compressed, 2+bln ulrs

15. URL Index by Ilya Kreymer of @webrecorder_io
http://index.commoncrawl.org/

17. https://about.commonsearch.org

18. Datasets
Tools
Scale and Budget

19. TOOLS OVERVIEW
Generic:
Grep, Python, Ruby, JVM - all good, but hard to
scale beyond single machine or data format
Hight-performance:
MPI, Hadoop, HPC - awesome but complex, not
easy, problem specific, not very accessible
New, scalable:
Spark, Flink, Zeppelin - easy (not simple) and
robust

20. Apache Software Foundation
http://www.apache.org/foundation/

21. Apache Software Foundation
1999 - 21 founders of 1 project
2016 - 9 Board of Directors
600 Foundation members
2000+ committers of 171 projects (+55 incubating)
Keywords: meritocracy, community over code, consensus
Provide: infrastructure, legal support, way of building
software
http://www.apache.org/foundation/

22. Apache Spark
Scala, Python, R
Apache Zeppelin
Modern Web GUI, plays nicely with Spark, Flink,
Elasticsearch, etc. Easy to set up.
Warcbase
Spark library for saved crawl data (WARC)
Juju
Scales, integration with Spark, Zeppelin, AWS, Ganglia
NEW, SCALABLE TOOLS

23. APACHE SPARK
From Berkeley AMP Labs, since 2010
Founded Databricks since 2013, joined
Apache since 2014
1000+ contributors
REPL + Java, Scala, Python, R APIs
http://spark.apache.org

24. APACHE SPARK
Has much more: GraphX, MLlib, SQL
https://spark.apache.org/examples.html
http://spark.apache.org
Parallel collections API (similar to FlumeJava, Crunch, Cascading)

25. • Notebook style GUI on top of backend
processing system
• Plays nicely with all the eco-system Spark,
Flink, SQL, Elasticsearch, etc.
• Easy to set up
APACHE ZEPPELIN (INCUBATING)
http://zeppelin.incubator.apache.org

27. Enters ASF Incubation12.2014
08.2013 NFLabs Internal project Hive/Shark
12.2012 Commercial App using AMP Lab Shark 0.5
10.2013 Prototype Hive/Shark
APACHE ZEPPELIN PROJECT TIMELINE
01.2016 3 major releases
http://zeppelin.incubator.apache.org

28. Interactive Visualization
APACHE ZEPPELIN

29. Pluggable Interpreters
APACHE ZEPPELIN

30. Spark library for WARC (Web ARChive) data processing
* text analysis
* site link structure
WARCBASE
https://github.com/lintool/warcbase
http://lintool.github.io/warcbase-docs

31. Service modeling at scale
Deploymentconfiguration automation
+ Integration with Spark, Zeppelin, Ganglia, etc
+ AWS, GCE, Azure, LXC, etc
JUJU
https://jujucharms.com/

32. $ apt-get install juju-core juju-quickstart
# or
$ brew install juju juju-quickstart
$ juju generate-config
#LXC, AWS, GCE, Azure, VMWare, OpenStack
$ juju bootstrap
$ juju quickstart apache-hadoop-spark-zeppelin
$ juju expose spark zeppelin
$ juju add-unit -n4 slave
JUJU
http://bigdata.juju.solutions/getstarted

33. JUJU
http://bigdata.juju.solutions/getstarted
7 node cluster designed to scale out

34. Datasets
Tools
Scale and Budget

35. 1 core
10s PC
1000 instances
APPROACH: SCALE AND BUDGET
Prototype
Estimate the cost
Scale out
Your laptop
AWS spot instances
Deployment automation

36. TAKEAWAY
There are plenty of free tools out there
To crunch the data for fun and profit
They are easy (not simple) to learn and generic enough

37. Thank you

38. Questions?
@seoul_engineer
Alexander
github.com/bzz