1.
Building a Data Product using
apache Zeppelin (incubating)
NFLabs for ApacheCon ’15 EU

2.
Alexander Bezzubov
Data engineer @NFLabs
based in Seoul, South Korea
bzz@apache.org

3.
So you want to build a data
product…
What do you need?

4.
To build a data product we need:
Idea
Data
Software (to process the data)
Hardware (to run the software)
…and a brain

5.
Idea
Data
Software (to process the data)
Hardware (to run the software)
…and a brain
To build a data product we need:

6.
Software

7.
Software

8.
Software
… …

9.
Software
… …

10.
Zeppelin

11.
Zeppelin
Opensouce analytical environment,
with pluggable backend data-processing systems,
with notebook-style GUI for visualisations

12.
ASF Incubation12.2014
08.2013 NFLabs Internal project Hive/Shark
http://zeppelin.incubator.apache.org
12.2012 Commercial App using AMP Lab Shark 0.5
10.2013 Prototype Hive/Shark
Zeppelin History

13.
Zeppelin History

14.
Zeppelin History

15.
Zeppelin History

16.
Zeppelin History

17.
Zeppelin Current Status
1 Release
63 Contributors worldwide
689 Stars on GH
~300/900 Emails at users/dev
@i.a.o
http://zeppelin.incubator.apache.org

18.
Zeppelin Architecture

19.
Hardware

20.
Hardware
.
.
.
.
.
.

21.
Idea

22.
An Idea
Would not it be cool if …

23.
An Idea
Would not it be cool if …Would not it be cool if …
you could have your own Google
Analytics?

24.
An Idea
Would not it be cool if …Would not it be cool if …
you could have your own Google
Analytics?
sorry, we already saw it in eCG
talk..
ok, let’s pick something else

25.
An Idea
you could be the first to know when
there is a new interesting*
opensouce project
Would not it be cool if …

26.
Data

27.
Data: Github archive
https://www.githubarchive.org• Github logs, hosted in the
cloud
• Collaboration between Github
and Google engineers
• 20+ events, 250+Gb since
2012
• Proprietary software
• available on BigQuery

28.
But what if you could:
analyse this data independently, without asking
permission or paying anybody?

29.
But what if you could:
analyse this data independent, without asking
permission or paying anybody?
well, with ASF you can!

30.
Let’s build a data product for ourselves!

31.
Building a product

32.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join logs w/ more data from Github API calls
• Shows simple HTML template, to visualise the
list
• Sends email notifications

33.
We are going to build a Notebook that:
basically, sends you digest emails:

34.
Start Zeppelin
./bin/zeppelin-daemon.sh start
& create a new notebook
http://zeppelin.incubator.apache.org/download.html

35.
Zeppelin Architecture

36.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join logs w/ more data from Github API calls
• Shows simple HTML template, to visualise the
list
• Sends email notifications

37.
Load Dependency

38.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join logs w/ more data from Github API calls
• Shows simple HTML template, to visualise the
list
• Sends email notifications

39.
Download Data
In serial, sample, using shell interpreter

40.
Download Data
In serial, whole day, using shell interpreter
Don’t need this as we have data
prepared

41.
Download Data
In parallel, using Spark interpreter

42.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join on external information through remote API
call
• Shows simple HTML template to visualise the
list
• Sends email notifications

43.
Read Data

44.
Explore Data #1
Pie Chart of Event types

45.
Explore Data #2
Top 10 organisations by event type

46.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join on external information through remote API
call
• Shows simple HTML template to visualise the
list
• Sends email notifications

47.
Filter: cleanup
Only organisations open sourcing
repositories

48.
Filter: interesting companies
Only organisations open sourcing
repositories

49.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join on external information through remote API
• Shows simple HTML template to visualize the
list
• Sends email notifications

50.
Call Github API
Getting more information about
repository
GitHub personal access token to rise rate-limit
github.com/<username> => Edit Profile => Personal access tokens => Generate new
token

51.
Join orgs and repos

52.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join on external information through remote API
call
• Shows simple HTML template
• Sends email notifications

53.
HTML Preview
To generate a template

54.
HTML Preview
To output the results

55.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join on external information through remote API
call
• Shows simple HTML template to visualise the
list
• Sends email notifications

56.
Send Email

57.
Send Email

58.
Send Email

59.
We are going to build a Notebook that:
• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join on external information through remote API
call
• Shows simple HTML template to visualise the
list
• Sends email notifications

60.
Schedule

61.
Kylin: interpreter setup

62.
Kylin: consume data

63.
Alexander Bezzubov
bzz@apache.org
http://s.apache.org/zeppelin-workshop