SlideShare a Scribd company logo

Sparkly Notebook: Interactive Analysis and Visualization with Spark

F
felixcss

Sparkly Notebook: Interactive Analysis and Visualization with Spark - presented Apr 15, 2015 at Seattle Spark Meetup http://www.meetup.com/Seattle-Spark-Meetup/events/208711962/

1 of 63
Download to read offline
SPARKLY NOTEBOOK: INTERACTIVE
ANALYSIS AND VISUALIZATION WITH SPARK
FELIX CHEUNG
APRIL 2015
HTTP://WWW.MEETUP.COM/SEATTLE-SPARK-MEETUP/EVENTS/208711962/
SETUP
• Spark on CDH cluster
• Vagrant - 2-nodes - custom provisioning
AGENDA
• IPython + PySpark cluster
• Zeppelin
• Spark’s Streaming k-means
• Lightning
Sparkly Notebook: Interactive Analysis and Visualization with Spark
SPARK - 10 SEC INTRODUCTION
• Spark
• Spark SQL + Data Frame + data source
• Spark Streaming
• MLlib
• GraphX
It’s a lot of time looking at data..
REPL
• Read-Eval-Print-Loop
Set of REPL related to Spark…
$	
  spark-­‐shell	
  
Welcome	
  to	
  
	
  	
  	
  	
  	
  	
  ____	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  __	
  
	
  	
  	
  	
  	
  /	
  __/__	
  	
  ___	
  _____/	
  /__	
  
	
  	
  	
  	
  _	
  /	
  _	
  /	
  _	
  `/	
  __/	
  	
  '_/	
  
	
  	
  	
  /___/	
  .__/_,_/_/	
  /_/_	
  	
  	
  version	
  1.2.0-­‐SNAPSHOT	
  
	
  	
  	
  	
  	
  	
  /_/	
  
Using	
  Scala	
  version	
  2.10.4	
  (Java	
  HotSpot(TM)	
  64-­‐Bit	
  Server	
  VM,	
  Java	
  1.7.0_67)	
  
Type	
  in	
  expressions	
  to	
  have	
  them	
  evaluated.	
  
Type	
  :help	
  for	
  more	
  information.	
  
15/04/15	
  11:31:28	
  INFO	
  SparkILoop:	
  Created	
  spark	
  context..	
  
Spark	
  context	
  available	
  as	
  sc.	
  
scala>	
  val	
  a	
  =	
  sc.parallelize(1	
  to	
  100)	
  
a:	
  org.apache.spark.rdd.RDD[Int]	
  =	
  ParallelCollectionRDD[0]	
  at	
  parallelize	
  at	
  <console>:12	
  
scala>	
  a.collect.foreach(x	
  =>	
  println(x))	
  
1	
  
2	
  
3	
  
4
GOOD
• See results instantly
NOT SO GOOD
• Ok as an IDE
• No Save / Repeat
• No visualization
NOTEBOOK
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Jupyter
IPython will continue to exist as a Python kernel for Jupyter, but
the notebook and other language-agnostic parts of IPython will
move to new projects under the Jupyter name. IPython 3.0 will
be the last monolithic release of IPython.
!
“IPython” http://ipython.org/
• interactive shell
• browser-based notebook
• 'Kernel'
• great support for visualization library (eg. matplotlib)
• built on pyzmq, tornado
IPYTHON/JUPYTER
IPYTHON NOTEBOOK

NOTEBOOK == BROWSER-BASED REPL
IPython Notebook is a web-based interactive
computational environment for creating IPython
notebooks. An IPython notebook is a JSON
document containing an ordered list of input/output
cells which can contain code, text, mathematics,
plots and rich media.
MATPLOTLIB
matplotlib tries to make easy things easy and hard things
possible. You can generate plots, histograms, power
spectra, bar charts, errorcharts, scatterplots, etc, with just a
few lines of code, with familiar MATLAB APIs.
plt.barh(y_pos,	
  performance,	
  xerr=error,	
  
align='center',	
  alpha=0.4)	
  
plt.yticks(y_pos,	
  people)	
  
plt.xlabel('Performance')	
  
plt.title('How	
  fast	
  do	
  you	
  want	
  to	
  go	
  today?')	
  
plt.show()
PYSPARK
• Spark on Python, this serves as the Kernel,
integrating with IPython
• Each notebook spins up a new instance of the
Kernel (ie. PySpark running as the Spark Driver, in
different deploy mode Spark/PySpark supports)
(All notebook examples are a subset of those in
the Meetup reconstructed here)
Markdown
Spark in
Python
Source: http://nbviewer.ipython.org/github/ResearchComputing/
scientific_computing_tutorials/blob/master/spark/02_word_count.ipynb
Sparkly Notebook: Interactive Analysis and Visualization with Spark
WORD2VEC EXAMPLE
Word2Vec computes distributed vector
representation of words. Distributed vector
representation is showed to be useful in many
natural language processing applications such as
named entity recognition, disambiguation, parsing,
tagging and machine translation.

https://code.google.com/p/word2vec/
Spark MLlib implements the Skip-gram approach.
With Skip-gram we want to predict a window of
words given a single word.
WORD2VEC DATASET
Wikipedia dump http://mattmahoney.net/dc/
textdata
grep	
  -­‐o	
  -­‐E	
  'w+(W+w+){0,15}'	
  text8	
  >	
  text8_lines	
  
then randomly sampled to ~200k lines
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Spark
matplotlib: http://matplotlib.org
Seaborn: http://stanford.edu/~mwaskom/software/seaborn/
Bokeh: http://bokeh.pydata.org/en/latest/
MORE VISUALIZATIONS Seaborn
Bokeh
matplotlib
SETUP
To setup IPython
• Python 2.7.9 (separate from CentOS default 2.6.6), on all
nodes
• matplotlib, on the host running IPython
To run IPython with the PySpark Kernel, set these in the environment

(Please check out my handy script on github)
!
!
!
PYSPARK_PYTHON command to run python, eg. “python2.7”
PYSPARK_DRIVER_PYTHON command to run ipython
PYSPARK_DRIVER_PYTHON_OPTS “notebook —profile”
PYSPARK_SUBMIT_ARGS pyspark commandline, eg. --master --deploy_mode
YARN_CONF_DIR if YARN mode
LD_LIBRARY_PATH for matplotlib
IPYTHON/JUPYTER KERNELS
• IPython
• IGo
• Bash
• IR
• IHaskell
• IMatlab
• ICSharp
• IScala
• IRuby
• IJulia
.. and more https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-
languages
ZEPPELIN
Apache Zeppelin (incubating) is interactive data analytics environment
for distributed data processing system. It provides beautiful interactive
web-based interface, data visualization, collaborative work
environment and many other nice features to make your data analytics
more fun and enjoyable.
Zeppelin has been incubating since Dec 2014.

https://zeppelin.incubator.apache.org/
Sparkly Notebook: Interactive Analysis and Visualization with Spark
shell script &

calling library package
Load and process data

with Spark
SQL query powered by
Spark SQL -

progress &

parameterization via
dynamic form
Python &

data passing across
languages (interpreters)
ZEPPELIN ARCHITECTURE
Realtime collaboration
- enabled by
websocket
communications
Frontend: AngularJS 

Backend server: Java 

Interpreters: Java

Visualization: NVD3
INTERPRETERS
• Spark group
• Spark (Scala)
• PySpark
• Spark SQL
• Dependency
• Markdownjs
• Shell
• Hive
• Coming: jdbc, Tajo, etc.
CLUSTERING
• Clustering tries to find natural groupings in
data. It puts objects into groups in which
those within a group are more similar to each
other than to those in other groups.
• Unsupervised learning
K-MEANS
• First, given an initial set of k cluster centers,
we find which cluster each data point is
closest to
• Then, we compute the average of each of the
new clusters and use the result to update our
cluster centers
Sparkly Notebook: Interactive Analysis and Visualization with Spark
K-MEANS|| IN MLLIB
• a parallelized variant of the k-means++

http://theory.stanford.edu/~sergei/papers/vldb12-kmpar.pdf
Parameters:
• k is the number of desired clusters.
• maxIterations is the maximum number of iterations to run.
• initializationMode specifies either random initialization or initialization via
k-means||.
• runs is the number of times to run the k-means algorithm (k-means is not
guaranteed to find a globally optimal solution, and when run multiple
times on a given dataset, the algorithm returns the best clustering result).
• initializationSteps determines the number of steps in the k-means||
algorithm.
• epsilon determines the distance threshold within which we consider k-
means to have converged.
CASE STUDY:

K-MEANS - ZEPPELIN
Details on github at: http://bit.ly/1JWOPh8
ANOMALY DETECTION WITH K-MEANS
Using Spark DataFrame, csv data source, to process KDDCup’99 data

Scoring with different k values
COMING SOON (NOW!)
Realtime updates
Dashboard
Spark-notebook: https://github.com/andypetrella/spark-notebook
ISpark: https://github.com/tribbloid/ISpark
Spark Kernel: https://github.com/ibm-et/spark-kernel
Jove: https://github.com/jove-sh/jove-notebook
Beaker: https://github.com/twosigma/beaker-notebook
OTHER NOTEBOOKS
• Spark-notebook
• ISpark
• Spark Kernel
• Jove Notebook
• Beaker
• Databricks Cloud notebook
PART 2
STREAMING K-MEANS
WHY STREAMING?
• Train - model - predict works well on static
data
• What if data is
• Coming in streams
• Changing over time?
STREAMING K-MEANS DESIGN
• Proposed by Dr Jeremy Freeman (here)
STREAMING K-MEANS
• key concept: forgetfulness
• balances the relative importance of new
data versus past history
• half-life
• time it takes before past data contributes to
only one half of the current model
STREAMING K-MEANS
• time unit
• batches (which have a fixed duration in
time), or points
• eliminate dying clusters

VISUALIZING

STREAMING K-MEANS - LIGHTNING
LIGHTNING
• Lightning - data visualization server

http://lightning-viz.org
• provides API-based access to reproducible, web-
based, interactive visualizations. It includes a core set
of visualization types, but is built for extendability
and customization. Lightning supports modern
libraries like d3.js and three.js, and is designed for
interactivity over large data sets and continuously
updating data streams.
VISUALIZING STREAMING K-
MEANS ON IPYTHON + LIGHTNING
RUNNING LIGHTNING
• API: node.js, Python, Scala
• Extension support for custom chart (eg. d3.js)
• Requirements:
• Postgres recommended (SQLlite ok)
• node.js (npm , gulp)
The Freeman Lab at Janelia Research Campus uses Lightning to visualize
large-scale neural recordings from zebrafish, in collaboration with the
Ahrens Lab
SPARK STREAMING K-MEANS
DEMO
Environment
• requires: numpy, scipy, scikit-learn
• IPython/Python requires: lightning-python package
Demo consists of 3 parts:

https://github.com/felixcheung/spark-ml-streaming
• Python driver script, data generator
• Scala job - Spark Streaming & Streaming k-means
• IPython notebook to process result, visualize with Lightning

Originally this was part of the Python driver script - it has
been modified for this talk to run within IPython
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Spark
CHALLENGES
• Package management
• Version/build conflicts!
YOU CAN RUN THIS TOO!
• Notebooks available at http://bit.ly/1JWOPh8
• Everything is heavily scripted and automated

Vagrant config for local, virtual environment
available at http://bit.ly/1DB3OLw
QUESTION?
!
https://github.com/felixcheung
linkedin: http://linkd.in/1OeZDb7
blog: http://bit.ly/1E2z6OI
!

Recommended

ビットバンクのデプロイ戦略について
ビットバンクのデプロイ戦略についてビットバンクのデプロイ戦略について
ビットバンクのデプロイ戦略についてbitbank, Inc. Tokyo, Japan
 
Zabbixで学ぶ統計解析入門
Zabbixで学ぶ統計解析入門Zabbixで学ぶ統計解析入門
Zabbixで学ぶ統計解析入門Takeo Noda
 
01 awsとsalesforceを使ったエコシステム ファイル編
01 awsとsalesforceを使ったエコシステム ファイル編01 awsとsalesforceを使ったエコシステム ファイル編
01 awsとsalesforceを使ったエコシステム ファイル編TerraSky
 
メールシステムのおはなし #Mailerstudy
メールシステムのおはなし #Mailerstudyメールシステムのおはなし #Mailerstudy
メールシステムのおはなし #MailerstudyYasuhiro Araki, Ph.D
 
20191016 AWS Black Belt Online Seminar Amazon Route 53 Resolver
20191016 AWS Black Belt Online Seminar Amazon Route 53 Resolver20191016 AWS Black Belt Online Seminar Amazon Route 53 Resolver
20191016 AWS Black Belt Online Seminar Amazon Route 53 ResolverAmazon Web Services Japan
 
AWSのログ管理ベストプラクティス
AWSのログ管理ベストプラクティスAWSのログ管理ベストプラクティス
AWSのログ管理ベストプラクティスAkihiro Kuwano
 

More Related Content

What's hot

The Deep Web, TOR Network and Internet Anonymity
The Deep Web, TOR Network and Internet AnonymityThe Deep Web, TOR Network and Internet Anonymity
The Deep Web, TOR Network and Internet AnonymityAbhimanyu Singh
 
Hyper v ネットワークの基本
Hyper v ネットワークの基本Hyper v ネットワークの基本
Hyper v ネットワークの基本Syuichi Murashima
 
Grafana LokiではじめるKubernetesロギングハンズオン(NTT Tech Conference #4 ハンズオン資料)
Grafana LokiではじめるKubernetesロギングハンズオン(NTT Tech Conference #4 ハンズオン資料)Grafana LokiではじめるKubernetesロギングハンズオン(NTT Tech Conference #4 ハンズオン資料)
Grafana LokiではじめるKubernetesロギングハンズオン(NTT Tech Conference #4 ハンズオン資料)NTT DATA Technology & Innovation
 
インターネットの仕組みとISPの構造
インターネットの仕組みとISPの構造インターネットの仕組みとISPの構造
インターネットの仕組みとISPの構造Taiji Tsuchiya
 
Prometheus Operator 入門(Kubernetes Novice Tokyo #26 発表資料)
Prometheus Operator 入門(Kubernetes Novice Tokyo #26 発表資料)Prometheus Operator 入門(Kubernetes Novice Tokyo #26 発表資料)
Prometheus Operator 入門(Kubernetes Novice Tokyo #26 発表資料)NTT DATA Technology & Innovation
 
OpenStackをさらに”使う”技術 概要と基礎操作
OpenStackをさらに”使う”技術 概要と基礎操作OpenStackをさらに”使う”技術 概要と基礎操作
OpenStackをさらに”使う”技術 概要と基礎操作irix_jp
 
Bluetooth LEとiBeaconを使った、すれ違い通信
Bluetooth LEとiBeaconを使った、すれ違い通信Bluetooth LEとiBeaconを使った、すれ違い通信
Bluetooth LEとiBeaconを使った、すれ違い通信幸雄 村上
 
AvailabilityZoneとHostAggregate
AvailabilityZoneとHostAggregateAvailabilityZoneとHostAggregate
AvailabilityZoneとHostAggregateHiroki Ishikawa
 
TTN V2からTTN V3への移行ハンズオン by TTN高崎イニシエーター/エルスピーナヴェインズ株式会社代表取締役青谷浩二様 
TTN V2からTTN V3への移行ハンズオン  by TTN高崎イニシエーター/エルスピーナヴェインズ株式会社代表取締役青谷浩二様 TTN V2からTTN V3への移行ハンズオン  by TTN高崎イニシエーター/エルスピーナヴェインズ株式会社代表取締役青谷浩二様 
TTN V2からTTN V3への移行ハンズオン by TTN高崎イニシエーター/エルスピーナヴェインズ株式会社代表取締役青谷浩二様 CRI Japan, Inc.
 
Fantiaから学ぶgcp運用のノウハウ
Fantiaから学ぶgcp運用のノウハウFantiaから学ぶgcp運用のノウハウ
Fantiaから学ぶgcp運用のノウハウ虎の穴 開発室
 
「これ危ない設定じゃないでしょうか」とヒアリングするための仕組み @AWS Summit Tokyo 2018
「これ危ない設定じゃないでしょうか」とヒアリングするための仕組み @AWS Summit Tokyo 2018「これ危ない設定じゃないでしょうか」とヒアリングするための仕組み @AWS Summit Tokyo 2018
「これ危ない設定じゃないでしょうか」とヒアリングするための仕組み @AWS Summit Tokyo 2018cyberagent
 
MariaDBとMroongaで作る全言語対応超高速全文検索システム
MariaDBとMroongaで作る全言語対応超高速全文検索システムMariaDBとMroongaで作る全言語対応超高速全文検索システム
MariaDBとMroongaで作る全言語対応超高速全文検索システムKouhei Sutou
 
分離されたネットワークでの複合機/プリンターの共有
分離されたネットワークでの複合機/プリンターの共有分離されたネットワークでの複合機/プリンターの共有
分離されたネットワークでの複合機/プリンターの共有Hiroshi Ota
 
Ansible 2.8 アップデート情報 -機能追加と注意点-
Ansible 2.8 アップデート情報 -機能追加と注意点-Ansible 2.8 アップデート情報 -機能追加と注意点-
Ansible 2.8 アップデート情報 -機能追加と注意点-akira6592
 
0から始めるコンテナの学び方(Kubernetes Novice Tokyo #14 発表資料)
0から始めるコンテナの学び方(Kubernetes Novice Tokyo #14 発表資料)0から始めるコンテナの学び方(Kubernetes Novice Tokyo #14 発表資料)
0から始めるコンテナの学び方(Kubernetes Novice Tokyo #14 発表資料)NTT DATA Technology & Innovation
 

What's hot (20)

Chapter 7
Chapter 7Chapter 7
Chapter 7
 
The Deep Web, TOR Network and Internet Anonymity
The Deep Web, TOR Network and Internet AnonymityThe Deep Web, TOR Network and Internet Anonymity
The Deep Web, TOR Network and Internet Anonymity
 
Hyper v ネットワークの基本
Hyper v ネットワークの基本Hyper v ネットワークの基本
Hyper v ネットワークの基本
 
Grafana LokiではじめるKubernetesロギングハンズオン(NTT Tech Conference #4 ハンズオン資料)
Grafana LokiではじめるKubernetesロギングハンズオン(NTT Tech Conference #4 ハンズオン資料)Grafana LokiではじめるKubernetesロギングハンズオン(NTT Tech Conference #4 ハンズオン資料)
Grafana LokiではじめるKubernetesロギングハンズオン(NTT Tech Conference #4 ハンズオン資料)
 
インターネットの仕組みとISPの構造
インターネットの仕組みとISPの構造インターネットの仕組みとISPの構造
インターネットの仕組みとISPの構造
 
AWS IoT Greengrass V2 の紹介
AWS IoT Greengrass V2 の紹介AWS IoT Greengrass V2 の紹介
AWS IoT Greengrass V2 の紹介
 
Prometheus Operator 入門(Kubernetes Novice Tokyo #26 発表資料)
Prometheus Operator 入門(Kubernetes Novice Tokyo #26 発表資料)Prometheus Operator 入門(Kubernetes Novice Tokyo #26 発表資料)
Prometheus Operator 入門(Kubernetes Novice Tokyo #26 発表資料)
 
自宅vSphereからニフクラに引っ越ししてみた
自宅vSphereからニフクラに引っ越ししてみた自宅vSphereからニフクラに引っ越ししてみた
自宅vSphereからニフクラに引っ越ししてみた
 
OpenStackをさらに”使う”技術 概要と基礎操作
OpenStackをさらに”使う”技術 概要と基礎操作OpenStackをさらに”使う”技術 概要と基礎操作
OpenStackをさらに”使う”技術 概要と基礎操作
 
Bluetooth LEとiBeaconを使った、すれ違い通信
Bluetooth LEとiBeaconを使った、すれ違い通信Bluetooth LEとiBeaconを使った、すれ違い通信
Bluetooth LEとiBeaconを使った、すれ違い通信
 
L2延伸を利用したクラウド移行とクラウド活用術
L2延伸を利用したクラウド移行とクラウド活用術L2延伸を利用したクラウド移行とクラウド活用術
L2延伸を利用したクラウド移行とクラウド活用術
 
AvailabilityZoneとHostAggregate
AvailabilityZoneとHostAggregateAvailabilityZoneとHostAggregate
AvailabilityZoneとHostAggregate
 
TTN V2からTTN V3への移行ハンズオン by TTN高崎イニシエーター/エルスピーナヴェインズ株式会社代表取締役青谷浩二様 
TTN V2からTTN V3への移行ハンズオン  by TTN高崎イニシエーター/エルスピーナヴェインズ株式会社代表取締役青谷浩二様 TTN V2からTTN V3への移行ハンズオン  by TTN高崎イニシエーター/エルスピーナヴェインズ株式会社代表取締役青谷浩二様 
TTN V2からTTN V3への移行ハンズオン by TTN高崎イニシエーター/エルスピーナヴェインズ株式会社代表取締役青谷浩二様 
 
Fantiaから学ぶgcp運用のノウハウ
Fantiaから学ぶgcp運用のノウハウFantiaから学ぶgcp運用のノウハウ
Fantiaから学ぶgcp運用のノウハウ
 
Deep Web & Dark Web
Deep Web & Dark WebDeep Web & Dark Web
Deep Web & Dark Web
 
「これ危ない設定じゃないでしょうか」とヒアリングするための仕組み @AWS Summit Tokyo 2018
「これ危ない設定じゃないでしょうか」とヒアリングするための仕組み @AWS Summit Tokyo 2018「これ危ない設定じゃないでしょうか」とヒアリングするための仕組み @AWS Summit Tokyo 2018
「これ危ない設定じゃないでしょうか」とヒアリングするための仕組み @AWS Summit Tokyo 2018
 
MariaDBとMroongaで作る全言語対応超高速全文検索システム
MariaDBとMroongaで作る全言語対応超高速全文検索システムMariaDBとMroongaで作る全言語対応超高速全文検索システム
MariaDBとMroongaで作る全言語対応超高速全文検索システム
 
分離されたネットワークでの複合機/プリンターの共有
分離されたネットワークでの複合機/プリンターの共有分離されたネットワークでの複合機/プリンターの共有
分離されたネットワークでの複合機/プリンターの共有
 
Ansible 2.8 アップデート情報 -機能追加と注意点-
Ansible 2.8 アップデート情報 -機能追加と注意点-Ansible 2.8 アップデート情報 -機能追加と注意点-
Ansible 2.8 アップデート情報 -機能追加と注意点-
 
0から始めるコンテナの学び方(Kubernetes Novice Tokyo #14 発表資料)
0から始めるコンテナの学び方(Kubernetes Novice Tokyo #14 発表資料)0から始めるコンテナの学び方(Kubernetes Novice Tokyo #14 発表資料)
0から始めるコンテナの学び方(Kubernetes Novice Tokyo #14 発表資料)
 

Viewers also liked

Principles of Data Visualization
Principles of Data VisualizationPrinciples of Data Visualization
Principles of Data VisualizationEamonn Maguire
 
Big Data Real Time Analytics - A Facebook Case Study
Big Data Real Time Analytics - A Facebook Case StudyBig Data Real Time Analytics - A Facebook Case Study
Big Data Real Time Analytics - A Facebook Case StudyNati Shalom
 
Data Visualization - What can you see? #baai17
Data Visualization - What can you see? #baai17Data Visualization - What can you see? #baai17
Data Visualization - What can you see? #baai17Eugene O'Loughlin
 
Data Science with Spark & Zeppelin
Data Science with Spark & ZeppelinData Science with Spark & Zeppelin
Data Science with Spark & ZeppelinVinay Shukla
 
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...Spark Summit
 
Manual de programacion_con_robots_para_la_escuela
Manual de programacion_con_robots_para_la_escuelaManual de programacion_con_robots_para_la_escuela
Manual de programacion_con_robots_para_la_escuelaAngel De las Heras
 
Introduction to Streaming Distributed Processing with Storm
Introduction to Streaming Distributed Processing with StormIntroduction to Streaming Distributed Processing with Storm
Introduction to Streaming Distributed Processing with StormBrandon O'Brien
 
Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...
Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...
Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...Brandon O'Brien
 
Apache Zeppelin으로 데이터 분석하기
Apache Zeppelin으로 데이터 분석하기Apache Zeppelin으로 데이터 분석하기
Apache Zeppelin으로 데이터 분석하기SangWoo Kim
 
Real time data viz with Spark Streaming, Kafka and D3.js
Real time data viz with Spark Streaming, Kafka and D3.jsReal time data viz with Spark Streaming, Kafka and D3.js
Real time data viz with Spark Streaming, Kafka and D3.jsBen Laird
 
Big Data visualization with Apache Spark and Zeppelin
Big Data visualization with Apache Spark and ZeppelinBig Data visualization with Apache Spark and Zeppelin
Big Data visualization with Apache Spark and Zeppelinprajods
 
Brief introduction to data visualization
Brief introduction to data visualizationBrief introduction to data visualization
Brief introduction to data visualizationZach Gemignani
 

Viewers also liked (13)

Principles of Data Visualization
Principles of Data VisualizationPrinciples of Data Visualization
Principles of Data Visualization
 
Data Visualization Tools
Data Visualization ToolsData Visualization Tools
Data Visualization Tools
 
Big Data Real Time Analytics - A Facebook Case Study
Big Data Real Time Analytics - A Facebook Case StudyBig Data Real Time Analytics - A Facebook Case Study
Big Data Real Time Analytics - A Facebook Case Study
 
Data Visualization - What can you see? #baai17
Data Visualization - What can you see? #baai17Data Visualization - What can you see? #baai17
Data Visualization - What can you see? #baai17
 
Data Science with Spark & Zeppelin
Data Science with Spark & ZeppelinData Science with Spark & Zeppelin
Data Science with Spark & Zeppelin
 
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
 
Manual de programacion_con_robots_para_la_escuela
Manual de programacion_con_robots_para_la_escuelaManual de programacion_con_robots_para_la_escuela
Manual de programacion_con_robots_para_la_escuela
 
Introduction to Streaming Distributed Processing with Storm
Introduction to Streaming Distributed Processing with StormIntroduction to Streaming Distributed Processing with Storm
Introduction to Streaming Distributed Processing with Storm
 
Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...
Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...
Real Time Data Processing With Spark Streaming, Node.js and Redis with Visual...
 
Apache Zeppelin으로 데이터 분석하기
Apache Zeppelin으로 데이터 분석하기Apache Zeppelin으로 데이터 분석하기
Apache Zeppelin으로 데이터 분석하기
 
Real time data viz with Spark Streaming, Kafka and D3.js
Real time data viz with Spark Streaming, Kafka and D3.jsReal time data viz with Spark Streaming, Kafka and D3.js
Real time data viz with Spark Streaming, Kafka and D3.js
 
Big Data visualization with Apache Spark and Zeppelin
Big Data visualization with Apache Spark and ZeppelinBig Data visualization with Apache Spark and Zeppelin
Big Data visualization with Apache Spark and Zeppelin
 
Brief introduction to data visualization
Brief introduction to data visualizationBrief introduction to data visualization
Brief introduction to data visualization
 

Similar to Sparkly Notebook: Interactive Analysis and Visualization with Spark

Apache Spark Tutorial
Apache Spark TutorialApache Spark Tutorial
Apache Spark TutorialAhmet Bulut
 
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...Spark Summit
 
Microservices Application Tracing Standards and Simulators - Adrians at OSCON
Microservices Application Tracing Standards and Simulators - Adrians at OSCONMicroservices Application Tracing Standards and Simulators - Adrians at OSCON
Microservices Application Tracing Standards and Simulators - Adrians at OSCONAdrian Cockcroft
 
Real-time Streaming Pipelines with FLaNK
Real-time Streaming Pipelines with FLaNKReal-time Streaming Pipelines with FLaNK
Real-time Streaming Pipelines with FLaNKData Con LA
 
Apache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupApache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupNed Shawa
 
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...Spark Summit
 
Scaling PyData Up and Out
Scaling PyData Up and OutScaling PyData Up and Out
Scaling PyData Up and OutTravis Oliphant
 
Dissecting Open Source Cloud Evolution: An OpenStack Case Study
Dissecting Open Source Cloud Evolution: An OpenStack Case StudyDissecting Open Source Cloud Evolution: An OpenStack Case Study
Dissecting Open Source Cloud Evolution: An OpenStack Case StudySalman Baset
 
Building analytical microservices powered by jupyter kernels
Building analytical microservices powered by jupyter kernelsBuilding analytical microservices powered by jupyter kernels
Building analytical microservices powered by jupyter kernelsLuciano Resende
 
2018 03 25 system ml ai and openpower meetup
2018 03 25 system ml ai and openpower meetup2018 03 25 system ml ai and openpower meetup
2018 03 25 system ml ai and openpower meetupGanesan Narayanasamy
 
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...Jason Dai
 
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraReal-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraJoe Stein
 
Yet another intro to Apache Spark
Yet another intro to Apache SparkYet another intro to Apache Spark
Yet another intro to Apache SparkSimon Lia-Jonassen
 
Running Emerging AI Applications on Big Data Platforms with Ray On Apache Spark
Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkRunning Emerging AI Applications on Big Data Platforms with Ray On Apache Spark
Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkDatabricks
 
HOW TO SCALE FROM ZERO TO BILLIONS!
HOW TO SCALE FROM ZERO TO BILLIONS!HOW TO SCALE FROM ZERO TO BILLIONS!
HOW TO SCALE FROM ZERO TO BILLIONS!Maziyar PANAHI
 
BDX 2015 - Scaling out big-data computation & machine learning using Pig, Pyt...
BDX 2015 - Scaling out big-data computation & machine learning using Pig, Pyt...BDX 2015 - Scaling out big-data computation & machine learning using Pig, Pyt...
BDX 2015 - Scaling out big-data computation & machine learning using Pig, Pyt...Ron Reiter
 

Similar to Sparkly Notebook: Interactive Analysis and Visualization with Spark (20)

Apache Spark Tutorial
Apache Spark TutorialApache Spark Tutorial
Apache Spark Tutorial
 
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
 
Microservices Application Tracing Standards and Simulators - Adrians at OSCON
Microservices Application Tracing Standards and Simulators - Adrians at OSCONMicroservices Application Tracing Standards and Simulators - Adrians at OSCON
Microservices Application Tracing Standards and Simulators - Adrians at OSCON
 
Real-time Streaming Pipelines with FLaNK
Real-time Streaming Pipelines with FLaNKReal-time Streaming Pipelines with FLaNK
Real-time Streaming Pipelines with FLaNK
 
Apache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupApache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetup
 
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...
 
Scaling PyData Up and Out
Scaling PyData Up and OutScaling PyData Up and Out
Scaling PyData Up and Out
 
Dissecting Open Source Cloud Evolution: An OpenStack Case Study
Dissecting Open Source Cloud Evolution: An OpenStack Case StudyDissecting Open Source Cloud Evolution: An OpenStack Case Study
Dissecting Open Source Cloud Evolution: An OpenStack Case Study
 
Building analytical microservices powered by jupyter kernels
Building analytical microservices powered by jupyter kernelsBuilding analytical microservices powered by jupyter kernels
Building analytical microservices powered by jupyter kernels
 
2018 03 25 system ml ai and openpower meetup
2018 03 25 system ml ai and openpower meetup2018 03 25 system ml ai and openpower meetup
2018 03 25 system ml ai and openpower meetup
 
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
 
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraReal-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
 
PyData Boston 2013
PyData Boston 2013PyData Boston 2013
PyData Boston 2013
 
04 open source_tools
04 open source_tools04 open source_tools
04 open source_tools
 
Typesafe spark- Zalando meetup
Typesafe spark- Zalando meetupTypesafe spark- Zalando meetup
Typesafe spark- Zalando meetup
 
20160908 hivemall meetup
20160908 hivemall meetup20160908 hivemall meetup
20160908 hivemall meetup
 
Yet another intro to Apache Spark
Yet another intro to Apache SparkYet another intro to Apache Spark
Yet another intro to Apache Spark
 
Running Emerging AI Applications on Big Data Platforms with Ray On Apache Spark
Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkRunning Emerging AI Applications on Big Data Platforms with Ray On Apache Spark
Running Emerging AI Applications on Big Data Platforms with Ray On Apache Spark
 
HOW TO SCALE FROM ZERO TO BILLIONS!
HOW TO SCALE FROM ZERO TO BILLIONS!HOW TO SCALE FROM ZERO TO BILLIONS!
HOW TO SCALE FROM ZERO TO BILLIONS!
 
BDX 2015 - Scaling out big-data computation & machine learning using Pig, Pyt...
BDX 2015 - Scaling out big-data computation & machine learning using Pig, Pyt...BDX 2015 - Scaling out big-data computation & machine learning using Pig, Pyt...
BDX 2015 - Scaling out big-data computation & machine learning using Pig, Pyt...
 

Recently uploaded

Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023maker Money
 
Ratio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxRatio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxSugumarVenkai
 
introduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfintroduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfSalamaAdel
 
presentation big data analytics on Apache spark
presentation big data analytics on Apache sparkpresentation big data analytics on Apache spark
presentation big data analytics on Apache sparkVarun Garg
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdfAlexia Trejo
 
Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Sarma Pydipally
 
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFEXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFProject Cubicle
 
AWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyAWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyVarun Garg
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...ThinkInnovation
 
HayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughsNikolas Markou
 
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Samuel Chukwuma
 
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxWOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxyosra Saidani
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelTope Osanyintuyi
 
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEarley Information Science
 
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
Customer Satisfaction Data -  Multiple Linear Regression Model.pdfCustomer Satisfaction Data -  Multiple Linear Regression Model.pdf
Customer Satisfaction Data - Multiple Linear Regression Model.pdfruwanp2000
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...DrSumathyV
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............mahetamanav24
 

Recently uploaded (18)

Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023
 
Ratio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxRatio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptx
 
introduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfintroduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdf
 
presentation big data analytics on Apache spark
presentation big data analytics on Apache sparkpresentation big data analytics on Apache spark
presentation big data analytics on Apache spark
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdf
 
Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5
 
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFEXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
 
AWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyAWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarify
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...
 
HayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docx
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
 
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
 
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxWOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft Excel
 
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
 
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
Customer Satisfaction Data -  Multiple Linear Regression Model.pdfCustomer Satisfaction Data -  Multiple Linear Regression Model.pdf
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............
 

Sparkly Notebook: Interactive Analysis and Visualization with Spark

  • 1. SPARKLY NOTEBOOK: INTERACTIVE ANALYSIS AND VISUALIZATION WITH SPARK FELIX CHEUNG APRIL 2015 HTTP://WWW.MEETUP.COM/SEATTLE-SPARK-MEETUP/EVENTS/208711962/
  • 2. SETUP • Spark on CDH cluster • Vagrant - 2-nodes - custom provisioning
  • 3. AGENDA • IPython + PySpark cluster • Zeppelin • Spark’s Streaming k-means • Lightning
  • 5. SPARK - 10 SEC INTRODUCTION • Spark • Spark SQL + Data Frame + data source • Spark Streaming • MLlib • GraphX
  • 6. It’s a lot of time looking at data..
  • 8. Set of REPL related to Spark…
  • 9. $  spark-­‐shell   Welcome  to              ____                            __            /  __/__    ___  _____/  /__          _  /  _  /  _  `/  __/    '_/        /___/  .__/_,_/_/  /_/_      version  1.2.0-­‐SNAPSHOT              /_/   Using  Scala  version  2.10.4  (Java  HotSpot(TM)  64-­‐Bit  Server  VM,  Java  1.7.0_67)   Type  in  expressions  to  have  them  evaluated.   Type  :help  for  more  information.   15/04/15  11:31:28  INFO  SparkILoop:  Created  spark  context..   Spark  context  available  as  sc.   scala>  val  a  =  sc.parallelize(1  to  100)   a:  org.apache.spark.rdd.RDD[Int]  =  ParallelCollectionRDD[0]  at  parallelize  at  <console>:12   scala>  a.collect.foreach(x  =>  println(x))   1   2   3   4
  • 10. GOOD • See results instantly
  • 11. NOT SO GOOD • Ok as an IDE • No Save / Repeat • No visualization
  • 14. Jupyter IPython will continue to exist as a Python kernel for Jupyter, but the notebook and other language-agnostic parts of IPython will move to new projects under the Jupyter name. IPython 3.0 will be the last monolithic release of IPython. ! “IPython” http://ipython.org/ • interactive shell • browser-based notebook • 'Kernel' • great support for visualization library (eg. matplotlib) • built on pyzmq, tornado IPYTHON/JUPYTER
  • 15. IPYTHON NOTEBOOK
 NOTEBOOK == BROWSER-BASED REPL IPython Notebook is a web-based interactive computational environment for creating IPython notebooks. An IPython notebook is a JSON document containing an ordered list of input/output cells which can contain code, text, mathematics, plots and rich media.
  • 16. MATPLOTLIB matplotlib tries to make easy things easy and hard things possible. You can generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc, with just a few lines of code, with familiar MATLAB APIs. plt.barh(y_pos,  performance,  xerr=error,   align='center',  alpha=0.4)   plt.yticks(y_pos,  people)   plt.xlabel('Performance')   plt.title('How  fast  do  you  want  to  go  today?')   plt.show()
  • 17. PYSPARK • Spark on Python, this serves as the Kernel, integrating with IPython • Each notebook spins up a new instance of the Kernel (ie. PySpark running as the Spark Driver, in different deploy mode Spark/PySpark supports)
  • 18. (All notebook examples are a subset of those in the Meetup reconstructed here)
  • 23. WORD2VEC EXAMPLE Word2Vec computes distributed vector representation of words. Distributed vector representation is showed to be useful in many natural language processing applications such as named entity recognition, disambiguation, parsing, tagging and machine translation.
 https://code.google.com/p/word2vec/ Spark MLlib implements the Skip-gram approach. With Skip-gram we want to predict a window of words given a single word.
  • 24. WORD2VEC DATASET Wikipedia dump http://mattmahoney.net/dc/ textdata grep  -­‐o  -­‐E  'w+(W+w+){0,15}'  text8  >  text8_lines   then randomly sampled to ~200k lines
  • 27. matplotlib: http://matplotlib.org Seaborn: http://stanford.edu/~mwaskom/software/seaborn/ Bokeh: http://bokeh.pydata.org/en/latest/ MORE VISUALIZATIONS Seaborn Bokeh matplotlib
  • 28. SETUP To setup IPython • Python 2.7.9 (separate from CentOS default 2.6.6), on all nodes • matplotlib, on the host running IPython To run IPython with the PySpark Kernel, set these in the environment
 (Please check out my handy script on github) ! ! ! PYSPARK_PYTHON command to run python, eg. “python2.7” PYSPARK_DRIVER_PYTHON command to run ipython PYSPARK_DRIVER_PYTHON_OPTS “notebook —profile” PYSPARK_SUBMIT_ARGS pyspark commandline, eg. --master --deploy_mode YARN_CONF_DIR if YARN mode LD_LIBRARY_PATH for matplotlib
  • 29. IPYTHON/JUPYTER KERNELS • IPython • IGo • Bash • IR • IHaskell • IMatlab • ICSharp • IScala • IRuby • IJulia .. and more https://github.com/ipython/ipython/wiki/IPython-kernels-for-other- languages
  • 31. Apache Zeppelin (incubating) is interactive data analytics environment for distributed data processing system. It provides beautiful interactive web-based interface, data visualization, collaborative work environment and many other nice features to make your data analytics more fun and enjoyable. Zeppelin has been incubating since Dec 2014.
 https://zeppelin.incubator.apache.org/
  • 33. shell script &
 calling library package Load and process data
 with Spark
  • 34. SQL query powered by Spark SQL -
 progress &
 parameterization via dynamic form
  • 35. Python &
 data passing across languages (interpreters)
  • 36. ZEPPELIN ARCHITECTURE Realtime collaboration - enabled by websocket communications Frontend: AngularJS 
 Backend server: Java 
 Interpreters: Java
 Visualization: NVD3
  • 37. INTERPRETERS • Spark group • Spark (Scala) • PySpark • Spark SQL • Dependency • Markdownjs • Shell • Hive • Coming: jdbc, Tajo, etc.
  • 38. CLUSTERING • Clustering tries to find natural groupings in data. It puts objects into groups in which those within a group are more similar to each other than to those in other groups. • Unsupervised learning
  • 39. K-MEANS • First, given an initial set of k cluster centers, we find which cluster each data point is closest to • Then, we compute the average of each of the new clusters and use the result to update our cluster centers
  • 41. K-MEANS|| IN MLLIB • a parallelized variant of the k-means++
 http://theory.stanford.edu/~sergei/papers/vldb12-kmpar.pdf Parameters: • k is the number of desired clusters. • maxIterations is the maximum number of iterations to run. • initializationMode specifies either random initialization or initialization via k-means||. • runs is the number of times to run the k-means algorithm (k-means is not guaranteed to find a globally optimal solution, and when run multiple times on a given dataset, the algorithm returns the best clustering result). • initializationSteps determines the number of steps in the k-means|| algorithm. • epsilon determines the distance threshold within which we consider k- means to have converged.
  • 43. Details on github at: http://bit.ly/1JWOPh8 ANOMALY DETECTION WITH K-MEANS Using Spark DataFrame, csv data source, to process KDDCup’99 data
 Scoring with different k values
  • 47. Spark-notebook: https://github.com/andypetrella/spark-notebook ISpark: https://github.com/tribbloid/ISpark Spark Kernel: https://github.com/ibm-et/spark-kernel Jove: https://github.com/jove-sh/jove-notebook Beaker: https://github.com/twosigma/beaker-notebook OTHER NOTEBOOKS • Spark-notebook • ISpark • Spark Kernel • Jove Notebook • Beaker • Databricks Cloud notebook
  • 49. WHY STREAMING? • Train - model - predict works well on static data • What if data is • Coming in streams • Changing over time?
  • 50. STREAMING K-MEANS DESIGN • Proposed by Dr Jeremy Freeman (here)
  • 51. STREAMING K-MEANS • key concept: forgetfulness • balances the relative importance of new data versus past history • half-life • time it takes before past data contributes to only one half of the current model
  • 52. STREAMING K-MEANS • time unit • batches (which have a fixed duration in time), or points • eliminate dying clusters

  • 55. • Lightning - data visualization server
 http://lightning-viz.org • provides API-based access to reproducible, web- based, interactive visualizations. It includes a core set of visualization types, but is built for extendability and customization. Lightning supports modern libraries like d3.js and three.js, and is designed for interactivity over large data sets and continuously updating data streams. VISUALIZING STREAMING K- MEANS ON IPYTHON + LIGHTNING
  • 56. RUNNING LIGHTNING • API: node.js, Python, Scala • Extension support for custom chart (eg. d3.js) • Requirements: • Postgres recommended (SQLlite ok) • node.js (npm , gulp)
  • 57. The Freeman Lab at Janelia Research Campus uses Lightning to visualize large-scale neural recordings from zebrafish, in collaboration with the Ahrens Lab
  • 58. SPARK STREAMING K-MEANS DEMO Environment • requires: numpy, scipy, scikit-learn • IPython/Python requires: lightning-python package Demo consists of 3 parts:
 https://github.com/felixcheung/spark-ml-streaming • Python driver script, data generator • Scala job - Spark Streaming & Streaming k-means • IPython notebook to process result, visualize with Lightning
 Originally this was part of the Python driver script - it has been modified for this talk to run within IPython
  • 61. CHALLENGES • Package management • Version/build conflicts!
  • 62. YOU CAN RUN THIS TOO! • Notebooks available at http://bit.ly/1JWOPh8 • Everything is heavily scripted and automated
 Vagrant config for local, virtual environment available at http://bit.ly/1DB3OLw