Michal Malohlava talks about the PySparkling Water package for Spark and Python users. - Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai - To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata

1. H2O PySparkling
Water
Michal Malohlava
@mmalohlava and @h2oai
presents
2016/10/08 PyData

2. H2O.ai
Machine Intelligence
H2O
+
PySpark
=
PySparkling

3. H2O.ai
Machine Intelligence
H2O
Open-Source In-Memory Data Science
Platform
• Highly optimized Java code (in-house)
• Distributed in-memory K-V store and map/
reduce computation framework
• Data parser (HDFS, S3, NFS, HTTP, local
drives, etc.)
• Read/write access to distributed data
frames (R/Pandas-style)
• ML algos - Deep Learning, GBM, DRF, GLM,
GLRM, K-Means, PCA, CoxPH, Ensembles
• REST API: clients Interactive UI/R/Python
H2O Python
client
pySparkling

4. H2O.ai
Machine Intelligence
PySparkling
Provides
Transparent integration of H2O machine learning
platform with Spark ecosystem (PySpark)
Transparent use of H2O data structures (H2OFrame)
and algorithms with Spark Python API
Excels in existing Spark workflows requiring
advanced Machine Learning algorithms
FunctionalitymissinginH2Ocanbe
replacedbySparkandviceversa

5. H2O.ai
Machine Intelligence
Benefits
• Additional algorithms
• NLP features
• Powerful data munging
• ML Pipelines
• Advanced algorithms
• speed v. accuracy
• advanced parameters
• Fully distributed and
parallelized
• Graphical environment
• Fully fledged Python/R
interfaces

6. H2O.ai
Machine Intelligence
PySparkling
Use-Cases

7. H2O.ai
Machine Intelligence
Model Building
Data
Source

8. H2O.ai
Machine Intelligence
Model Building
Data
Source
Data munging

9. H2O.ai
Machine Intelligence
Model Building
Data
Source
Data munging Modelling
Deep Learning, GBM
DRF, GLM, GLRM
K-Means, PCA
CoxPH, Ensembles

10. H2O.ai
Machine Intelligence
Model Building
Data
Source
Data munging Modelling
Deep Learning, GBM
DRF, GLM, GLRM
K-Means, PCA
CoxPH, Ensembles
Prediction
processing

11. H2O.ai
Machine Intelligence
Model Building
Data
Source
Data munging Modelling
Deep Learning, GBM
DRF, GLM, GLRM
K-Means, PCA
CoxPH, Ensembles
Prediction
processing
Steam
Model
management

12. H2O.ai
Machine Intelligence
Data Munging
Data
Source

13. H2O.ai
Machine Intelligence
Data Munging
Data
Source

14. H2O.ai
Machine Intelligence
Data Munging
Data
Source

15. H2O.ai
Machine Intelligence
Data Munging
Data
Source
Data load/munging/
exploration (H2O Flow UI)

16. H2O.ai
Machine Intelligence
Data Munging
Data
Source
Data load/munging/
exploration (H2O Flow UI)
Modelling

17. H2O.ai
Machine Intelligence
Stream processing

18. H2O.ai
Machine Intelligence
Stream processing
Data
Source
Off-line
model
training

19. H2O.ai
Machine Intelligence
Stream processing
Data
Source
Off-line
model
training
Data munging

20. H2O.ai
Machine Intelligence
Stream processing
Data
Source
Off-line
model
training
Data munging Modelling

21. H2O.ai
Machine Intelligence
Stream processing
Data
Source
Off-line
model
training
Data munging
Stream
processing
Data
Stream
Spark Streaming/Storm/Flink
Modelling

22. H2O.ai
Machine Intelligence
Stream processing
Data
Source
Off-line
model
training
Data munging
Stream
processing
Data
Stream
Spark Streaming/Storm/Flink
Export model
Modelling

23. H2O.ai
Machine Intelligence
Stream processing
Data
Source
Off-line
model
training
Data munging
Stream
processing
Data
Stream
Spark Streaming/Storm/Flink
Model
prediction
Deploy
the model
Export model
Modelling

24. H2O.ai
Machine Intelligence
Stream processing
Data
Source
Off-line
model
training
Data munging
Stream
processing
Data
Stream
Spark Streaming/Storm/Flink
Model
prediction
Deploy
the model
Export model
Modelling
Steam
Model
management

25. H2O.ai
Machine Intelligence
What is
inside?

26. H2O.ai
Machine Intelligence
Cluster
Worker node
PySpark main
program
Driver node Worker nodeWorker node

27. H2O.ai
Machine Intelligence
Cluster
Worker node
PySpark main
program
Driver node
SparkContext
Worker nodeWorker node
sc = SparkContext.getOrCreate()

28. H2O.ai
Machine Intelligence
Cluster
Worker node
Spark executor Spark executorSpark executor
PySpark main
program
Driver node
SparkContext
Worker nodeWorker node
sc = SparkContext.getOrCreate()

29. H2O.ai
Machine Intelligence
Cluster
Worker node
Spark executor Spark executorSpark executor
PySpark main
program
Driver node
SparkContext
Worker nodeWorker node
sc = SparkContext.getOrCreate()

30. H2O.ai
Machine Intelligence
Cluster
Worker node
Spark executor Spark executorSpark executor
PySpark main
program
Driver node
SparkContext
Worker nodeWorker node
H2OContext
sc = SparkContext.getOrCreate()
h2o_context = H2OContext.getOrCreate()

31. H2O.ai
Machine Intelligence
Cluster
Worker node
Spark executor Spark executorSpark executor
PySpark main
program
Driver node
SparkContext
Worker nodeWorker node
H2OContext
sc = SparkContext.getOrCreate()
h2o_context = H2OContext.getOrCreate()

32. H2O.ai
Machine Intelligence
Data
Source
SparkExecutorSparkExecutorSparkExecutor
Spark Cluster
DataFrame
H2OStore
H2OFrame
Data
Source
H2OStoreH2OStore
DataIngest

33. H2O.ai
Machine Intelligence
Data
Source
SparkExecutorSparkExecutorSparkExecutor
Spark Cluster
DataFrame
H2OStore
H2OFrame
h2o_context.as_spark_frame
H2OFrame serves data for DataFrame operations
Data
Source
H2OStoreH2OStore
DataExchange
PyAPI

34. H2O.ai
Machine Intelligence
Data
Source
SparkExecutorSparkExecutorSparkExecutor
Spark Cluster
DataFrame
H2OStore
H2OFrame
h2o_context.as_h2o_frame Materializes
DataFrame as H2OFrame (H2O as a clever cache)
Data
Source
H2OStoreH2OStore
DataExchange
PyAPI

35. H2O.ai
Machine Intelligence
Sentiment
Analysis
with
PySparkling
DEMO

36. H2O.ai
Machine Intelligence
Start PySparkling
Opens Jupyter
Notebook
Download from
h2o.ai/download

37. H2O.ai
Machine Intelligence
Future

38. H2O.ai
Machine Intelligence
The Plan
Separation of H2O cluster from Spark infrastructure
✓ Preserving existing API
h2oContext = H2OContext.getOrCreate(ip=“…”, port=…)
Better integration into PySpark pipelines
✓ Support of H2O Ensembles (right now only as R-package)
Integration with Steam platform to support model
management
DeepWater integration H2O DeepWater with Python
early sneak
Fabrizio MiloSunday
3pm

39. H2O.ai
Machine Intelligence
Checkout GitHub & Contribute
https://github.com/h2oai/sparkling-water
Checkout H2O.ai Training Books
http://h2o.ai/resources
Checkout H2O.ai Blog
http://h2o.ai/blog/
Checkout H2O.ai Youtube Channel
https://www.youtube.com/user/0xdata
More info

40. H2O.ai
Machine Intelligence
Learn more at h2o.ai
Follow us at @h2oai
Come to see us at Open Tour
in Dallas! See open.h2o.ai
PySparkling is
open-source
ML application platform
combining
power of PySpark and H2O
Thank you!
DALLAS, TX
OCT 26th