Downloaded 14 times
Uploaded byProvectus
"Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer, at H2O.ai
The document presents an overview of H2O.ai's Sparkling Water, which integrates H2O machine learning algorithms with Apache Spark. It highlights features such as seamless integration, advanced modeling capabilities, and support for different programming languages, along with various operational modes of the H2O cluster. Additional resources for documentation, tutorials, and community engagement are also provided.
More Related Content
Similar to "Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer, at H2O.ai
![[DSC Europe 25] Miodrag Pesovic & Vladislav Radonjic - Federated Data Archite...](https://cdn.slidesharecdn.com/ss_thumbnails/gsbe3y5it5uhndi4e08e-1-251212103249-f1008e0c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Vladimir Jelic - The AI-Driven Security Shift From Reactive D...](https://cdn.slidesharecdn.com/ss_thumbnails/6g5gj25mtjwayniqem1t-6-251209104645-7a5a5fc6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Debmalya Biswas - Agentification: the art of transforming man...](https://cdn.slidesharecdn.com/ss_thumbnails/r5azlggvtqiaiiusrqdr-4-251212103249-5a12c89b-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Bassam Maharmeh - Artificial Intelligence: Opportunities and ...](https://cdn.slidesharecdn.com/ss_thumbnails/thhfmr2fqpawzj7hsjpg-5-251211083048-2c23204f-thumbnail.jpg?width=640&height=640&fit=bounds)
"Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer, at H2O.ai
- 1. The Next Generationof Machine Learning on Apache Spark Jakub Háva jakub@h2o.ai Provectus Meetup, Kyiv Oct 12, 2018
- 2. Who am I •Senior Software engineer at H2O.ai - Core Sparkling Water • Finished Master’s at Charles Uni in Prague • Implemented high-performance cluster monitoring tool for JVM based languages ( JNI, JVMTI, instrumentation ) • Climbing & Tea lover, ( doesn’t mean I don’t like beer!)
- 3. H2O Products In-Memory, Distributed MachineLearning Algorithms with H2O Flow GUI H2O AI Open Source Engine Integration with Spark Lightning Fast machine learning on GPUs Automatic feature engineering, machine learning and interpretability Secure multi-tenant H2O clusters H2O AI Open Source Engine Integration with Spark
- 4. Distributed Sparkling Team •Michal - Mt. View, CA • Kuba - Prague, CZ
- 5.
- 6. Sparkling Water • Transparentintegration of H2O with Spark ecosystem - MLlib and H2O side-by-side • Transparent use of H2O data structures and algorithms with Spark API • Platform for building Smarter Applications • Excels in existing Spark workflows requiring advanced Machine Learning algorithms Functionality missing in H2O can be replaced by Spark and vice versa
- 7. Benefits • Additional algorithms –NLP • Powerful data munging • ML Pipelines • Advanced algorithms • speed v. accuracy • advanced parameters • Fully distributed and parallelised • Graphical environment • R/Python interface
- 8. H2O.ai Machine Intelligence How touse Sparkling Water?
- 9. Start spark withSparkling Water
- 10. Model Building Data Source Data mungingModelling Deep Learning, GBM DRF, GLM, GLRM K-Means, PCA CoxPH, Ensembles Prediction processing
- 11.
- 12. Stream Processing Data Source Off-line model training Data munging Model prediction Deploy themodel Stream processing Data Stream Spark Streaming/Storm/Flink Export model in a binary format or as code Modelling
- 13. Scoring • POJO o PlainOld Java Object • MOJO o Model Object Optimised • MOJO Pipeline o MOJO from DAI with additional future transformations • No runtime dependency on H2O framework
- 14.
- 15. Scala code inH2O Flow • New type of cell • Access Spark from Flow UI • Experimenting made easy
- 16. H2O Frame asSpark’s Datasource • Use native Spark API to load and save data • Spark can optimise the queries when loading data from H2O Frame • Use of Spark query optimiser
- 17. Machine learning pipelines •Wrap our algorithms as Transformers and Estimators • Support for embedding them into Spark ML Pipelines • Can serialise fitted/unfitted pipelines • Unified API => Arguments are set in the same way for Spark and H2O Models • Integration with Mojo pipelines
- 18. PySparkling • PySparkling ison PyPi • Contains all H2O and Sparkling Water dependencies, no need to worry about them • Just add in on your Python path and that’s it • Correctly specifies dependency on PySpark
- 19. RSparkling • Sparkling Waterfor R • Based on Sparklyr package • Independent on Spark and Sparkling Water version
- 20. And others! • Supportfor Datasets • Zeppelin notebook support • XGBoost Support in Distributed Mode • Support for high-cardinality fast joins • Secure Communication - SSL • Support for Sparse Data conversions • External Cluster Stabilisation • Enterprise security support - LDAP, Kerberos • Driverless AI Pipeline deployment availability
- 21. Coming features • Integrationwith Steam o Deploy on premises o Deploy to Cloud
- 22.
- 23. Two Backends • SparklingWater has two backends o External o Internal • The backend determines where H2O cluster is located • Each backend is good for different use-cases
- 24.
- 25. Sparkling Water InternalBackend Sparkling App jar file Spark Master JVM spark-submit Spark Worker JVM Spark Worker JVM Spark Worker JVM Sparkling Water Cluster Spark Executor JVM H2O Spark Executor JVM H2O Spark Executor JVM H2O
- 26. Internal Backend Cluster Worker node Sparkexecutor Spark executorSpark executor Driver node SparkContext Worker nodeWorker node H2OContext
- 27.
- 28. Pros & Cons •Advantages • Easy to configure • Faster ( no need to send data to different cluster) • Disadvantages • Spark kills or joins new executor => H2O goes down • No way how to discover all executors
- 29.
- 30. Overview • Sparkling Wateris using external H2O cluster instead of starting H2O in each executor • Spark executors can come and go and H2O won’t be affected • Start H2O cluster on YARN automatically
- 31. Separation Approach • SeparatingSpark and H2O • But preserving same API: val h2oContext = H2OContext.getOrCreate(“http://h2o:54321/) • Spark and H2O can be submitted as Yarn job and controlled in separation • But H2O still needs non-elastic environment (H2O itself does not implement HA yet)
- 32. Sparkling Water ExternalBackend Sparkling App jar file Spark Master JVM spark-submit Spark Worker JVM Spark Worker JVM Spark Worker JVM Sparkling Water Cluster Spark Executor JVM H2O Spark Executor JVM H2O Spark Executor JVM H2O H2O Cluster H2O
- 33. External Backend Cluster Worker node Sparkexecutor Spark executorSpark executor Driver node SparkContext Worker nodeWorker node H2OContext
- 34. Pros & Cons •Advantages • H2O does not crash when Spark executor goes down • Better resource management since resources can be planned per tool • Disadvantages • Transfer overhead between Spark and H2O processes • under measurement with cooperation of a customer
- 35. Modes • Auto Startmode • Start h2o cluster automatically on YARN • Manual Start Mode • User is responsible for starting the cluster manually
- 36.
- 37. Upcoming Events • ProductionizingH2O Models with Apache Spark
- 38. More Info • Documentation:http://docs.h2o.ai • Tutorials: https://github.com/h2oai/h2o-tutorials • Slidedecks: https://github.com/h2oai/h2o-meetups • Videos: https://www.youtube.com/user/0xdata • Events & Meetups: http://h2o.ai/events • Stack Overflow: https://stackoverflow.com/tags/sparkling-water • Google Group: https://tinyurl.com/h2ostream • Gitter: http://gitter.im/h2oai/sparkling-water
- 39. Learn more ath2o.ai Follow us at @h2oai PS: We are hiring! Thank you! Sparkling Water is open-source ML application platform combining power of Spark and H2O