Download as PDF, PPTX
![Solved challenges
Large number of columns in SQL schema
•>22 columns (case class restriction)
•Solved via Product interface
class Airlines( year :Option[Int], // 0!
month :Option[Int], // 1!
dayOfMonth :Option[Int], // 2!
dayOfWeek :Option[Int], // 3!
crsDepTime :Option[Int], // 5!
crsArrTime :Option[Int], // 7!
uniqueCarrier :Option[String], // 8!
flightNum :Option[Int], // 9!
tailNum :Option[Int], // 10!
crsElapsedTime:Option[Int], // 12!
origin :Option[String], // 16!
dest :Option[String], // 17!
distance :Option[Int], // 18!
isArrDelayed :Option[Boolean],// 29!
isDepDelayed :Option[Boolean] // 30!
) extends Product { … }](https://image.slidesharecdn.com/sparklingwater5-28-14-140529134833-phpapp02/85/Sparkling-Water-5-28-14-38-320.jpg)
![Solved challenges
Handling NAs during load
•Store them in SQL RDD
•Solved by https://github.com/apache/spark/pull/
658
•Use Option[T] or non-primitive Java type
!
Handling NAs during save
•A simple sql.Row serializer handling NA values](https://image.slidesharecdn.com/sparklingwater5-28-14-140529134833-phpapp02/85/Sparkling-Water-5-28-14-39-320.jpg)
!
!
// Now make a blocking call of GBM directly via Java API!
gbm(frame, frame.vec("isDepDelayed"), 100, true)!
}](https://image.slidesharecdn.com/sparklingwater5-28-14-140529134833-phpapp02/85/Sparkling-Water-5-28-14-41-320.jpg)
!
!
// Now make a blocking call of GBM directly via Java API!
gbm(frame, frame.vec("isDepDelayed"), 100, true)!
}
Demo code](https://image.slidesharecdn.com/sparklingwater5-28-14-140529134833-phpapp02/85/Sparkling-Water-5-28-14-42-320.jpg)
Uploaded bySri Ambati
Sparkling Water 5 28-14
1. The document summarizes steps towards integrating the H2O and Spark frameworks, including allowing data sharing between Spark and H2O. 2. A demonstration is shown of loading airline data from a CSV into a Spark SQL table, querying the table, and transferring the results to an H2O frame to run a GBM algorithm. 3. Next steps discussed include optimizing data transfers between Spark and H2O, developing an H2O backend for MLlib, and addressing open challenges in areas like transferring results and supporting Parquet.
More Related Content
![제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계](https://cdn.slidesharecdn.com/ss_thumbnails/3-1boaz23rdconferencemboax-260203102709-9d519923-thumbnail.jpg?width=640&height=640&fit=bounds)
Sparkling Water 5 28-14
- 1.
- 2. Who am I? Background •PhDin CS from Charles University in Prague, 2012 •1 year PostDoc at Purdue University experimenting with algos for large computation •1 year at 0xdata helping to develop H2O engine for big data computation ! Experience with domain-specific languages, distributed system, software engineering, and big data.
- 3. Overview 1.Towards H2O andSpark integration 2.Details and demo 3.Next steps…
- 4. Vision Towards Spark andH2O integration
- 5. User-friendly API ! Large andactive community ! Platform components - SQL ! Multitenancy Memory efficient ! Performance of computation ! Machine learning algorithms ! Parser, R-interface
- 6. Combine benefits ofboth tools and makes “H2O a killer application for Spark”
- 7. Steps towards ! interoperability 1.Datasharing between Spark to H2O 2.Optimize & improve ! 3.Low-level integration
- 8. Steps towards ! interoperability 1.Datasharing between Spark to H2O 2.Optimize & improve ! 3.Low-level integration
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15. Data sharing scenario FrameRDD SQLquery Algo
- 16. Data sharing scenario FrameRDD SQLquery Algo
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24. Data sharing! via TachyonTachyon
- 25. Data sharing! via Tachyon H2Onode with Spark driver Tachyon
- 26. Data sharing! via TachyonTachyon Invoke Spark driver
- 27. Data sharing! via TachyonTachyon Load data
- 28.
- 29. Data sharing! via TachyonTachyon Persist data to Tachyon
- 30. Data sharing! via TachyonTachyon Load data into H2O frame
- 31. Data sharing! via TachyonTachyon Invoke GBM on data
- 32. Spark 1.0-rc11 ! SQL component ! Implementedproper parser/ serializer to satisfy H2O parser
- 33. Latest H2O version- 2.5-SNAPSHOT ! With Tachyon support included ! Embedded Spark driver
- 34. Key requirements Transparent approach ! Workwith many columns ! Preserve NAs ! Preserve headers
- 35. Key requirements Transparent approach ! Workwith many columns ! Preserve NAs ! Preserve headers
- 36. Key requirements Transparent approach ! Workwith many columns ! Preserve NAs ! Preserve headers
- 37. Key requirements Transparent approach ! Workwith many columns ! Preserve NAs ! Preserve headers
- 38. Solved challenges Large numberof columns in SQL schema •>22 columns (case class restriction) •Solved via Product interface class Airlines( year :Option[Int], // 0! month :Option[Int], // 1! dayOfMonth :Option[Int], // 2! dayOfWeek :Option[Int], // 3! crsDepTime :Option[Int], // 5! crsArrTime :Option[Int], // 7! uniqueCarrier :Option[String], // 8! flightNum :Option[Int], // 9! tailNum :Option[Int], // 10! crsElapsedTime:Option[Int], // 12! origin :Option[String], // 16! dest :Option[String], // 17! distance :Option[Int], // 18! isArrDelayed :Option[Boolean],// 29! isDepDelayed :Option[Boolean] // 30! ) extends Product { … }
- 39. Solved challenges Handling NAsduring load •Store them in SQL RDD •Solved by https://github.com/apache/spark/pull/ 658 •Use Option[T] or non-primitive Java type ! Handling NAs during save •A simple sql.Row serializer handling NA values
- 40.
- 41. Step-by-step Start Spark cloud- 1 worker Start Tachyon storage Start H2O slave node Start H2O master node with Scala master program override def run(conf: DemoConf): Unit = {! // Dataset! val dataset = "data/allyears2k_headers.csv"! // Row parser! val rowParser = AirlinesParser! // Table name for SQL! val tableName = "airlines_table"! // Select all flights with destination == SFO! val query = """SELECT * FROM airlines_table WHERE dest="SFO" """! ! // Connect to shark cluster and make a query over prostate, transfer data into H2O! val frame:Frame = executeSpark[Airlines](dataset, rowParser, ! ! ! ! conf.extractor, tableName, query, local=conf.local)! ! // Now make a blocking call of GBM directly via Java API! gbm(frame, frame.vec("isDepDelayed"), 100, true)! }
- 42. Step-by-step Start Spark cloud- 1 worker Start Tachyon storage Start H2O slave node Start H2O master node with Scala master program override def run(conf: DemoConf): Unit = {! // Dataset! val dataset = "data/allyears2k_headers.csv"! // Row parser! val rowParser = AirlinesParser! // Table name for SQL! val tableName = "airlines_table"! // Select all flights with destination == SFO! val query = """SELECT * FROM airlines_table WHERE dest="SFO" """! ! // Connect to shark cluster and make a query over prostate, transfer data into H2O! val frame:Frame = executeSpark[Airlines](dataset, rowParser, ! ! ! ! conf.extractor, tableName, query, local=conf.local)! ! // Now make a blocking call of GBM directly via Java API! gbm(frame, frame.vec("isDepDelayed"), 100, true)! } Demo code
- 43. Next steps… Optimize datatransfers •Have notion of H2O RDD inside Spark H2O Backend for MLlib •Based on H2O RDD •Use H2O algos
- 44. Open challenges See http://jira.0xdata.comand Sparkling component •PUB-730 Transfer results from H2O frame into RDD •PUB-732 Parquet support for H2O •PUB-733 MLlib backend •PUB-734 H2O-based RDD
- 45.
- 46. Learn more aboutH2O at 0xdata.com or Thank you! Follow us at @hexadata neo> for r in h2o h2o-sparkling; do ! git clone “git@github.com:0xdata/$r.git”! done