Download as PDF, PPTX
![Operating on domain objects
val personRDD = sc.makeRDD(Seq(Person("A",10), Person("B",20)))
//Create RDD[Person]
val personDF = sqlContext.createDataFrame(personRDD)
//Create dataframe from a RDD[Person]
personDF.rdd
//We get back RDD[Row] and not RDD[Person]](https://image.slidesharecdn.com/introduction-to-dataset-160215173515/85/Introduction-to-dataset-9-320.jpg)
![Compile time safety check
case class Person(name: String, age: Long)
val dataframe = sqlContext.read.json("people.json")
val ds : Dataset[Person] = dataframe.as[Person]
ds.filter(p => p.age > 25)
ds.filter(p => p.salary > 12500)
//error: value salary is not a member of Person](https://image.slidesharecdn.com/introduction-to-dataset-160215173515/85/Introduction-to-dataset-15-320.jpg)
![Operating on domain objects
val personRDD = sc.makeRDD(Seq(Person("A",10), Person("B",20)))
//Create RDD[Person]
val personDS = sqlContext.createDataset(personRDD)
//Create Dataset from a RDD
personDS.rdd
//We get back RDD[Person] and not RDD[Row] in Dataframe](https://image.slidesharecdn.com/introduction-to-dataset-160215173515/85/Introduction-to-dataset-16-320.jpg)
![Functional programming
case class Person(name: String, age: Int)
val dataframe = sqlContext.read.json("people.json")
val ds : Dataset[Person] = dataframe.as[Person]
// Compute histogram of age by name
val hist = ds.groupBy(_.name).mapGroups({
case (name, people) => {
val buckets = new Array[Int](10)
people.map(_.age).foreach { a =>
buckets(a / 10) += 1
}
(name, buckets)
}
})](https://image.slidesharecdn.com/introduction-to-dataset-160215173515/85/Introduction-to-dataset-17-320.jpg)
(
implicit val encoder: Encoder[T])
class DataFrame(
sqlContext: SQLContext,
queryExecution: QueryExecution)
extends Dataset[Row](sqlContext, queryExecution)(new RowEncoder)](https://image.slidesharecdn.com/introduction-to-dataset-160215173515/85/Introduction-to-dataset-24-320.jpg)
Uploaded bydatamantra
Introduction to dataset
The document introduces the Dataset API in Spark, which provides type safety and performance benefits over DataFrames. Datasets allow operating on domain objects using compiled functions rather than Rows. Encoders efficiently serialize objects to and from the JVM. This allows type checking of operations and retaining objects in distributed operations. The document outlines the history of Spark APIs, limitations of DataFrames, and how Datasets address these through compiled encoding and working with case classes rather than Rows.
More Related Content
![ict_presentation_final_final_final[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/ictpresentationfinalfinalfinal1-251230145259-2b4839bd-thumbnail.jpg?width=640&height=640&fit=bounds)
Introduction to dataset
- 1. Introduction to DatasetAPI Overcoming limitations of Dataframes https://github.com/shashankgowdal/introduction_to_dataset
- 2. ● Shashank L ●Big data consultant and trainer at datamantra.io ● www.shashankgowda.com
- 3. Agenda ● History ofSpark APIs ● Limitations of Dataframes ● Dataset ● Encoders ● Dataset hierarchy ● Performance ● Roadmap
- 4. RDD API (2011) ●Distributed collection for JVM objects ● Immutable and Fault tolerant ● Processing structured and unstructured data ● Functional transformations
- 5. Limitations of RDDAPI ● No schema associated ● Optimization should be done by from user’s end ● Reading from multiple sources is difficult ● Combining multiple sources is difficult
- 6. DataFrame API (2013) ●Distributed collection for Row objects ● Immutable and Fault tolerant ● Processing structured data ● Optimization from Catalyst optimizer ● Data source API
- 7. Limitations of Dataframe ●Compile time type safety ● Cannot operate on domain objects ● Functional programming API
- 8. Compile time safety valdataframe = sqlContext.read.json("people.json") dataframe.filter("salary > 1000").show() Throws Runtime exception org.apache.spark.sql.AnalysisException: cannot resolve 'salary' given input columns age, name;
- 9. Operating on domainobjects val personRDD = sc.makeRDD(Seq(Person("A",10), Person("B",20))) //Create RDD[Person] val personDF = sqlContext.createDataFrame(personRDD) //Create dataframe from a RDD[Person] personDF.rdd //We get back RDD[Row] and not RDD[Person]
- 10.
- 11. Dataset an extension ofthe DataFrame API that provides a type-safe, object-oriented programming interface
- 12. Dataset API ● Type-safe:Operate on domain objects with compiled lambda functions ● Fast: Code generated encoders for fast serialization ● Interoperable: Easily convert Dataframe Dataset without boilerplate code
- 13.
- 14. Encoders ● Encoder convertsfrom JVM object into a Dataset row ● Code generated encoders for fast serialization JVM Object Dataset row Encoder
- 15. Compile time safetycheck case class Person(name: String, age: Long) val dataframe = sqlContext.read.json("people.json") val ds : Dataset[Person] = dataframe.as[Person] ds.filter(p => p.age > 25) ds.filter(p => p.salary > 12500) //error: value salary is not a member of Person
- 16. Operating on domainobjects val personRDD = sc.makeRDD(Seq(Person("A",10), Person("B",20))) //Create RDD[Person] val personDS = sqlContext.createDataset(personRDD) //Create Dataset from a RDD personDS.rdd //We get back RDD[Person] and not RDD[Row] in Dataframe
- 17. Functional programming case classPerson(name: String, age: Int) val dataframe = sqlContext.read.json("people.json") val ds : Dataset[Person] = dataframe.as[Person] // Compute histogram of age by name val hist = ds.groupBy(_.name).mapGroups({ case (name, people) => { val buckets = new Array[Int](10) people.map(_.age).foreach { a => buckets(a / 10) += 1 } (name, buckets) } })
- 18. Dataset hierarchy SQL Dataframe (&Dataset) Python R Scala/Java Tungsten execution
- 19. Hands on ● Creatingdataset ○ From Collections ○ From File ● Comparison with RDD ○ Operations ○ Distributed Wordcount ● Semistructured data ○ Downcast ○ Upcast
- 20.
- 21.
- 22.
- 23. Roadmap ● Dataset, thename itself may change ● Performance optimizations ● Unification of DataFrames with Dataset ● Public API for Encoders ● Support for most of the RDD operators on Dataset
- 24. Unification of DataFrameswith Dataset class Dataset[T]( val sqlContext: SQLContext, val queryExecution: QueryExecution)( implicit val encoder: Encoder[T]) class DataFrame( sqlContext: SQLContext, queryExecution: QueryExecution) extends Dataset[Row](sqlContext, queryExecution)(new RowEncoder)
- 25.