Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Processing Fast Data with Apache Spark: The Tale of Two APIs
Similar to Processing Fast Data with Apache Spark: The Tale of Two APIs (20)
More from Matt Stubbs
More from Matt Stubbs (20)
Recently uploaded
Recently uploaded (20)
Processing Fast Data with Apache Spark: The Tale of Two APIs
- 1. Processing Fast Data with Apache Spark: The Tale of Two APIs Big Data Lnd Gerard Maas Señor SW Engineer 16/Nov/2017
- 2. Gerard Maas Señor SW Engineer Computer Engineer Scala Programmer Early Spark Adopter (v0.9) Spark Notebook Contributor Cassandra MVP (2015, 2016) Stack Overflow Top Contributor (Spark, Spark Streaming, Scala) Wannabe { IoT Maker Drone crasher/tinkerer } @maasg https://github.com/maasg https://www.linkedin.com/ in/gerardmaas/ https://stackoverflow.com /users/764040/maasg
- 8. Apache Spark Core Spark SQL SparkMLLib SparkStreaming Structured Streaming DataFrames DataSets GraphFrames Data Sources
- 9. Apache Spark Core Spark SQL SparkMLLib SparkStreaming Structured Streaming DataFrames DataSets GraphFrames Data Sources
- 14. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, reading: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"reading" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()
- 15. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, reading: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"reading" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()
- 16. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, reading: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"reading" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()
- 17. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, reading: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"reading" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()
- 18. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, reading: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"reading" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()
- 19. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, reading: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"reading" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()
- 20. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, reading: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"reading" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()
- 21. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, value: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"value" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()
- 22. Structured Streaming - Usecases ● ETL ● Stream aggregations, windows ● Event-time oriented analytics ● Join Streams with Fixed Datasets ● Apply Machine Learning Models
- 24. Spark Streaming Kafka Flume Kinesis Twitter Sockets HDFS/S3 Custom Apache Spark SparkSQL SparkML ... Databases HDFS API Server Streams
- 26. DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1
- 27. DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformation T -> U
- 28. DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformation T -> U Actions
- 31. API: Transformations transform val iotDstream = MQTTUtils.createStream(...) val devicePriority = sparkContext.cassandraTable(...) val prioritizedDStream = iotDstream.transform{rdd => rdd.map(d => (d.id, d)).join(devicePriority) }
- 33. Actions print ------------------------------------------- Time: 1459875469000 ms ------------------------------------------- data1 data2 saveAsTextFiles, saveAsObjectFiles, saveAsHadoopFiles xxx yyy zzz foreachRDD * Spark SQL Dataframes GraphFrames Any API
- 34. Structured Streaming - Usecases ● Stream-stream joins ● Complex state management (local + cluster state) ● Streaming Machine Learning ○ Learn ○ Score ● Join Streams with Updatable Datasets ● Event-time oriented analytics ● Continuous processing
- 36. Spark Streaming + Structured Streaming 36 val parse: Dataset[String] => Dataset[Record] = ??? val process: Dataset[Record] => Dataset[Result] = ??? val serialize: Dataset[Result] => Dataset[String] = ??? val kafkaStream = spark.readStream… val f = parse andThen process andThen serialize val result = f(kafkaStream) result.writeStream .format("kafka") .option("kafka.bootstrap.servers",bootstrapServers) .option("topic", writeTopic) .option("checkpointLocation", checkpointLocation) .start() val dstream = KafkaUtils.createDirectStream(...) dstream.map{rdd => val ds = sparkSession.createDataset(rdd) val f = parse andThen process andThen serialize val result = f(ds) result.write.format("kafka") .option("kafka.bootstrap.servers", bootstrapServers) .option("topic", writeTopic) .option("checkpointLocation", checkpointLocation) .save() } Structured StreamingSpark Streaming
- 37. Streaming Pipelines Structured Streaming Keyword Extraction Keyword Relevance Similarity DB Storage
- 40. Features 1. One-click component installations 2. Automatic dependency checks 3. One-click access to install logs 4. Real-time cluster visualization 5. Access to consolidated production logs Benefits: 1. Easy to get started 2. Ready access to all components 3. Increased developer velocity Fast Data Platform Manager, for Managing Running Clusters
- 41. Fast Data Platform Visit us at the Booth!