Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IoT Austin CUG talk

431 views

Published on

Join our meetup: http://www.meetup.com/Austin-Cloudera-User-Group/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

IoT Austin CUG talk

  1. 1. 1© Cloudera, Inc. All rights reserved. IoT with Spark Streaming Anand Iyer, Senior Product Manager
  2. 2. 2© Cloudera, Inc. All rights reserved. Spark Streaming • Incoming data stream is represented as DStreams (Discretized Streams) • Stream is broken down into micro-batches • Each micro-batch is an RDD – process using RDD operations • Micro-batches usually 0.5 sec in size
  3. 3. 3© Cloudera, Inc. All rights reserved. Cloudera customer use case examples – Streaming • On-line fraud detection Financial Services • On-line recommender systems • Inventory management Retail • Incident prediction (sepsis) Health • Analysis of ad performance in real-time Ad tech
  4. 4. 4© Cloudera, Inc. All rights reserved. Concrete end-to-end IoT Use Case Using Spark Streaming with Kafka, HBase & Solr
  5. 5. 5© Cloudera, Inc. All rights reserved. Proactive maintenance and accident prevention in Railways • Sensor information continuously streaming in from railway carriages • Goal: Early detection of damage to rail carriage wheels or to railway tracks • Proactively fix issues before they become severe • Prevent derailments, save money and lives • Based on real-world use case, modified to fit the talk
  6. 6. 6© Cloudera, Inc. All rights reserved. Locomotive Wheel Axle Sensors Each Sensor Reading Contains: - Unique ID - Locomotive ID - Speed - Temperature - Pressure - Acoustic signals - GPS Co-ordinates - Timestamp - etc
  7. 7. 7© Cloudera, Inc. All rights reserved. Identify Damage to locomotive axle or wheels Manifests as sustained increase in sensor readings like temperature, pressure, acoustic noise, etc.
  8. 8. 8© Cloudera, Inc. All rights reserved. Identify Damage on railway tracks Manifests as a sudden spike in sensor readings for pressure or acoustic noise.
  9. 9. 9© Cloudera, Inc. All rights reserved. Real-Time Detection of Locomotive Wheel Damage Kafka - Enrich incoming events with relevant meta- data - Locomotive information from locomotive ID: type, weight, cargo,etc - Sensor information from Sensor ID: precise location, type, etc - GPS co-ordinates to location characteristics such as gradient of track. - Recommend HBase as metadata store. - Use HBase-spark module to fetch data. - Apply application logic to determine if sensor readings indicate damage - Simple rule based - Complex predictive machine learning model
  10. 10. 10© Cloudera, Inc. All rights reserved. Real-Time Detection of Locomotive Wheel Damage Kafka Kafka https://github.com/harishreedharan/spark-streaming-kafka-output HDFS
  11. 11. 11© Cloudera, Inc. All rights reserved. Real-Time Detection of Locomotive Wheel Damage - When an alert is thrown, technician will need to diagnose the event - Requires visualizing sensor data as a time-series: - Over arbitrary windows of time - Compare with values from prior trips - Software for visualization: http://grafana.org/ - Technician can take appropriate action based on analysis: - Send rail carriage for maintenance - Stop train immediately to prevent accident Visualize Time-Series Sensor Data
  12. 12. 12© Cloudera, Inc. All rights reserved. Data Store for Time-Series Data Ideal solution: Kudu - Time series data entails sequential scans for writes and reads, interspersed with random seeks Until Kudu is GA: - Use HBase and model tables for time-series data - OpenTSDB: - Built on top of HBase - Uses a HBase table schema optimized for time-series data - Simple HTTP API
  13. 13. 13© Cloudera, Inc. All rights reserved. Real-Time Detection of Locomotive Wheel Damage Kafka Kafka HDFS
  14. 14. 14© Cloudera, Inc. All rights reserved. Detecting damage to Railtracks • They manifest as a sharp spike in sensor readings (pressure, acoustic noise) • Multiple sensors will demonstrate the same spike at the same location (GPS co- ordinates) • Multiple sensors from multiple trains will give similar readings at the same location. How to detect? • Index each sensor reading, in Solr, such that they can be queried by GPS co- ordinates • When a “spike” is observed, and corresponding alert event is fired, trigger a search
  15. 15. 15© Cloudera, Inc. All rights reserved. Detecting damage to Rail tracks • Index each sensor reading, with the Morphlines library • Embed call to Morphlines in your Spark Streaming application • Values can be kept in the index for specified period of time, such as a month. Solr can automatically purge old documents from the index. • When a “spike” is observed, and corresponding alert event is fired, trigger a search (manually or programmatically) • Search for sensor readings at the same GPS co-ordinates as the latest spike. • Filter out irrelevant readings (e.g. readings on the left track, if spike was observed on the right track) • Sort results by time, latest to oldest • If majority of recent readings show a “spike”, indicative of track damage
  16. 16. 16© Cloudera, Inc. All rights reserved. Final Architecture Kafka Kafka HDFS
  17. 17. 17© Cloudera, Inc. All rights reserved. Noteworthy Streaming Constructs
  18. 18. 18© Cloudera, Inc. All rights reserved. Sliding Window Operations Example usages: - compute counts of items in latest window of time, such as occurrences of exceptions in a log or trending hashtags in a tweet stream - Join two streams by matching keys within same window Note: Provide adequate memory to hold a window’s worth of data Define operations on data within a sliding window. Window Parameters: - window length - sliding interval
  19. 19. 19© Cloudera, Inc. All rights reserved. Maintain and update arbitrary state updateStateByKey(...) • Define initial state • Provide state update function • Continuously update with new information Examples: • Running count of words seen in text stream • Per user session state from activity stream Note: Requires periodic check-pointing to fault-tolerant storage.
  20. 20. 20© Cloudera, Inc. All rights reserved. Lessons from Production
  21. 21. 21© Cloudera, Inc. All rights reserved. Use Kafka Direct Connector whenever possible • Better efficiency and performance than Receiver based Connectors • Automatic back-pressure: steady performance Kafka Spark Driver Executor Executor Executor Executor Receiver Receiver Spark Driver Executor Executor Executor Executor
  22. 22. 22© Cloudera, Inc. All rights reserved. The challenge with Checkpoints • Spark checkpoints are java serialized • Upgradeability can be an issue – upgrading the version of Spark or your application can make checkpointed data unreadable But long running applications need updates and upgrades!!
  23. 23. 23© Cloudera, Inc. All rights reserved. Upgrades with Checkpoints • Most often, all you need to pick up is some previous state – maybe an RDD or some “state”(updateStateByKey), or last processed Kafka offsets • The solution: Disable Spark Checkpoints • Use foreachRDD to persist state yourself, to HDFS, in a format your application can understand • E.g. Avro, Protobuf, Parquet… • For upateStateByKey, generate the new state - then persist
  24. 24. 24© Cloudera, Inc. All rights reserved. updateStateByKey(…) upcoming improvements • Time-out: Automatically delete data after a preset number of micro-batches • Efficient Updates: Only update a subset of the keys • Callback to persist state during graceful shutdown
  25. 25. 25© Cloudera, Inc. All rights reserved. Exactly Once Semantics What is it? Given a stream of incoming data, any operator is applied exactly once on each item. Why is it important? Prevent erroneous processing of data stream. E.g., Double counting of aggregations or throwing of redundant alerts Spark Streaming provides exactly one semantics for data transformations. However, output operations provide at-least once semantics!!
  26. 26. 26© Cloudera, Inc. All rights reserved. Exactly Once Semantics with Spark Streaming & Kafka • Associate a “key” with each value written to external store, that can be used for de-duping • This key needs to be unique for a given micro-batch • Kafka Direct Connector provides the following associated with each record, which will be the same for a given micro-batch: Kafka-Partition + start-offset + end-offset • Check out org.apache.spark.streaming.kafka.OffsetRanges and org.apache.spark.streaming.kafka.HasOffsetRanges
  27. 27. 27© Cloudera, Inc. All rights reserved. Thank You

×