Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Collecting Uncertain Data the Reactive Way Jeff Smith @jeffksmithjr
x.ai is a personal assistant who schedules meetings for you
Reactive Machine Learning
Machine Learning Systems
Machine Learning Systems
Machine Learning Systems
Traits of Reactive Systems
Traits of Reactive Systems
Reactive Strategies
Reactive Strategies
Reactive Machine Learning
Reactive Machine Learning
Reactive Machine Learning
Collecting Data
What’s for dinner?
Reactive Data Collection
Modeling Uncertain Data
Certain Data Model case class ZebraReading(sensorId: Int, locationId: Int, timestamp: Long, count: Int)
Uncertainty Interval 27 33
Uncertain Data Model case class PreyReading(sensorId: Int, locationId: Int, timestamp: Long, animalsLowerBound: Double, an...
Scaling Data Collection
Simple Data Architecture
Simple Data Architecture
Mutable State case class Region(id: Int) import collection.mutable.HashMap var densities = new HashMap[Region, Double]() d...
Scaling with Queues
Scaling with Queues
Out of Order Updates
Out of Order Updates densities.put(Region(6), 73.6) densities.put(Region(6), 0.5) densities.get(Region(6)).get
Out of Order Updates densities.put(Region(6), 73.6) densities.put(Region(6), 0.5) densities.get(Region(6)).get densities.p...
Concurrent Collections import collection.mutable._ var synchronizedDensities = new LinkedHashMap[Region, Double]() with Sy...
Scaling with Locks
Scaling with Locks
Immutable Facts case class PreyReading(sensorId: Int, locationId: Int, timestamp: Long, animalsLowerBound: Double, animals...
Immutable Facts val reading = PreyReading(36, 12, currentTimeMillis(), 12.0, 18.0, 0.60) val setDoc = bucket.set[PreyReadi...
Scaling with Distributed Databases
Scaling with Distributed Databases
Handling Incomplete Data
Distributed Data Storage
Querying Complete Data (bucket.searchValues[PreyReading]("prey", "by_sensor_id") (new Query().setIncludeDocs(true))) .enum...
Complete Data
Partition Tolerance
Partition Tolerance
Partition Tolerance
Partition Tolerance
Querying Incomplete Data (bucket.searchValues[PreyReading]("prey", "by_sensor_id") (new Query().setIncludeDocs(true))) .en...
Incomplete Data
Incomplete Data
Reactive Data Collection
For Later
reactivemachinelearning.com medium.com/data-engineering M A N N I N G Jeff Smith
x.ai @xdotai hello@human.x.ai New York, New York
skillsmatter.com/conferences/ 6862-scala-exchange-2015#skillscasts
Thank You
Collecting Uncertain Data the Reactive Way Jeff Smith @jeffksmithjr
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Upcoming SlideShare
Loading in …5
×

Collecting Uncertain Data the Reactive Way

41,240 views

Published on

Related Resources
----------------------
Video (starts at 25 minutes): https://skillsmatter.com/skillscasts/7038-lightning-talks-2
From Scala Exchange 2015: https://skillsmatter.com/conferences/1948-scala-exchange-2014
Reactive Machine Learning: http://www.reactivemachinelearning.com/
Data Engineering blogging: https://medium.com/data-engineering

Talk Summary
-----------------
Before you can ever get started building large-scale data analytic systems, you need to start with one crucial element: data. Collecting data, especially collecting lots of data, is harder than it seems. Data ingested with the wrong data model can be worse than no data at all. A data collection system that is too slow can bring an entire platform grinding to a halt.

Don't panic! Scalable, non-destructive data collection is possible. This talk will focus on strategies for data collection based on real world experience building large scale machine learning systems. It will introduce ideas from the emerging paradigm of reactive machine learning that are based on older ideas about immutable facts and pervasive, intrinsic uncertainty.

Published in: Data & Analytics
License: CC Attribution-NonCommercial-ShareAlike License
no profile picture user

  • Login to see the comments

Show More

Collecting Uncertain Data the Reactive Way

  1. Collecting Uncertain Data the Reactive Way Jeff Smith @jeffksmithjr
  2. x.ai is a personal assistant who schedules meetings for you
  3. Reactive Machine Learning
  4. Machine Learning Systems
  5. Machine Learning Systems
  6. Machine Learning Systems
  7. Traits of Reactive Systems
  8. Traits of Reactive Systems
  9. Reactive Strategies
  10. Reactive Strategies
  11. Reactive Machine Learning
  12. Reactive Machine Learning
  13. Reactive Machine Learning
  14. Collecting Data
  15. What’s for dinner?
  16. Reactive Data Collection
  17. Modeling Uncertain Data
  18. Certain Data Model case class ZebraReading(sensorId: Int, locationId: Int, timestamp: Long, count: Int)
  19. Uncertainty Interval 27 33
  20. Uncertain Data Model case class PreyReading(sensorId: Int, locationId: Int, timestamp: Long, animalsLowerBound: Double, animalsUpperBound: Double, percentZebras: Double)
  21. Scaling Data Collection
  22. Simple Data Architecture
  23. Simple Data Architecture
  24. Mutable State case class Region(id: Int) import collection.mutable.HashMap var densities = new HashMap[Region, Double]() densities.put(Region(4), 52.4)
  25. Scaling with Queues
  26. Scaling with Queues
  27. Out of Order Updates
  28. Out of Order Updates densities.put(Region(6), 73.6) densities.put(Region(6), 0.5) densities.get(Region(6)).get
  29. Out of Order Updates densities.put(Region(6), 73.6) densities.put(Region(6), 0.5) densities.get(Region(6)).get densities.put(Region(6), 0.5) densities.put(Region(6), 73.6) densities.get(Region(6)).get
  30. Concurrent Collections import collection.mutable._ var synchronizedDensities = new LinkedHashMap[Region, Double]() with SynchronizedMap[Region, Double]
  31. Scaling with Locks
  32. Scaling with Locks
  33. Immutable Facts case class PreyReading(sensorId: Int, locationId: Int, timestamp: Long, animalsLowerBound: Double, animalsUpperBound: Double, percentZebras: Double) implicit val preyReadingFormatter = Json.format[PreyReading]
  34. Immutable Facts val reading = PreyReading(36, 12, currentTimeMillis(), 12.0, 18.0, 0.60) val setDoc = bucket.set[PreyReading](readingId(reading), reading)
  35. Scaling with Distributed Databases
  36. Scaling with Distributed Databases
  37. Handling Incomplete Data
  38. Distributed Data Storage
  39. Querying Complete Data (bucket.searchValues[PreyReading]("prey", "by_sensor_id") (new Query().setIncludeDocs(true))) .enumerate.apply(Iteratee.foreach { doc => println(s"Prey Reading: $doc")})
  40. Complete Data
  41. Partition Tolerance
  42. Partition Tolerance
  43. Partition Tolerance
  44. Partition Tolerance
  45. Querying Incomplete Data (bucket.searchValues[PreyReading]("prey", "by_sensor_id") (new Query().setIncludeDocs(true))) .enumerate.apply(Iteratee.foreach { doc => println(s"Prey Reading: $doc")})
  46. Incomplete Data
  47. Incomplete Data
  48. Reactive Data Collection
  49. For Later
  50. reactivemachinelearning.com medium.com/data-engineering M A N N I N G Jeff Smith
  51. x.ai @xdotai hello@human.x.ai New York, New York
  52. skillsmatter.com/conferences/ 6862-scala-exchange-2015#skillscasts
  53. Thank You
  54. Collecting Uncertain Data the Reactive Way Jeff Smith @jeffksmithjr

×