Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Assaf Araki – Real Time Analytics at Scale

6,813 views

Published on

Flink Forward 2015

Published in: Technology
  • Be the first to comment

Assaf Araki – Real Time Analytics at Scale

  1. 1. REAL time Analytics AT SCALE SMART DATA PIPES For THE INTERNET OF THINGS Assaf Araki, Big Data Analytics Architect Big Data Analytics, Intel
  2. 2. Intro to Big Data Analytics @ Intel People (+100) Data Scientists Management Big Data Developers Analytics PMs 13% 41% 9% 37% CONTRIBUTION TO Data Center Group CONTRIBUTION TO INTEL Operations MISSIO N #1 Operational excellence #2 Help Intel win area of Intelligent machines VISION Analytics is a competitive advantage for Intel Industry / Academy Technical due-diligence assessment for Intel Capital Benchmark with startups Academy Collaborations Assist Intel Sales & Marketing DESIGN Cut validations time-to-market MANUFACTURI NGReduce test cost SALES & MARKETINGIncrease sales through analytics Stream Analytics Cloud Parkinson Research Machine Learning Strategy
  3. 3. The IOT challenge CloudIngestionThings Cloud Infrastructure Data Platform Analytics Platform UI Services
  4. 4. Use case : The Parkinson Disease research 44 CLINICAL TRIALS Create and Validate Algorithms & Measures POPULATION STUDY Generate insights Using Big data analytics
  5. 5. 10 Medication reporting Medication reminder Report PATIENT REPORTING OTHER Configurable data collections Contribution score Integrated Login and registration Pebble notifications OBJECTIVE MEASURES Gait Sleep Tremor Activity Level Controlled Tests
  6. 6. So, Why is it Big-Data Problem? 30 subjects 5 DaysperSubject 0.15TB Weekly 500 subjects 30 DaysperSubject 1GB PerSubjectperDay 15TB Monthly 1000 subjects 365 DaysperSubject 365TB Yearly 1GB PerSubjectperDay 1GB PerSubjectperDay
  7. 7. SERVICE BATCH ANAYTICS STREAM ANALYICS INGESTION STORAGE USER INTERFACE Mosquitt o 7 CLOUD COMPUTING SERVICES
  8. 8. Smart Ingestion characteristics Personalized Easy to use Smart Data Pipe • Per single device or user • Maintain state and required data for ML • Easily subscribe to any Stream • Use familiar development Languages (Java, Scala) • Developers focus on logic development • Apply analytics on the Stream • Trigger actions (close the feedback loop) in timely manner Scalability • Linear scalability (scale Out) • Extremely High concurrencies • High Throughput Fault Tolerance• No Single point of failure • Seamless recovery • Persistent
  9. 9. Smart Data Ingestion – High level overview 9 Device Device Device Device Scalable, Persistent Broker Processing, Stream Analytics
  10. 10. What is Akka? • Micro-service(Actor) oriented. • Message Driven • Lock-free • Location-transparent • High performance • Fault Tolerant • Scales linearly
  11. 11. Stream Processing - the Akka way… 11 Each actor is a small peace of Java or Scala code performing its role A set of actors creates a topology which is responsible for device’s data stream processing A single Akka node may have millions of concurrent actors handling different streams and operations Change detection Automatic change detection time rules matcher Detect & raise alert for matched rules Sleep quality calculating users’ sleep quality Tremor detection Tremor detection based on devices’ Aggregator Aggregation (50hz to minutes / hours) Sample Parkinson Disease re Subscriber Parser Aggregator HBase Writer Analytics Manager Change Detection UnZip Real Time Rules Sleep Quality
  12. 12. STREAM Processing MANAGEMENT Layer (“Pigeon”)
  13. 13. • Core OS & Docker containers enable portability and ease of deployment anywhere • Enables the flexibility of choosing a set of desired containers based on a given use case requirements Easy Portability With Docker & Core OS Preconfigured containers ready to be loaded
  14. 14. • IoT data Ingestion goes beyond moving the data into the cloud • We have deployed a scalable and fault tolerance, multi-protocol pipeline that enables stream Analytics • Stream Analytics platform is leveraged for Other IoT projects Summary
  15. 15. Thank You!

×