Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enabling New Business Capabilities with Cloud-based Streaming Data Architecture - StampedeCon Big Data Conference 2017

94 views

Published on

Using big data isn’t about doing the same things we’ve always done just with different technologies. The technology advances that we’ve chosen to label as big data create the opportunity for wholly new kinds of solutions. Two of the key advances that are enabling new business capabilities are cloud-based data management platforms and streaming data processing and analytics.

In this session, Paul Boal will drill into the cloud-based streaming data architecture that has made possible EVŌ, a new breakthrough health and wellness platform. EVŌ uses a game-changing approach that leverages over 60 billion data points and a predictive analytics engine to intervene BEFORE someone becomes critically ill. All of this is possible by leveraging data from smartphones and wearable fitness devices along with advanced analytics which then help users develop and sustain positive behaviors. Attendees will learn how to create a cloud- based architecture that can receive data, apply multiple layers of dynamic business rules, and drive alerts and decisions through real-time stream processing using technologies including web services, Amazon DynamoDB and Kinesis, Drools, and Apache Spark.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Enabling New Business Capabilities with Cloud-based Streaming Data Architecture - StampedeCon Big Data Conference 2017

  1. 1. New Business Capabilities through Streaming Data Architectures A Case Study from EVŌ – Paul Boal, Amitech Solutions July 25, 2017
  2. 2. Paul Boal paul.boal@amitechsolutions.com @paulboal Paul has been architecting healthcare analytics solutions for over 15 years, implementing a range of technologies from traditional data warehouses to Hadoop-based solutions, advanced analytics, and real-time clinical data integration. Paul is now Vice President of Delivery with Amitech Solutions, where he helps teams deliver big data solutions for healthcare. 2
  3. 3. Starting from batch 3 • Those systems sync once a day. • CMS sends us claim files every 30 days. • We get new benchmarks ratings annually. • Our vendor sends updated prices lists quarterly. • We balance that process at the end of the month. • We run weekly extracts to send to that partner. • We load and aggregate the data warehouse daily.
  4. 4. The results of batch 4 • The average person should drink 8 glasses of water a day. • The average person should take 10,000 steps per day. • The average person should get 6-8 hours of sleep per night. • The average person should…
  5. 5. Getting healthcare right Have the right information at the right time, and deliver it to the right person in the right way in the right moment. 5
  6. 6. The EVŌ Concept Wellness is a personal journey 6
  7. 7. 7 Have the right information at the right time, and deliver it to the right person in the right way in the right moment.
  8. 8. Right Information: Multiple Sources • Consumer-grade reliability • As much insight as possible from as little information as possible • Accelerometer • Heartrate • Steps • Activity • Sleep • Augment device data with personal feedback • Benchmark against population 8
  9. 9. Right Time: Data Acquisition • Data integration models • Device à Vendor Mobile App à Vendor Cloud à EVŌ Cloud • Device à EVŌ App à EVŌ Cloud • Device EVŌ App à Vendor Mobile SDK à EVŌ Cloud 9
  10. 10. Right Time: Data Acquisition • Data integration models • Device à Vendor Mobile App à Vendor Cloud à EVŌ Cloud • Device à EVŌ App à EVŌ Cloud • Device EVŌ App à Vendor Mobile SDK à EVŌ Cloud 10
  11. 11. Right Time: Data Acquisition • Data integration models • Device à Vendor Mobile App à Vendor Cloud à EVŌ Cloud • Device à EVŌ App à EVŌ Cloud • Device EVŌ App à Vendor Mobile SDK à EVŌ Cloud 11
  12. 12. Right Time: Data Processing • Processing is primarily data-driven rather than clock-driven • Processing intensive because the engine does “something” on every data entry • Take care not to create infinite loops • Use clock-based processing to look for missing data 12 Web Services NoSQL DB w/ Message Queue Streaming Data Processing Engine Missing Data Check
  13. 13. Right Person 13 Family & Friends Individual AdministratorsCare Managers • Push notifications • Mobile app • Encouragement • Motivation • Warnings • Alerts • Promote positive reinforcement • Outreach • Monitor progress • Identify risks • Referrals • Engagement • Program value • Cost savings • Adjust program
  14. 14. Right Way • Psychographic questions to understand how best to interact with a user. • Personalize notification rules based on user persona. • Create custom messages that use language and tone that will persuade this particular user. • Monitor interaction with messaging to notice what has an impact and what is ignored. • Adapt to persona changes over time. 14
  15. 15. Right Moment • Push Notification • Rules Engine • Range of times when each message can be delivered • Message timeout limit • Persona Specific • How many notifications • How much detail • Calendar Integration • Wait until after meetings 15 EVŌEVŌ Your house is in sleep mode. Time for you to start powering down, too. Press for details. Good morning! Here’s how you slept last night 4.5hrs deep / 7hrs total Press for details. 9:00Thursday, December 15 6:00Thursday, December 15
  16. 16. Right Moment Right Way Right Time Right Person Right Information The Right Architecture for EVŌ 16
  17. 17. The Right Technologies for EVŌ 17
  18. 18. Technical Lessons: Data Capture • Technologies: • Java web services API (Swagger) • AWS DynamoDB (NoSQL, wide, dynamic columns) • JSON data structures • Light-weight source data capture – let clients get in and out quickly. • Flexible data model • Allow clients to change without backend dependency. • Add new clients without impacts to existing clients. • Defer data mapping – back-end data integration and standardization. 18 "key" = "identifiers", "datetime" = YYYYMMDDhhmmss, "data" = { JSON LOB }
  19. 19. Technical Tips: Data Streaming • Technologies: • AWS DynamoDB Streams / Kinesis • Apache Flume • Light-weight ledger • Guaranteed to be first-in first-out within a shard. • Consuming process is responsible for check-pointing. • Data-driven • Consuming process defines the processes. • Consuming processes have to consume data in a timely fashion. • Built-in to database – every write creates a message. 19
  20. 20. Technical Tips: Data Processing • Technologies: • AWS EMR / Apache Spark • RedHat / JBoss Drools • Responsible for queue management • Spark is a micro-batch model, not single transaction. • Consuming process has to checkpoint what has been processed. • Watch out for infinite loops • If a process writes back to it’s own data source, filter! • Quickly filter records that don’t need to be processed. • Break up components – Not one monolithic process / not microservices. • Integrate with other tools – e.g. Drools 20
  21. 21. 21 1 Right Information • What information do you have access to? • What would it take to get more information? • What is the best balance of effort to value? 2 Right Time • How soon can each data source be available? • What are your options for where to process data? • Which processes are data-driven, user-driven, and clock-driven? 3 Right Person • Think about what role each possible user plays in the process. • What does each user need do their job? • Are there secondary users who might need information, too? 4 Right Way • What do I know about how this user would like to get information? • What words and images will this user respond to? • Leave reports and information access to secondary screens. 5 Right Moment • Can you provide timely feedback to this user? • What happens to information you want to communicate but can’t? • What information can wait for another time?
  22. 22. What can you make right? 22

×