[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics
•Download as PPTX, PDF•
2 likes•1,476 views
This document summarizes a presentation about big data analytics solutions from Think Big Analytics and Infochimps. It discusses using their platforms together to power applications with next-generation big data stacks. It highlights case studies, architecture diagrams, and polls to demonstrate how their services can accelerate time to value through a combination of data science, engineering, strategy, and hands-on training and education.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to [Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics
Similar to [Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics (20)
More from Infochimps, a CSC Big Data Business
More from Infochimps, a CSC Big Data Business (13)
Recently uploaded
Recently uploaded (20)
[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics
- 1. Measure Twice, Build Once #RTanalytics Douglas Moore Principal Consultant & Architect May 2013 Dhruv Bansal CSO
- 2. #RTanalytics | 2 About Us Next generation Big Data stack to power your applications Data science and engineering services that accelerate time to value Douglas Moore Principal Consultant & Architect Dhruv Bansal CSO & Co-Founder
- 3. #RTanalytics | 3 Agenda Think Big Use Case Infochimps Cloud: Streams, Queries, Batch Think Big & Infochimps Together Measure Twice, Build Once Understand the problem Model the solution Test locally Grow the infrastructure
- 5. #RTanalytics | 5 Poll Very Advanced 18% Advanced 19% Not Advanced 45% Not Started 18% RESULTS: How advanced is your organization's approach to Big Data?
- 6. #RTanalytics | 6 Accelerating Your Time to Value Strategy and Roadmap IMAGINE Training and Education ILLUMINATE Hands-On Data Science and Data Engineering IMPLEMENT Leading Provider of Data Science and Engineering Services
- 7. #RTanalytics | 7 Use Cases - Scale batch analysis pipeline - Generate lively stats - Recommendations - Better Predictions • #page views in next 30 days? Environment - AWS - Version 1 already in production Project Plan - 8-9 weeks - Combined Data Engineering + Data Science Engagement - Staff • 1 Arch + 1 PM • 1 Data Engineer • 2 Data Scientists • 3 Client Engineers The Beauty of Predictive Analytics
- 8. #RTanalytics | 8 Predictive Model Design & Build Process - Listening & Learning - Discovery (Digging through the data) - Creating a Research Agenda - Testing & Learning Production Quality Predictive Model Development - Data Cleansing, Aggregations, Conditioning - Predictive Model Training Process - Predictive Model Execution Process Challenges: - What functional forms predict future impression counts given counts up to time T? - Robust estimators, like medians rather than means, to cope with outliers - How do we distinguish between new articles, versus old articles we're seeing for the first time? - How well do impression counts correspond to real humans? Predictive Analytics Process
- 9. #RTanalytics | 9 Better end-user experience - View an ad, see the counter move. Need to catch fast moving events - Content half life measured at 3 hours (H Mason: http://bit.ly/nu7IDw) - Path to additional real-time capabilities - Example: Trend analysis to recommend ‘hot’ articles. Why Real-time?
- 10. #RTanalytics | 10 Overall Architecture NoSQL Memcache (Tuple fail tracking) Queue Hadoop Ad Serving LB Edge Edge Impression S3 S3 S3 DFS Archive Logs Management Server LB Edge Edge Relational Store Ad Management Ad Selling Storm - Queue Management - Simple Bot Filtering - Real-time Bucketization - Performance Counters - Event Logging View Ad Cleansing Model Training Recommendations Events Monitoring & Alerting (Metrics, Alarms, Notifications) Model Parameters getPrediction Performance Counters Impression Buckets
- 11. #RTanalytics | 11 Analytics Architecture Storm Web Server Time Series BucketBolt Simple Bot Annotator DFS Adapter Impression Spout Time Series Buckets (Batch) Time Series Buckets (Realtime) Impression Prediction Predictive Model Parameters Impressions Impressions Impressions Hadoop Impression Bucketization Predictive Model Training NoSQL Bolt Time
- 12. #RTanalytics | 12 Analyze Massive Historical Data Set Analyze Recent Past Realtime Prediction Solution Approach Historical Data Set = S3 Analyze = Hadoop + Pig + R Recent Past = Storm + NoSQL Analyze = R + Web Service
- 14. #RTanalytics | 14 Poll Less than 30 days 8% Less than 90 days 54% More than 6 months 38% RESULTS: Say you are building a Big Data project, which time frame would you want to build a production solution?
- 15. #RTanalytics | 15 Any Data Any Analytics Any Cloud
- 16. #RTanalytics | 16 Data Flow Architecture 3/18/2022
- 17. #RTanalytics | 17 Inside Cloud::Streams
- 18. #RTanalytics | 18 Twitter Gnip Powertrack Facebook Gnip EDC Blogs Moreover Metabase TV Transcription Radio Transcription Print Transcription New Media Data Sources Traditional Media Data Sources Traditional & Social Media Listening Platform 3/18/2022 Full Example
- 20. #RTanalytics | 20 Poll Hadoop 36% Queries 35% Real-time 29% Which element of the Big Data stack is most important to you?
- 21. #RTanalytics | 21 Don’t Build it Yourself 55% of enterprise Big Data projects fail* *According to a December 2012 survey of 300 IT organizations by SSWUG 5% 9% 9% 77% Project Costs by Function Compute Software Operations Staff Engineering Staff
- 22. #RTanalytics | 22 How Do We Compare to the Competition? Competition Think Big & Infochimps Speed 6+ months to value 30 days to value Experience New college grads Few successful implementations Advanced Degrees & Published Authors Quality Offshore Onshore, Managed Service Proven Learn on your dime Blue Chip Customers Methodology Waterfall Agile, test & learn
- 24. #RTanalytics | 24 Let’s continue the conversation! infochimps.com/demo thinkbiganalytics.com/about/contact
Editor's Notes
- We’ll leave this slide up in the time before we start the webinar.
- Douglas & Dhruv: Introduce ourselves and our companies.
- Dhruv
- Which element of the Big Data stack is most important to you? Hadoop Databases (SQL, NoSQL) Real-time (Storm, Kafka, Flume, Esper)
- Which element of the Big Data stack is most important to you? Hadoop Databases (SQL, NoSQL) Real-time (Storm, Kafka, Flume, Esper)
- DOUGLAS Thanks, Dhruv. Thanks for your time today everyone. Excited to share how Think Big Analytics can help you make big data come alive to allow you to accelerate your time to value. In case you don’t know us, Think Big Analytics provides data science and engineering services that create value from Big Data We help you IMAGINE your possibilities for big data and identify a plan for profitable projects We ILLUMINATE your team through hands-on training and education on the latest in big data technologies We then help you IMPLEMENT your analytics plan with hands-on data scientists and engineers who rapidly build data solutions to deliver value.
- Douglas Predictive analytics is helping companies generate competitive advantages and real value. I want to walk you through one project we completed for a 2012 SXSW tech finalist, which is an online ad publishing company. My comments will demonstrate how we used certain big data technologies to enhance the company’s product offering so they could charge their advertisers premium rates. Let me first explain the challenge this company addresses: Let’s say you own a local Mexican restaurant and you want to gear up for Cinco De Mayo, by putting adds into your local online newspaper. The paper is running an article about the planned festivities in the nearby park. The restaurant owner buys banner ads with the newspaper, goes to the page about the festival, and looks for her advertisement? Clicks refresh, hits refresh again and again. She might never see her ad. The company plans on solving this problem, by letting the ad buyer glue their ad to a piece of content. Taking ownership of that content, promoting it, and being satisfied that her family, friends and customers can see her ad and coupon.
- Douglas Creating a research agenda What features to use? Content or Behavior based? What features have predictive value? Which functional forms will work best? How successful is the prediction? Will the approach scale?
- Douglas Roll your own – Complicated, brittle, scalable, loss of focus. Multi-node scaling (which requires message routing, load balancing etc)
- For this customer, the overall architecture looked like this Netty edge server Change DynamoDB to NoSQL
- And Storm supported the analytics architecture by…. The result was….
- Douglas The theory is that one can analyze and mine a massive historical data set in batch at your leisure and create a parameterized predictive model. In real time one is collecting the parameters, and upon request we can deliver the prediction by executing the training model over the latest parameters Not unlike the Lambda Architecture described by Nathan Marz of Twitter.
- How advanced is your organization’s approach to Big Data? Not Advanced Advanced Very Advanced Not Started
- How advanced is your organization’s approach to Big Data? Not Advanced Advanced Very Advanced Not Started
- Dhruv Any Data Any Analytics Any Cloud
- Dhruv
- Dhruv [NOTE: ADD INFOCHIMPS LOGO ON HEADER]
- Dhruv
- ]NOTE: ADD INFOCHIMPS LOGO ON HEADER] If you are engaged in a Big Data project, over what timeframe are you interested in deploying a production solution? Less than 30 days Less than 90 days More than 6 months
- ]NOTE: ADD INFOCHIMPS LOGO ON HEADER] If you are engaged in a Big Data project, over what timeframe are you interested in deploying a production solution? Less than 30 days Less than 90 days More than 6 months
- Dhruv [NOTE: ADD INFOCHIMPS LOGO ON HEADER]
- Dhruv [NOTE: ADD INFOCHIMPS LOGO ON HEADER]
- Dhruv & Douglas