AQP-as-a-middleware

1. Approximate Query Processing
using Verdict
Deepak Goyal @WalmartLabs

2. About me
Interests
• Distributed Processing and Database platforms
Present
• Customer Backbone, Walmart Labs, Bengaluru
Past
• Knowledge Graph, Bing, Microsoft, Hyderabad
• B. Tech. in CSE, International Institute of Information Technology, Hyderabad

3. Data Analytics
• Processing and exploitation of cleansed data generates information.
• Data analysis converts information to intelligence.
• Exploring various forms of data.
• Searching for new data insights.
• Hence, making better business decisions.

4. Email marketing @Walmart
• Email marketing
Dedicated email campaigns for specific customers based on intelligence
gathered from data analytics.
• Customer Segmentation
Dividing a broad business market into smaller targeted audience segments for
email advertising.
• Segmentation base as a combination the following facets for example
Demographics age, gender, income, etc.
Geographic country, state, city, postal code, etc.
Psychographics lifestyle, social, personality, etc.

5. The Problem: Slow and Costly Data Analytics
Contributing Factors
• Limited cluster resources
• Long running analytic jobs
• Large number of short/medium length analytic jobs
• Large volumes of data
• Slow response times
implies Slow data analytics

6. The solution: Verdict
• A next generation approximate query processor
• Built upon the theories of approximate query processing (AQP)
• Based on the novel architecture of AQP-as-a-middleware
• Can reliably estimate many important statistics from a small fraction
of the entire data
• Exploits the state-of-the-art techniques from statistics
• Paves the way from slow exact results to fast resource-efficient good
approximates

7. AQP-as-a-middleware

8. Runs all on SQL-based engines

9. Approximate Query Processing using Verdict
• Reduces the query processing cost
• Fewer resources
• No changes to the application
• Compatible with all SQL engines
• Highly accurate
• Faster results
• Enable continuous data exploration

10. Notes and References
• Verdict is developed primarily by the database group at the University
of Michigan
• Links
• http://verdictdb.org/
• https://github.com/mozafari/verdict
• Discuss further
• Mailing List: verdict-user@umich.edu
• Walmart folks
• Deepak Goyal deepak.goyal@walmartlabs.com
• Giridhar Addepalli gaddepalli@walmartlabs.com
• Chirag Singla cchirag@walmartlabs.com

11. Thank you.
Q&A