Vertica the convertro way
•Download as PPTX, PDF•
5 likes•1,546 views
How we load into Vertica How we query Vertica. Check our Hadoop Loader and Hydro . Get couple of Tips from our experience.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (19)
Similar to Vertica the convertro way
Similar to Vertica the convertro way (20)
Recently uploaded
Recently uploaded (20)
Vertica the convertro way
- 1. 1
- 2. Vertica The Convertro way Zvika Gutkin Big data dbOps zvika.gutkin@convertro.com
- 3. Agenda Why Vertica How we Load How we query Not all good Vertica The Convertro waConvertro Who We are Tips
- 4. 0% 0% 0% 0% 0% 100% Last click model Convertro Vertica The Convertro wa
- 5. 100% 15% 10% 5% 10% 20% 40% Multi attribution Convertro Vertica The Convertro wa
- 6. Convertro Reports Analytics Dashboard Batch Dashboard Aggregation Regression R Vertica The Convertro wa
- 7. Convertro Batch Dashboard Active active Copy aggregated data Predictable customer experience Vertica The Convertro wa Client facing
- 8. How we Load Convertro Vertica The Convertro wa
- 9. Pro Tip HDFS Vs. Hadoop Vs. Application Loader Big Bulks Bring your files to your cluster Load from Several Nodes Convertro Pro Tip Pro Tip Vertica The Convertro wa
- 10. Unified Temp Table Target Table/Partition Stream COPY Number of parallel loads1 Number of parallel nodes2 Chunk size per loads3 TEMP TEMP TEMP MOVE PARTITIONS MOVE PARTITIONS Vertica The Convertro waConvertro
- 11. How we query Vertica The Convertro waConvertro
- 12. Denormalize Use DBD for encoding Events DBD Check query_events system table Vertica The Convertro waConvertro
- 13. MMM HydroImprovements Out of the box improvements Conversion Table ~2B/Month Oracle (raw store) 418GB / 147GB Vertica (column store) 21 GB Vertica The Convertro waConvertro
- 14. MMM HydroImprovements Measure, Measure, Measure! Queries Locks Resources … … Input Size Resources Errors Query … Trends Anomalies
- 15. Measure, Measure, Measure! Select M1,.. From… Where filter1 = ‘a’ and filter2 = ‘b’ Select M1,.. From… Where filter1 = ‘c’ and filter 2 = ‘b’ Select M1,.. From… Where filter1 = “” and filter2 = “” MMM HydroImprovements
- 16. MMM HydroImprovements Real Time ETR select A from B where C=‘D’ Business Logic Topology Sampling Lookup Aggregate Hydro Web Service
- 17. Not all good Vertica The Convertro waConvertro
- 18. High concurrency Deletes Updates Short queries with high concurrency. (Routable queries) Vertica The Convertro waConvertro
- 19. Vertica The Convertro waConvertro Tips
- 20. ROS CONTAINERS Vertica The Convertro waConvertro
- 21. ROS CONTAINERS Vertica The Convertro waConvertro
- 22. ROS CONTAINERS Vertica The Convertro waConvertro
- 23. ROS CONTAINERS Convertro Vertica The Convertro wa
- 24. ROS CONTAINERS Convertro Vertica The Convertro wa
- 25. ROS CONTAINERS Convertro Vertica The Convertro wa
- 26. ROS CONTAINERS New Table Convertro Vertica The Convertro wa
- 27. ROS CONTAINERS New Table Convertro Vertica The Convertro wa
- 28. ROS CONTAINERS Convertro Vertica The Convertro wa
- 29. Vertica The Convertro waConvertro Tips
- 30. Convertro Node Crash Slow Recover Process Checking Recovery Status Incremental recovery replay- delete Vertica The Convertro wa Many Deletes / Updates
- 31. Convertro Node Crash Slow Recover Process Checking Recovery Status Incremental recovery replay- delete Solution 1 Wait Vertica The Convertro wa Many Deletes / Updates
- 32. Convertro Node Crash Slow Recover Process Checking Recovery Status Incremental recovery replay- delete Solution 2 Set make_ahm_now Incremental By Containers Vertica The Convertro wa Many Deletes / Updates
- 33. Convertro Many Deletes / Updates Node Crash Slow Recover Process Checking Recovery Status Incremental recovery replay- delete Solution 3 Delete only one file Incremental By Containers Vertica The Convertro wa
- 34. Great Database Even Steph Curry can’t Do it all Keep It Simple Convertro
- 35. Great Database Even Lebron can’t Do it all Keep It Simple Convertro
- 36. Convertro Great Database Even Lebron can’t Do it all Keep It Simple
Editor's Notes
- The data for the model is in vertica Process in R Attribution run on hadoop based on the model that R calculated .
- Vertica : Raw : 150TB DB Size : 60 TB Top Clients: (top 8 above 1 TB ) Intuit 3 TB qvc 2 TB Dashboard : qvc => 260 GB => 8 Billion rows intuit => 200 GB => less than 8 Billion rows Vertica scan it and fetches a result in less than a second Complex analytics on a year takes about 5 seconds.
- Facebook 35 TB per hour 2 years ago !!
- 150K rows per sec Regular days 10 – 15 billion rows per day 40 billion rows per day New vertica feature => Vhash
- out of the box improvements => denormalize => data model changes are Game changer !!!.Vertica can handle big joins => merge joins
- out of the box improvements => denormalize => data model changes are Game changer !!!.Vertica can handle big joins => merge joins MMM => measure measure measure => data collector tables .
- out of the box improvements => denormalize => data model changes are Game changer !!!.Vertica can handle big joins => merge joins MMM => measure measure measure => data collector tables .
- out of the box improvements => denormalize => data model changes are Game changer !!!.Vertica can handle big joins => merge joins MMM => measure measure measure => data collector tables .
- out of the box improvements => denormalize => data model changes are Game changer !!!.Vertica can handle big joins => merge joins MMM => measure measure measure => data collector tables .
- out of the box improvements => denormalize => data model changes are Game changer !!!.Vertica can handle big joins => merge joins MMM => measure measure measure => data collector tables .
- great database => out of the box performance even lebron => don’t put billing on it right tool => hadoop loader,extended analytics ,flex table,udx Keep it simple => easy to debug easy to maintain
- great database => out of the box performance even lebron => don’t put billing on it right tool => hadoop loader,extended analytics ,flex table,udx Keep it simple => easy to debug easy to maintain
- great database => out of the box performance even lebron => don’t put billing on it right tool => hadoop loader,extended analytics ,flex table,udx Keep it simple => easy to debug easy to maintain