Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Vertica the convertro way

1,071 views

Published on

How we load into Vertica
How we query Vertica.
Check our Hadoop Loader and Hydro .
Get couple of Tips from our experience.

Published in: Data & Analytics
  • Be the first to comment

Vertica the convertro way

  1. 1. 1
  2. 2. Vertica The Convertro way Zvika Gutkin Big data dbOps zvika.gutkin@convertro.com
  3. 3. Agenda Why Vertica How we Load How we query Not all good Vertica The Convertro waConvertro Who We are Tips
  4. 4. 0% 0% 0% 0% 0% 100% Last click model Convertro Vertica The Convertro wa
  5. 5. 100% 15% 10% 5% 10% 20% 40% Multi attribution Convertro Vertica The Convertro wa
  6. 6. Convertro Reports Analytics Dashboard Batch Dashboard Aggregation Regression R Vertica The Convertro wa
  7. 7. Convertro Batch Dashboard Active active Copy aggregated data Predictable customer experience Vertica The Convertro wa Client facing
  8. 8. How we Load Convertro Vertica The Convertro wa
  9. 9. Pro Tip HDFS Vs. Hadoop Vs. Application Loader Big Bulks Bring your files to your cluster Load from Several Nodes Convertro Pro Tip Pro Tip Vertica The Convertro wa
  10. 10. Unified Temp Table Target Table/Partition Stream COPY Number of parallel loads1 Number of parallel nodes2 Chunk size per loads3 TEMP TEMP TEMP MOVE PARTITIONS MOVE PARTITIONS Vertica The Convertro waConvertro
  11. 11. How we query Vertica The Convertro waConvertro
  12. 12. Denormalize Use DBD for encoding Events DBD Check query_events system table Vertica The Convertro waConvertro
  13. 13. MMM HydroImprovements Out of the box improvements Conversion Table ~2B/Month Oracle (raw store) 418GB / 147GB Vertica (column store) 21 GB Vertica The Convertro waConvertro
  14. 14. MMM HydroImprovements Measure, Measure, Measure! Queries Locks Resources … … Input Size Resources Errors Query … Trends Anomalies
  15. 15. Measure, Measure, Measure! Select M1,.. From… Where filter1 = ‘a’ and filter2 = ‘b’ Select M1,.. From… Where filter1 = ‘c’ and filter 2 = ‘b’ Select M1,.. From… Where filter1 = “” and filter2 = “” MMM HydroImprovements
  16. 16. MMM HydroImprovements Real Time ETR select A from B where C=‘D’ Business Logic Topology Sampling Lookup Aggregate Hydro Web Service
  17. 17. Not all good Vertica The Convertro waConvertro
  18. 18. High concurrency Deletes Updates Short queries with high concurrency. (Routable queries) Vertica The Convertro waConvertro
  19. 19. Vertica The Convertro waConvertro Tips
  20. 20. ROS CONTAINERS Vertica The Convertro waConvertro
  21. 21. ROS CONTAINERS Vertica The Convertro waConvertro
  22. 22. ROS CONTAINERS Vertica The Convertro waConvertro
  23. 23. ROS CONTAINERS Convertro Vertica The Convertro wa
  24. 24. ROS CONTAINERS Convertro Vertica The Convertro wa
  25. 25. ROS CONTAINERS Convertro Vertica The Convertro wa
  26. 26. ROS CONTAINERS New Table Convertro Vertica The Convertro wa
  27. 27. ROS CONTAINERS New Table Convertro Vertica The Convertro wa
  28. 28. ROS CONTAINERS Convertro Vertica The Convertro wa
  29. 29. Vertica The Convertro waConvertro Tips
  30. 30. Convertro Node Crash Slow Recover Process Checking Recovery Status Incremental recovery replay- delete Vertica The Convertro wa Many Deletes / Updates
  31. 31. Convertro Node Crash Slow Recover Process Checking Recovery Status Incremental recovery replay- delete Solution 1 Wait Vertica The Convertro wa Many Deletes / Updates
  32. 32. Convertro Node Crash Slow Recover Process Checking Recovery Status Incremental recovery replay- delete Solution 2 Set make_ahm_now Incremental By Containers Vertica The Convertro wa Many Deletes / Updates
  33. 33. Convertro Many Deletes / Updates Node Crash Slow Recover Process Checking Recovery Status Incremental recovery replay- delete Solution 3 Delete only one file Incremental By Containers Vertica The Convertro wa
  34. 34. Great Database Even Steph Curry can’t Do it all Keep It Simple Convertro
  35. 35. Great Database Even Lebron can’t Do it all Keep It Simple Convertro
  36. 36. Convertro Great Database Even Lebron can’t Do it all Keep It Simple
  37. 37. Thank You zvika.gutkin@convertro.com https://github.com/Convertro/Hydro http://www.meetup.com/Tech-Talk-Teach/

×