SFPUG Lightning Talk
•Download as PPTX, PDF•
1 like•509 views
This document discusses using distributed PostgreSQL databases to enable interactive analysis of analytics data captured by Heap. It describes how Heap captures large amounts of user data, but making this data interactive to query is challenging. The solution involves storing the denormalized analytics data across multiple PostgreSQL databases using CitusDB to enable distributed queries. Custom user-defined functions are also created to allow expressive analysis of the distributed data, such as tracking users through multi-step funnels.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
SFPUG Lightning Talk
- 1. Retroactive Analytics With Distributed PostgreSQL Dan Robinson Heap
- 2. ● Heap: web/iOS analytics that captures everything
- 7. ● Heap: web/iOS analytics that captures everything ● Making this interactive is hard!
- 8. app_id user_id properties HSTORE events HSTORE[] 12345 102756 email=>’coolguy@mail.com’, ab_test_grp=>’A’ ... 12345 300732 ab_test_grp=>’B’ ... 67890 628537 ... 49964 368868 utm_campaign=>’social’ ...
- 9. app_id user_id properties HSTORE events HSTORE[] 12345 102756 email=>’coolguy@mail.com’, ab_test_grp=>’A’ ... 12345 300732 ab_test_grp=>’B’ ... 67890 628537 ... 49964 368868 utm_campaign=>’social’ ... app_id user_i d properties HSTORE events HSTORE[] 75632 257186 ... ... 75632 120554 ... ... app_id user_i d properties HSTORE events HSTORE[] … … users_001 users_002 users
- 10. app_id user_id properties HSTORE events HSTORE[] 12345 102756 email=>’coolguy@mail.com’, ab_test_grp=>’A’ ... 12345 300732 ab_test_grp=>’B’ ... 67890 628537 ... 49964 368868 utm_campaign=>’social’ ... … … users_001 users_002 users app_id user_i d properties HSTORE events HSTORE[] 75632 257186 ... ... 75632 120554 ... ... SELECT COUNT(*) FROM users WHERE app_id = 12345 GROUP BY events[1]->'path'
- 11. users_001 users SELECT COUNT(*) FROM users WHERE app_id = 12345 GROUP BY events[1]->'path' SELECT COUNT(*) FROM users_001 WHERE app_id = 12345 GROUP BY events[1]->'path'
- 12. ● Denormalized → fast, no joins. ● Subqueries are just postgres. ● Add UDFs for more expressiveness.
- 13. funnel_events(events hstore[], pattern_array text[]) RETURNS int[] -- Returns an array with 1s corresponding to steps completed -- in the funnel, 0s in the other positions
- 14. funnel_events(events hstore[], pattern_array text[]) RETURNS int[] -- Returns an array with 1s corresponding to steps completed -- in the funnel, 0s in the other positions SELECT sum( funnel_events( events, ARRAY['"path"=>"/","object"=>"pageview"', '"type"=>"submit","hierarchy"=>like "%@form;#signup;%"'] ) ) AS "funnel_results" FROM users WHERE app_id = 12345
- 15. ● Denormalized schema. (No joins.) ● CitusDB to distribute queries. ● Express any analysis with UDFs.
- 16. Questions?