Downloaded 35 times
Uploaded byNoSQLmatters
Gregorry Letribot - Druid at Criteo - NoSQL matters 2015
The document discusses the use of Druid Elixir for analytics, emphasizing its capabilities for real-time ingestion and fast querying on large datasets, particularly in a NoSQL context. It highlights the importance of scalability, availability, and low-latency insights for modern data analytics, while cautioning users about potential pitfalls in data modeling and system limitations. The performance metrics provided indicate high throughput and efficiency, with the system achieving rapid aggregate displays over significant data volumes.
More Related Content
Similar to Gregorry Letribot - Druid at Criteo - NoSQL matters 2015
Gregorry Letribot - Druid at Criteo - NoSQL matters 2015
- 1.
- 2.
- 3.
- 4. unique internet usersper month
- 5.
- 6.
- 7. Analytics the oldway CPOP
- 8.
- 9. Once upon atime, in an SQL Galaxy.. « Guys, database whatever_DB contains 3B rows Disks are full, it needs a purge Server will be reinstalled and will host only the last 30 days » « Well, talk with product »
- 10. SQL limit is reachedProductworking, but infrastructure falling apart
- 12. We want more! More dimensions User centric Interactive Realtime
- 13. NOSQL ? Precomputing givesfast queries Not flexible ! Scales exponentially on dimensions
- 14. Suming up Food forBI Interactive, sub-second insights Arbitrarily drill into data Scalability, availability…
- 15.
- 16. Built for analytics Scalableand Available Real-time ingestion and Queries
- 17. Read-oriented store Column oriented In Memory Fast Filtering Segment based
- 18.
- 19.
- 20.
- 21.
- 22.
- 23. Back to reallife Data workflow Real-time or Batch ingestion Lambda architecture !
- 24.
- 25. Columnar Store Only reads relevantdataIphone Google Computer 0.1€08:12:37 Android Yahoo Cloth 0.2€08:12:38 Select sum(cost) where Device = Iphone
- 26. High compressi on Wacken Hellfest Fall of Summer Wacken Hellfest 1 2 3 1 2 Metadata: Wacken=> 1 Hellfest=> 2 Fall of summer=> 3
- 27. Inverted indexWacken 1,0,0,0,1 Hellfest 0,1,0,1,0 Fallof Summer 0,0,1,0,0 Wacken Hellfest Fall of Summer Wacken Hellfest Fast binary operations
- 28. Don’t ! « …WHERE value >= 10 » cardinality 1500 1490 binary OR…
- 29. Userflow Unique user countain’t only an interview exercise !
- 30. Sketching algorithmsHyperLogLog Approximate uniquecount Extreme storage reduction Constant time computation
- 31.
- 32.
- 33. Performances No downtimein 6 months Aggregate displays, clicks, sales & revenue generated for our biggest advertiser grouped by device over 7 months = 197 ms
- 34. Performances According to metamarkets 33Mrows per second per core Scaled up to 26B rows per second 10k event per second ingestion per node
- 35. What’s wrong ? Be carefull with your data model Immutable is.. Immutable ! No joins, no full sql capabilities A couple of bugs.. But very active and friendly team !
- 36.
- 37.