Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cassandra Materialized Views

7,602 views

Published on

New for 3.0, Materialized Views deliver automated server-side denormalization. This is the presentation given at Cassandra Summit 2015.

Published in: Software

Cassandra Materialized Views

  1. 1. Materialized Views Carl Yeksigian
  2. 2. What are Materialized Views? • Two copies of the data using different partitioning and placed on different replicas • Automated, server-side denormalization of data • Native Cassandra read performance • Write penalty, but acceptable performance
  3. 3. Basic Rules of Data Modeling, Refresher • Best practice: Denormalization • Start by understanding the queries you need • Create a table for each query
  4. 4. Why is Denormalization Hard? • Implemented by every application • No guarantees on performance or consistency • Updates to existing rows require cleanup, read-before-write
  5. 5. Denormalization Example: User Playlists Queries • All Songs for a given playlist • Track Users who like the same Artist • Find most recently played song
  6. 6. Denormalization in Practice CREATE TABLE playlists (user_name text, playlist_name text, song_id text, artist_name text, last_played timestamp) SELECT song_id FROM playlists WHERE user_name=‘carl’ AND playlist_name=‘jams’ SELECT COUNT(song_id) FROM playlists WHERE artist_name=‘Weezer’ SELECT last_played, song_id FROM playlists WHERE user_name=‘carl’ AND playlist_name=‘jams’ ORDER BY last_played DESC
  7. 7. Denormalization in Practice CREATE TABLE playlists (user_name text, playlist_name text, song_id text, artist_name text, last_played timestamp, PRIMARY KEY (user_name, playlist_name, song_id)) CREATE TABLE artists_to_playlists (artist_name text, user_name text, playlist_name text, song_id text, PRIMARY KEY (artist_name, user_name, playlist_name, song_id))
  8. 8. Denormalization in Practice CREATE TABLE last_played (user_name text, playlist_name text, last_played timestamp, song_id text, PRIMARY KEY (user_name, playlist_name, last_played, song_id)) CLUSTERING ORDER BY (last_played DESC)
  9. 9. Denormalization in Practice: Inserts BEGIN BATCH INSERT INTO playlists (user_name, playlist_name, song_id, artist_name, last_played) VALUES (‘carl’, ‘jams’, ‘Undone’, ‘Weezer’, ‘2015-09-24 09:00’); INSERT INTO artists_by_playlist (artist_name, user_name, playlist_name, song_id) VALUES (‘Weezer’, ‘carl’, ‘jams’, ‘Undone’); INSERT INTO last_played (user_name, playlist_name, last_played, song_id) VALUES (‘carl’, ‘jams’, ‘Undone’, ‘2015-09-24 09:00’); APPLY BATCH;
  10. 10. Denormalization in Concept: Updates UPDATE playlists SET last_updated=now() WHERE user_name=‘carl’ AND playlist_name=‘jams’ AND song_id=‘Undone’ DELETE FROM playlists WHERE user=‘carl’
  11. 11. Manual Denormalization with updates Client Batchlog Base Table View Table Coordinator
  12. 12. Manual Denormalization with updates Client Batchlog Base Table View Table Coordinator Query Existing Data
  13. 13. Manual Denormalization with updates Client Batchlog Base Table View Table Coordinator Query Existing Data Query Existing Data
  14. 14. Manual Denormalization with updates Client Batchlog Base Table View Table Coordinator Query Existing Data Query Existing Data Return Existing Data Return Existing Data
  15. 15. Manual Denormalization with updates Client Batchlog Base Table View Table Coordinator Write New Values
  16. 16. Manual Denormalization with updates Client Batchlog Base Table View Table Coordinator Write New Values Write New Values
  17. 17. Manual Denormalization with updates Client Batchlog View Table Coordinator Write New Values Write New Values Base Table Write New Values
  18. 18. Manual Denormalization with updates
  19. 19. Manual Denormalization Limitations • Updates need to keep view in sync, including tombstoning previous values • How to keep the view and base in sync on failure? • Contentious updates can potentially cause extra values • Your application doesn’t always know what is a update or an insert (i.e. upsert)
  20. 20. Manual Denormalization: Contentious Updates Client 1 Cassandra Client 2 playlists:(‘carl’, ‘jams’, ‘Undone’, 2015-09-24 9:00) last_played:(‘carl’, ‘jams’, 2015-09-24 9:00, ‘Undone’) Query existing last_played Query existing last_played
  21. 21. Manual Denormalization: Contentious Updates Client 1 Cassandra Client 2 playlists:(‘carl’, ‘jams’, ‘Undone’, 2015-09-24 9:02) last_played:(‘carl’, ‘jams’, 2015-09-24 9:01, ‘Undone’) last_played:(‘carl’, ‘jams’, 2015-09-24 9:02, ‘Undone’) Update last played time to 9:02 last_played: 2015-09-24 9:00 last_played: 2015-09-24 9:00 Update last played time to 9:01
  22. 22. Manual Denormalization Limitations
  23. 23. Materialized Views • Provide automated server-side denormalization • No need for read-before-write on the application side • Simplify application code • Provide safer guarantees
  24. 24. Materialized Views: Guarantees • If a write is acknowledged, at least CL number of base and view replicas will receive the write • If a write is actually an update, the previous value will be cleaned up in the view • Even with contentious updates, view synchronized with base for each update • Takes care of deletions properly • When a base table is repaired, the data will also be inserted into the view • TTL’d base data will remain TTL’d in view
  25. 25. Why Not Just Use Secondary Indexes? • We can get most of the same functionality by using secondary indexes • Secondary indexes query each node, not being able to use the ring • On a node, not a single access
  26. 26. Secondary Indexes: Query Pattern Client
  27. 27. Secondary Indexes: Query Pattern Client
  28. 28. Secondary Indexes: Query Pattern Client
  29. 29. Materialized Views: Query Pattern Client
  30. 30. Materialized Views in Practice CREATE TABLE playlists (user_name text, playlist_name text, song_id text, artist_name text, last_played timestamp, PRIMARY KEY (user_name, playlist_name, song_id))
  31. 31. Materialized Views in Practice CREATE MATERIALIZED VIEW artist_to_user AS SELECT song_id, user_name FROM playlists WHERE song_id IS NOT NULL AND playlist_name IS NOT NULL AND user_name IS NOT NULL AND artist_name IS NOT NULL PRIMARY KEY (artist_name, user_name, playlist_name, song_id)
  32. 32. Replica Placement user_name:carl
  33. 33. Replica Placement artist_name:Weezer user_name:carl
  34. 34. Materialized Views in Practice • On creation, a new materialized view will be populated with existing base data • Each node tracks the completion of the build independently SELECT * FROM system.built_views WHERE keyspace=‘ks’ AND view_name=‘view’
  35. 35. Materialized Views in Practice CREATE MATERIALIZED VIEW last_played AS SELECT last_played, song_id FROM playlists WHERE user_name IS NOT NULL AND last_played IS NOT NULL AND song_id IS NOT NULL PRIMARY KEY (user_name, playlist_name, last_played, song_id) CLUSTERING ORDER BY (last_played DESC)
  36. 36. Materialized Views: Performance • https://github.com/tjake/mvbench • Uses java-driver to simulate MV and manual denormalization
  37. 37. Materialized Views: Performance (ops/s) Manual Denormalization Materialized Views
  38. 38. Materialized Views: Performance (p95 latency) Manual Denormalization Materialized Views
  39. 39. • Adding WHERE clause support (#9664) CREATE MATERIALIZED VIEW carls_last_played AS SELECT last_played, song_id FROM plays WHERE user_name=‘carl’ PRIMARY KEY (last_played, song_id) • Knowing when a view is completely finished building without querying each node (#9967) • Insert only tables can skip read-before-write, lock acquisition (#9779) Materialized Views: Future Features
  40. 40. write<p1, p2, c1, c2, v1> Node A Node B Node C Coordinator Base View Node D write<p1, p2, c1, c2, v1> del<v0, p1, p2, c1, c2> Client write<v1, p1, p2, c1, c2> BL
  41. 41. Materialized Views: Under the Hood • If update is partial, we will reinsert data from the read when generating a new row • If no tombstone generated, only new columns are written to view
  42. 42. Materialized Views: The Edge • Materialized Views have different failure properties than the rest of the system • Data from a single base replica can be on many view replicas
  43. 43. Materialized Views: The Edge • When data is lost on all replicas of the base table, it can not be cleaned up in the view (#10346) • No read repair between the base and the view table • Repair of base table will clean up view • Requires local read-before-write. • If you will never ever update/delete use manual MVs
  44. 44. Materialized Views: The Edge • An update from a base table is asynchronously applied to the view, so it is possible there will be a delay • A MV on a low-cardinality table can cause hotspots in the ring, overloading some nodes
  45. 45. Thanks

×