Highload2o013 osipv

308 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
308
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Highload2o013 osipv

  1. 1. Популярные алгоритмы хранения данных на диске Konstantin Osipov, kostja@tarantool.org October 28th, 2013
  2. 2. Случай в квадрате 36-80 • • • B-tree – most popular disk-based data structure B-tree balances INSERT, UPDATE and SELECT speed DELETEs can be slow
  3. 3. СУБД быстрая, настраивать надо уметь
  4. 4. B-tree: внутреннее устройство
  5. 5. Что означает сache-oblivious?
  6. 6. Что означает сache-oblivious? (2) BLOCK­MULT(A,B,C,n): 1 for i = 1 to n/s do: 2    for j = 1 to n/s do: 3         for k = 1 to n/s do: 4             ORD­MULT(Aik, Bkj, Cij, s)
  7. 7. LSM-tree: архитектура
  8. 8. LSM-tree: архитектура (2)
  9. 9. LevelDB: архитектура
  10. 10. LevelDB: insert RPS
  11. 11. LSM-tree: применение ● Данные с разной степенью актуальности – – Стена в соцсети – Чаты – ● Ленты сообщений События Сегрегация данных – Данные в LSM, индекс в памяти
  12. 12. COLA: архитектура O(logB(N)) vs. O(logB(N)/B)
  13. 13. PUT(37), PUT(16) Self-Balancing Tree Memory Disk WAL:
  14. 14. 16 37 Self-Balancing Tree Memory Disk WAL: 37, 16
  15. 15. 7 41 Self-Balancing Tree Memory 16 37 Sorted String Table WAL: 41, 7, 37, 16 Disk
  16. 16. Memory 7 37 7 16 37 41 WAL: 41, 7, 28, 16 Disk
  17. 17. 10 28 Memory 7 37 Disk 7 16 37 41 WAL: 10, 28, 41, 7, 37, 16
  18. 18. Memory 10 28 7 16 37 41 WAL: 10, 28, 41, 7, 37, 16 Disk
  19. 19. 2 47 Memory 10 28 Disk 7 16 37 41 WAL: 47, 2, 10, 28, 41, 7, 37, 16
  20. 20. 2 28 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  21. 21. 6 49 2 28 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  22. 22. 6 49 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  23. 23. 23 32 6 49 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  24. 24. 6 32 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  25. 25. 30 45 Memory 6 32 Disk 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  26. 26. 14 38 Memory 30 45 Disk 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 38, 14, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  27. 27. 6 10 Memory 2 30 Disk 2 14 30 41 2 2 6 7 14 23 30 37 41 47 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  28. 28. Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  29. 29. GET(16) Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  30. 30. GET(16) Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  31. 31. BitCask: архитектора AOF
  32. 32. BitCask: архитектура keydir
  33. 33. Sophia: архитектура Memory 15 26 40 84 Key-value index 10, 25 26, 31 39, 85 86, 96 Page index split Disk 10, 15, 16, 25 26, 31 39, 40, 84, 85 86, 96
  34. 34. 39, 16, 85, 96 Insert Memory Key-value index Disk WAL Page index
  35. 35. Insert Memory 16 39 85 96 Key-value index Disk Page index
  36. 36. 31, 25, 10, 86 Insert 16, 96 Memory Key-value index Disk Page index WAL 16, 39, 85, 96
  37. 37. Memory 10 25 31 86 Key-value index 16, 96 Page index merge Disk 16, 39, 85, 96
  38. 38. Memory 10 25 31 86 Key-value index 10, 31 39, 96 Page index split Disk 10, 16, 25, 31 39, 85, 86, 96
  39. 39. 15, 26, 40, 84 Insert 10, 31 Memory Key-value index Disk WAL 39, 96 Page index 10, 16, 25, 31 39, 85, 86, 96
  40. 40. Memory 15 26 40 84 Key-value index 10, 31 39, 96 Page index merge Disk 10, 16, 25, 31 39, 85, 86, 96
  41. 41. Memory 15 26 40 84 Key-value index 10, 25 26, 31 39, 85 86, 96 Page index split Disk 10, 15, 16, 25 26, 31 39, 40, 84, 85 86, 96
  42. 42. Эпилог: choose your db wisely ?
  43. 43. Links ● ● ● ● ● ● ● ● ● ● Bitcask A Log-Structured Hash Table for Fast Key/Value Data, Justin Sheehy David Smith with inspiration from Eric Brewer The Log-Structured Merge-Tree (LSM-Tree) Patrick O'Neil , Edward Cheng, Dieter Gawlick, Elizabeth O'Neil Cache-Oblivious Algorithms by Harald Prokop (Master theses) Space/time trade-offs in hash coding with allowable errors, Burton H. Bloom Data Structures and Algorithms for Big Databases, Michael A. Bender Stony Brook & Tokutek Bradley C. Kuszmaul (XLDB tutorial) http://github.com/pmwkaa/sophia, http://sphia.org http://codecapsule.com/2012/12/30/implementing-a-key-value-store-part-3-comparative-analysis-of-the-ar http://stackoverflow.com/questions/6079890/cache-oblivious-lookahead-array http://www.youtube.com/watch?v=88NaRUdoWZM(Tim Callaghan: Fractal Tree indexes) http://code.google.com/p/leveldb/downloads/list

×