Highload2o013 osipv
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
255
On Slideshare
255
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
4
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Популярные алгоритмы хранения данных на диске Konstantin Osipov, kostja@tarantool.org October 28th, 2013
  • 2. Случай в квадрате 36-80 • • • B-tree – most popular disk-based data structure B-tree balances INSERT, UPDATE and SELECT speed DELETEs can be slow
  • 3. СУБД быстрая, настраивать надо уметь
  • 4. B-tree: внутреннее устройство
  • 5. Что означает сache-oblivious?
  • 6. Что означает сache-oblivious? (2) BLOCK­MULT(A,B,C,n): 1 for i = 1 to n/s do: 2    for j = 1 to n/s do: 3         for k = 1 to n/s do: 4             ORD­MULT(Aik, Bkj, Cij, s)
  • 7. LSM-tree: архитектура
  • 8. LSM-tree: архитектура (2)
  • 9. LevelDB: архитектура
  • 10. LevelDB: insert RPS
  • 11. LSM-tree: применение ● Данные с разной степенью актуальности – – Стена в соцсети – Чаты – ● Ленты сообщений События Сегрегация данных – Данные в LSM, индекс в памяти
  • 12. COLA: архитектура O(logB(N)) vs. O(logB(N)/B)
  • 13. PUT(37), PUT(16) Self-Balancing Tree Memory Disk WAL:
  • 14. 16 37 Self-Balancing Tree Memory Disk WAL: 37, 16
  • 15. 7 41 Self-Balancing Tree Memory 16 37 Sorted String Table WAL: 41, 7, 37, 16 Disk
  • 16. Memory 7 37 7 16 37 41 WAL: 41, 7, 28, 16 Disk
  • 17. 10 28 Memory 7 37 Disk 7 16 37 41 WAL: 10, 28, 41, 7, 37, 16
  • 18. Memory 10 28 7 16 37 41 WAL: 10, 28, 41, 7, 37, 16 Disk
  • 19. 2 47 Memory 10 28 Disk 7 16 37 41 WAL: 47, 2, 10, 28, 41, 7, 37, 16
  • 20. 2 28 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  • 21. 6 49 2 28 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  • 22. 6 49 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  • 23. 23 32 6 49 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  • 24. 6 32 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
  • 25. 30 45 Memory 6 32 Disk 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  • 26. 14 38 Memory 30 45 Disk 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 38, 14, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  • 27. 6 10 Memory 2 30 Disk 2 14 30 41 2 2 6 7 14 23 30 37 41 47 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  • 28. Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  • 29. GET(16) Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  • 30. GET(16) Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
  • 31. BitCask: архитектора AOF
  • 32. BitCask: архитектура keydir
  • 33. Sophia: архитектура Memory 15 26 40 84 Key-value index 10, 25 26, 31 39, 85 86, 96 Page index split Disk 10, 15, 16, 25 26, 31 39, 40, 84, 85 86, 96
  • 34. 39, 16, 85, 96 Insert Memory Key-value index Disk WAL Page index
  • 35. Insert Memory 16 39 85 96 Key-value index Disk Page index
  • 36. 31, 25, 10, 86 Insert 16, 96 Memory Key-value index Disk Page index WAL 16, 39, 85, 96
  • 37. Memory 10 25 31 86 Key-value index 16, 96 Page index merge Disk 16, 39, 85, 96
  • 38. Memory 10 25 31 86 Key-value index 10, 31 39, 96 Page index split Disk 10, 16, 25, 31 39, 85, 86, 96
  • 39. 15, 26, 40, 84 Insert 10, 31 Memory Key-value index Disk WAL 39, 96 Page index 10, 16, 25, 31 39, 85, 86, 96
  • 40. Memory 15 26 40 84 Key-value index 10, 31 39, 96 Page index merge Disk 10, 16, 25, 31 39, 85, 86, 96
  • 41. Memory 15 26 40 84 Key-value index 10, 25 26, 31 39, 85 86, 96 Page index split Disk 10, 15, 16, 25 26, 31 39, 40, 84, 85 86, 96
  • 42. Эпилог: choose your db wisely ?
  • 43. Links ● ● ● ● ● ● ● ● ● ● Bitcask A Log-Structured Hash Table for Fast Key/Value Data, Justin Sheehy David Smith with inspiration from Eric Brewer The Log-Structured Merge-Tree (LSM-Tree) Patrick O'Neil , Edward Cheng, Dieter Gawlick, Elizabeth O'Neil Cache-Oblivious Algorithms by Harald Prokop (Master theses) Space/time trade-offs in hash coding with allowable errors, Burton H. Bloom Data Structures and Algorithms for Big Databases, Michael A. Bender Stony Brook & Tokutek Bradley C. Kuszmaul (XLDB tutorial) http://github.com/pmwkaa/sophia, http://sphia.org http://codecapsule.com/2012/12/30/implementing-a-key-value-store-part-3-comparative-analysis-of-the-ar http://stackoverflow.com/questions/6079890/cache-oblivious-lookahead-array http://www.youtube.com/watch?v=88NaRUdoWZM(Tim Callaghan: Fractal Tree indexes) http://code.google.com/p/leveldb/downloads/list