FiresideGrade of the Steel    Thomas Kejser
Grade of the Steel1. Pick a simple workload2. Do the naïve thing3. Tune the … out of it4. Find the limit5. Get smart (opti...
Grade of the Steel Examples• 16M BULK Rows/sec• 50M rows SELECT (4 x 10Gbit)• Backup at 6GB/sec• 1.1M INSERT/sec
Step 1 – Theory and BaselineCommit                            SQLIO.EXE            BufferOffset           –t1             ...
Step 2 – Design Workload         File 1   FG 1   Table 1         File 2   FG 2   Table 2                                  ...
Step 3 – Tune• Add Files / Tables                          Theory        Measure• Thread Sweet spot• Affinitize I/O• Place...
Messages Queues Part 1 – Naïve
Problem Statement       Push                                 Pop                    Queue Structure300B                   ...
The Naïve ApproachPush                      Pop• Seek first value page   • Seek last value page• INSERT a row at page    •...
Why this doesn’t Scale!                             Root               Min     Min         Min         Min    Min   Min   ...
Messages QueuesPart 2 – Getting smart
Summarising the Problem• Hot stuff   –   Root   –   Min page   –   Max page   –   Intermediate pages• Alloc/Dealloc• BUT: ...
Cooling it down
Allocations - What if?...Push                         Pop• Seek first value page      • Seek last value page• UPDATE Refer...
Dissipate the Heat      Last Digit = 0         Last Digit = 1               Last Digit = 2Min                Max    Min   ...
Eliminating Thread Contention                           VERY                           fast!  PopSequence++               ...
Ring BufferingPushSequence+=2   PopSequence++                                   PushSequence+=1    Mod 100          Mod 10...
Summing it up• UPDATE instead of  INSERT/DELETE• Multi partitions =  multiple trees• Ring buffer using modulo• Sweet spot ...
Upcoming SlideShare
Loading in...5
×

Fireside fusion talk - message queues

590

Published on

A presentation about creating message queues in relational database. Presented at SQL BITS X

Published in: Technology, Business
1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total Views
590
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Fireside fusion talk - message queues

  1. 1. FiresideGrade of the Steel Thomas Kejser
  2. 2. Grade of the Steel1. Pick a simple workload2. Do the naïve thing3. Tune the … out of it4. Find the limit5. Get smart (optional)
  3. 3. Grade of the Steel Examples• 16M BULK Rows/sec• 50M rows SELECT (4 x 10Gbit)• Backup at 6GB/sec• 1.1M INSERT/sec
  4. 4. Step 1 – Theory and BaselineCommit SQLIO.EXE BufferOffset –t1 –o1 –b60 ... Slot 1 Slot –kW ... n -fsequential Writer COREINFO.EXE > info.txt Queue Log Writer
  5. 5. Step 2 – Design Workload File 1 FG 1 Table 1 File 2 FG 2 Table 2 SQLCMDStripe File 3 FG 3 Threads Table 3 File X FG X Table X
  6. 6. Step 3 – Tune• Add Files / Tables Theory Measure• Thread Sweet spot• Affinitize I/O• Placement of Log Change
  7. 7. Messages Queues Part 1 – Naïve
  8. 8. Problem Statement Push Pop Queue Structure300B Msg Msg Msg Msg Msgmsg Ordered
  9. 9. The Naïve ApproachPush Pop• Seek first value page • Seek last value page• INSERT a row at page • DELETE/Output key on page Min Max Min Max Msg Msg Msg
  10. 10. Why this doesn’t Scale! Root Min Min Min Min Min Min Min Min MinMin Max
  11. 11. Messages QueuesPart 2 – Getting smart
  12. 12. Summarising the Problem• Hot stuff – Root – Min page – Max page – Intermediate pages• Alloc/Dealloc• BUT: We Must have order!
  13. 13. Cooling it down
  14. 14. Allocations - What if?...Push Pop• Seek first value page • Seek last value page• UPDATE Reference Count • UPDATE Reference Count Min Max Min Max Msg++ Msg--
  15. 15. Dissipate the Heat Last Digit = 0 Last Digit = 1 Last Digit = 2Min Max Min Max Min MaxMsg-- Msg++ Msg-- Msg++ Msg-- Msg++
  16. 16. Eliminating Thread Contention VERY fast! PopSequence++ PushSequence++ Queue Structure 4 5 6 7 8 Ordered
  17. 17. Ring BufferingPushSequence+=2 PopSequence++ PushSequence+=1 Mod 100 Mod 100 Mod 100 MyQ Slot:0 Slot:42 Slot: 97 Slot: 98 Slot: 99Msg: 200 Msg: 142 Msg: 197 Msg: 198 Msg: 199 Ordered
  18. 18. Summing it up• UPDATE instead of INSERT/DELETE• Multi partitions = multiple trees• Ring buffer using modulo• Sweet spot concurrency

×