Falcon Storage Engine Designed For Speed Presentation

1,018 views
926 views

Published on

Published in: Technology, Sports
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,018
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Falcon Storage Engine Designed For Speed Presentation

  1. 1. Falcon - built for speed Ann Harrison Kevin Lewis MySQL Users' Conference April 2009
  2. 2. If it's so fast, why isn't it done yet?
  3. 3. Talk overview Falcon at a glance Project history Multi-threading for the database developer Cycle locking
  4. 4. Falcon at a glance – read first record MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
  5. 5. Falcon at a glance – read complete MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
  6. 6. Falcon at a glance – read again MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
  7. 7. Falcon at a glance – write new record MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
  8. 8. Falcon at a glance – commit MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
  9. 9. Falcon at a glance – write complete MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
  10. 10. Falcon history Origin Transactional SQL Engine for Web App Environment Bought by MySQL in 2006 MVCC Consistent Read Verisons control write access Memory only – no steal Indexes and data separate Data encoded on disk and in memory Fine grained multi-threading
  11. 11. Falcon Goals circa 2006 Exploit large memory for more than just a bigger cache Use threads and processors for data migration Eliminate tradeoffs, minimize tuning Scale gracefully to very heavy loads Support web applications
  12. 12. Web application characteristics Large archive of data Smaller active set High read:write ratio Uneven, bursty activity
  13. 13. What we did instead Enforce limit on record cache size Respond to simple atypical loads Autocommit single record access Repeat “insert ... select” Single pass read of large data set Challenge InnoDB on DBT2 Large working set Continuous heavy load Hired the world's most vicious test designer
  14. 14. Record Cache Record Cache contains: Committed records with no versions
  15. 15. Record Cache Record Cache contains: Committed records with no versions New, uncommitted records
  16. 16. Record Cache Record Cache contains: Committed records with no versions New, uncommitted records Records with multiple versions
  17. 17. Record Cache cleanup – step 1 Cleanup old committed single version records Scavenger Runs on schedule or demand Removes oldest mature records Settable limits – start and stop
  18. 18. Record Cache Cleanup – step 2 Clean out record versions too old to be useful Prune Remove old, unneeded versions
  19. 19. Record Cache Cleanup – step 3 Clean up a cache full of new records Chill Copy new record data to log Done by transaction thread Settable start size
  20. 20. Record Cache Cleanup – step 4 Clean up multiple versions of a single record created by a single transaction Remove intermediate versions Created by a single transaction Rolled back to save point Repeated updates
  21. 21. Record Cache Cleanup – step 5 Clean up records with multiple versions, still potentially visible Backlog Copy entire record tree to disk Expensive Not yet working
  22. 22. Simple, atypical loads Challenge: Autocommit single record access Record cache is useless Record encoding is useless Transaction creation / destruction is too expensive Response: Reuse read only transactions Result: Multi-threaded bookkeeping nightmare
  23. 23. Simple, atypical loads Challenge: Repeat “insert ... select...” Fill cache with old and new records
  24. 24. Simple, atypical loads Challenge: Repeat “insert ... select...” Fill cache with old and new records First solution Scavenge old records Chill new record data
  25. 25. Simple, atypical loads Challenge: Repeat “insert ... select...” Fill cache with old and new records First solution Scavenge old records Chill new records Second solution Move the records headers out Also helps index creation
  26. 26. Simple, atypical loads Single pass read of large data set Read more records than Read them over and over Caches are useless Encoding is overhead Response: Make encoding optional?
  27. 27. Challenge InnoDB on DBT2 Initial results were not encouraging (2007) 30000 25000 20000 Transactions Falcon2007 15000 InnoDB2007 10000 5000 0 10 20 50 100 150 200 Connections
  28. 28. Challenge InnoDB on DBT2 But Falcon has improved a lot since April 2007 30000 25000 20000 Transactions Falcon2007 15000 InnoDB2007 Falcon2009 10000 5000 0 10 20 50 100 150 200 Connections
  29. 29. Challenge InnoDB on DBT2 So did InnoDB 30000 25000 20000 Transactions Falcon2007 InnoDB2007 15000 Falcon2009 InnoDB2009 10000 5000 0 10 20 50 100 150 200 Connections
  30. 30. Bug trends
  31. 31. Multi-threading Databases are a natural fit for multi-threading Connections Gophers Scavenger Disk reader/writer Except for shared structures Locking blocks parallel operations Challenge – sharing without locking
  32. 32. Multi-threading Non-locking operation Purge old record versions
  33. 33. Multi-threading Non-locking operation Purge old record versions
  34. 34. Multi-threading Locking operation Remove intermediate versions
  35. 35. Multi-threading Locking operation Remove intermediate versions What granularity of lock?
  36. 36. Multi-threading – Lock granularity One per record: Too many interlocked instructions One per record group: Thread reading one record prevents scavenge of another No answer is right – more options?
  37. 37. Cycle locking – read record chain Before starting to read a record chain, get a shared lock on a “cycle” Cycle 1 = 3 Cycle 2 shared inactive Transaction A Transaction B Transaction C
  38. 38. Cycle locking – clean a record chain Before starting to read a record chain, get a shared lock on a “cycle” Cycle 1 = 4 Cycle 2 shared inactive Transaction A active in Cycle 1 Transaction B active in Cycle 1 Transaction C active in Cycle 1 Scavenger unlinks versions from record chain and links them to a “to be deleted” list.
  39. 39. Cycle locking – records relinked Cycle 1 = 1 Cycle 2 shared inactive Transaction A releases lock Transaction B releases lock Transaction C still active Scavenger releases lock
  40. 40. Cycle locking – swap cycles New access locks cycle 2 Cycle 1 = 1 Cycle 2 = 1 shared shared Transaction C holds Cycle 1 lock Cycle Manager requests exclusive on Cycle 1 (pumps cycle) Transaction A acquires Cycle 2 lock
  41. 41. Cycle locking – cleanup phase Cycle 1 = 0 Cycle 2 = 2 shared shared exclusive Transaction C releases lock Transaction B acquires Cycle 2 lock Cycle manager exclusive Cycle 1
  42. 42. Cycle locking – cleanup complete Cycle 1 Cycle 2 = 2 exclusive shared Transaction C acquires Cycle 2 lock Cycle manager exclusive Cycle 1 Remove unlinked, unloved, old versions When cleanup is done, Cycle manager releases cycle 1
  43. 43. Questions

×