Zarafa Scaling & Performance


Published on

Presentation of Steve Hardy about Scaling and performance tuning at Zarafa SummerCamp 2011

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Zarafa Scaling & Performance

  1. 1. Scaling & Performance: Seven new insights<br />Steve Hardy, Zarafa<br />
  2. 2. Basics presented at last summercamp<br />Split-table ‘properties’<br />One for row-order data<br />One for column-order data<br />Zarafa 7.0: I/O improvements<br />
  3. 3. Row order<br />Row-order and column-order<br />
  4. 4. Column order<br />Row-order and column-order<br />
  5. 5. Why is hybrid better ?<br />Best of both worlds<br />Drawback: needs more storage<br />But this storage can be cheaper<br />Not double<br />Lots of data not needed in column order (eg attachments)<br />Much faster load of e-mails (less IOPS)<br />
  6. 6. Read flags<br />Read flags no longer in index<br />Not actually used, even in Zarafa 6<br />Modifying read flag meant key change, causing I/O<br />
  7. 7. Sorting<br />With many folders, Zarafa 6 becomes less efficient<br />Assumes N folders with equally distributed messages<br />
  8. 8. Counters<br />Many counters are used:<br />Total count<br />Unread count<br />Deleted count<br />Folder count<br />Deleted folder count<br />Associated count<br />Deleted associated count<br />In Zarafa 7, counters are tracked incrementally<br />Assumes folder with 10k messages<br />
  9. 9. More efficient when writes are grouped together<br />Writes are ‘deferred’ until later<br />Separate process in Zarafa purges deferred write<br />Saves about up to 50% in I/O while writing to tproperties<br />Deferred writes to tproperties (column order)<br />Write data<br />Defer<br />Write tproperties<br />
  10. 10. Control number of deferred objects<br />Counter reset (set to no for higher performance)<br />Counter resets shown in stats (zarafa-stats –system)<br />Deferred purges shows in stats<br />New options in Zarafa 7.0<br />
  11. 11. Biggest advantages in slower I/O subsystems<br />Should show most advantages on low-RPM disks (eg SATA)<br />Allows to have about 15% of your data on SSD, which uses 50% of IOPS<br />Example: 100GB of SSD is enough to reduce IOPS by half for total data size of 600GB<br />Depending on your vender 100GB of SSD is between EUR100 and EUR1600<br />Example 2: Instead of buying 6 15k SAS disks @ 200 IOPS each (total 1200 IOPS), also possible to buy 1 SSD disk and 5 SATA disks @ 120 IOPS, which even has a higher capacity.<br />Especially interesting for large-scale (cloud) rollouts<br />I/O changes: things to remember<br />
  12. 12. Records are much more compact in memory inside MySQL in Zarafa 7.0 than in Zarafa 6<br />Possibly more cache hit ratio achievable by using more cache memory in MySQL, less in Zarafa<br />No large-scale testing done yet, this will follow<br />Even when cache is more efficiently packed in MySQL RAM, latency between Zarafa and MySQL server is still an issue, so you always need some cache in Zarafa<br />Cache tuning<br />
  13. 13. MySQL 5.5 introduced linux native AIO<br />AIO = asynchronous I/O, meaning that multiple requests can be sent to RAID at once<br />Before, there was ‘emulated AIO’ which could do at most X requests simultaneously (X = 4 for most installations)<br />Some requirements:<br />O_DIRECT enabled<br />ext3, ext4, xfs or direct block device<br />AIO-capable kernel (was introduced in 2.6.24 or so)<br />
  14. 14. Multiple requests are only sent for parallel queries<br />If you run one query, you only can have one outstanding I/O request<br />Makes some things slow:<br />Mysqldump<br />Alter table<br />Queries doing non-trivial range queries (eg select * from a where id > 10 and id < 10000)<br />Why AIO is not as great as it sounds<br />
  15. 15.<br />Enables prefetch mechanism for sequential reads in innodb<br />Main advantage for Zarafa:<br />Table queries for large tables (sorting)<br />Faster mysqldumps<br />Our patch for MySQL<br />
  16. 16. Anyone who signs up to mysql’sbugtracker and comments on the bug, asking for it to be reviewed will receive a free beer from me this evening (no kidding)<br />Free Beer (Free as in beer)<br /><br /><br />
  17. 17. MSR: Mail Store Relocator<br />Used to migrate user stores between cluster nodes<br />Server1<br />Server2<br />
  18. 18. Little known feature of MSR<br />Normally used for user migration between nodes<br />Now also allows bi-directional synchronisation<br />Possible within cluster AND outside cluster<br />ACLs only synchronized when inside cluster<br />Applications:<br />Distributed public store<br />Requires EVERY server in cluster to contain public replica<br />Shared store between geographic locations (eg. Shared ‘info’ inbox)<br />Shared store between organisations<br />Will improve:<br />Currently does realtime message sync, but no realtime folder sync<br />Demo in session Friday 10:00<br />Scaling: store replication (kind of beta)<br />
  19. 19. Epoll() instead of select() interface for main socket dispatcher (improves cpu overhead/delay)<br />Compressed records<br />E-mail currently stores bodies uncompressed in MySQL<br />Compressing plaintext & HTML bodies can save about 20% storage<br />Schema already compression-ready (no schema change needed in the future)<br />Future additions<br />