Your SlideShare is downloading. ×
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Zarafa Scaling & Performance
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Zarafa Scaling & Performance

4,682

Published on

Presentation of Steve Hardy about Scaling and performance tuning at Zarafa SummerCamp 2011

Presentation of Steve Hardy about Scaling and performance tuning at Zarafa SummerCamp 2011

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
4,682
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Scaling & Performance: Seven new insights
    Steve Hardy, Zarafa
  • 2. Basics presented at last summercamp
    Split-table ‘properties’
    One for row-order data
    One for column-order data
    Zarafa 7.0: I/O improvements
  • 3. Row order
    Row-order and column-order
  • 4. Column order
    Row-order and column-order
  • 5. Why is hybrid better ?
    Best of both worlds
    Drawback: needs more storage
    But this storage can be cheaper
    Not double
    Lots of data not needed in column order (eg attachments)
    Much faster load of e-mails (less IOPS)
  • 6. Read flags
    Read flags no longer in index
    Not actually used, even in Zarafa 6
    Modifying read flag meant key change, causing I/O
  • 7. Sorting
    With many folders, Zarafa 6 becomes less efficient
    Assumes N folders with equally distributed messages
  • 8. Counters
    Many counters are used:
    Total count
    Unread count
    Deleted count
    Folder count
    Deleted folder count
    Associated count
    Deleted associated count
    In Zarafa 7, counters are tracked incrementally
    Assumes folder with 10k messages
  • 9. More efficient when writes are grouped together
    Writes are ‘deferred’ until later
    Separate process in Zarafa purges deferred write
    Saves about up to 50% in I/O while writing to tproperties
    Deferred writes to tproperties (column order)
    Write data
    Defer
    Write tproperties
  • 10. Control number of deferred objects
    Counter reset (set to no for higher performance)
    Counter resets shown in stats (zarafa-stats –system)
    Deferred purges shows in stats
    New options in Zarafa 7.0
  • 11. Biggest advantages in slower I/O subsystems
    Should show most advantages on low-RPM disks (eg SATA)
    Allows to have about 15% of your data on SSD, which uses 50% of IOPS
    Example: 100GB of SSD is enough to reduce IOPS by half for total data size of 600GB
    Depending on your vender 100GB of SSD is between EUR100 and EUR1600
    Example 2: Instead of buying 6 15k SAS disks @ 200 IOPS each (total 1200 IOPS), also possible to buy 1 SSD disk and 5 SATA disks @ 120 IOPS, which even has a higher capacity.
    Especially interesting for large-scale (cloud) rollouts
    I/O changes: things to remember
  • 12. Records are much more compact in memory inside MySQL in Zarafa 7.0 than in Zarafa 6
    Possibly more cache hit ratio achievable by using more cache memory in MySQL, less in Zarafa
    No large-scale testing done yet, this will follow
    Even when cache is more efficiently packed in MySQL RAM, latency between Zarafa and MySQL server is still an issue, so you always need some cache in Zarafa
    Cache tuning
  • 13. MySQL 5.5 introduced linux native AIO
    AIO = asynchronous I/O, meaning that multiple requests can be sent to RAID at once
    Before, there was ‘emulated AIO’ which could do at most X requests simultaneously (X = 4 for most installations)
    Some requirements:
    O_DIRECT enabled
    ext3, ext4, xfs or direct block device
    AIO-capable kernel (was introduced in 2.6.24 or so)
  • 14. Multiple requests are only sent for parallel queries
    If you run one query, you only can have one outstanding I/O request
    Makes some things slow:
    Mysqldump
    Alter table
    Queries doing non-trivial range queries (eg select * from a where id > 10 and id < 10000)
    Why AIO is not as great as it sounds
  • 15. http://bugs.mysql.com/bug.php?id=60087
    Enables prefetch mechanism for sequential reads in innodb
    Main advantage for Zarafa:
    Table queries for large tables (sorting)
    Faster mysqldumps
    Our patch for MySQL
  • 16. Anyone who signs up to mysql’sbugtracker and comments on the bug, asking for it to be reviewed will receive a free beer from me this evening (no kidding)
    Free Beer (Free as in beer)
    http://bugs.mysql.com/bug.php?id=60087
    http://bit.ly/kVlQVv
  • 17. MSR: Mail Store Relocator
    Used to migrate user stores between cluster nodes
    Server1
    Server2
  • 18. Little known feature of MSR
    Normally used for user migration between nodes
    Now also allows bi-directional synchronisation
    Possible within cluster AND outside cluster
    ACLs only synchronized when inside cluster
    Applications:
    Distributed public store
    Requires EVERY server in cluster to contain public replica
    Shared store between geographic locations (eg. Shared ‘info’ inbox)
    Shared store between organisations
    Will improve:
    Currently does realtime message sync, but no realtime folder sync
    Demo in session Friday 10:00
    Scaling: store replication (kind of beta)
  • 19. Epoll() instead of select() interface for main socket dispatcher (improves cpu overhead/delay)
    Compressed records
    E-mail currently stores bodies uncompressed in MySQL
    Compressing plaintext & HTML bodies can save about 20% storage
    Schema already compression-ready (no schema change needed in the future)
    Future additions

×