XPDS13: Xen and XenServer Storage Performance - Felipe Franciosi, Citrix

2,509 views

Published on

The development of low latency storage media such as modern Solid State Drives (SSD) brings new challenges to virtualisation platforms. For the first time, we are witnessing storage back ends which are so fast that the CPU time spent in processing data significantly impacts the delivered throughput. This is aggravated by CPU speeds remaining largely constant while storage solutions get faster by orders of magnitude. To meet user demands and fully exploit SSD performance under Xen, new technologies are necessary. This talk will discuss the Xen storage virtualisation data path when using various back ends (e.g. blkback, tapdisk, qemu). It will explain why it is hard to exploit SSD performance with current technologies and present measurement data for a variety of workloads. Finally, it will show how techniques such as persistent grants and indirect I/O can help to mitigate the problem.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,509
On SlideShare
0
From Embeds
0
Number of Embeds
59
Actions
Shares
0
Downloads
68
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

XPDS13: Xen and XenServer Storage Performance - Felipe Franciosi, Citrix

  1. 1. Xen and XenServer Storage Performance Low Latency Virtualisation Challenges Dr Felipe Franciosi XenServer Engineering Performance Team e-mail: felipe.franciosi@citrix.com freenode: felipef #xen-api twitter: @franciozzy
  2. 2. Agenda • Where do Xen and XenServer stand? ๏ When is the virtualisation overhead most noticeable? • Current implementation: ๏ blkfront, blkback, blktap2+tapdisk, blktap3, qemu-qdisk • Measurements over different back ends Throughput and latency analysis ๏ Latency breakdown: where are we losing time when virtualising? ๏ • Proposals for improving 2 © 2013 Citrix
  3. 3. Where do Xen and XenServer Stand? When is the virtualisation overhead noticeable?
  4. 4. Where do Xen and XenServer stand? • What kind of throughput can I get from dom0 to my device? ๏ 4 Using 1 MiB reads, this host reports 118 MB/s from dom0 © 2013 Citrix
  5. 5. Where do Xen and XenServer stand? • What kind of throughput can I get from domU to my device? ๏ Using 1 MiB reads, this host reports 117 MB/s from a VM IMPERCEPTIBLE virtualisation overhead 5 © 2013 Citrix
  6. 6. Where do Xen and XenServer stand? • That’s not always the case... ๏ Same test on different hardware (from dom0) my disks can do 700 MB/s !!!! 6 © 2013 Citrix | Confidential - Do Not Distribute
  7. 7. Where do Xen and XenServer stand? • That’s not always the case... ๏ Same test on different hardware (from domU) VISIBLE virtualisation overhead why is my VM only doing 300 MB/s ??? 7 © 2013 Citrix | Confidential - Do Not Distribute
  8. 8. Current Implementation How we virtualise storage with Xen
  9. 9. Current Implementation (bare metal) • How does that compare to storage performance again? ๏ There are different ways a user application can do storage I/O • We will use simple read() and write() libc wrappers as examples BD block layer device driver 1. char buf[4096]; 2. int fd = open(“/dev/sda”, O_RDONLY | O_DIRECT); HW Interrupt on completion sys_read(fd, buf, 4096) vfs_read() f_op->read()** 3. read(fd, buf, 4096); kernel space user space libc user process 9 © 2013 Citrix buf fd 4. buf now has the data!
  10. 10. Current Implementation (Xen) • The “Upstream Xen” use case The virtual device in the guest is implemented by blkfront ๏ Blkfront connects to blkback, which handles the I/O in dom0 ๏ dom0 BD device driver block layer blkback xen’s blkif protocol domU VDI device blkfront driver block layer syscall / etc() kernel space user space kernel space user space libc user process 10 © 2013 Citrix buf fd
  11. 11. Current Implementation (Xen) • The XenServer 6.2.0 use case XenServer provides thin provisioning, snapshot, clones, etc. hello VHD ๏ This is easily implemented in user space. hello TAPDISK ๏ dom0 BD device driver block layer tap data stored in VHD aio syscalls kernel space user space libaio tapdisk2 11 © 2013 Citrix blktap2 block layer blkback xen’s blkif protocol domU VDI blkfront block layer syscall / etc() kernel space user space libc user process buf fd
  12. 12. Current Implementation (Xen) • The blktap3 and qemu-qdisk use case ๏ Have the entire back end in user space dom0 BD block layer device driver domU VDI blkfront block layer data stored in VHD aio syscalls kernel space user space evtchn dev libaio tapdisk3 / qemu-qdisk tapdisk2 12 gntdev © 2013 Citrix xen’s blkif protocol syscall / etc() kernel space user space libc user process buf fd
  13. 13. Measurements Over Different Back Ends Throughput and latency analysis
  14. 14. Measurement Over Different Back Ends • Same host, different RAID0 logical volumes on a PERC H700 ๏ 14 All have 64 KiB stripes, adaptive read-ahead and write-back cache enabled © 2013 Citrix
  15. 15. • dom0 had: • 4 vCPUs pinned • 4 GB of RAM • It becomes visible that certain back ends cope much better with larger block sizes. • This controller supports up to 128 KiB per request. • Above that, the Linux block layer splits the requests. 15 © 2013 Citrix
  16. 16. • Seagate ST (SAS) • blkback is slower, but it catches up with big enough requests. 16 © 2013 Citrix
  17. 17. • Seagate ST (SAS) • User space back ends are so slow they never catch up, even with bigger requests. • This is not always true: if the disks were slower, they would catch up. 17 © 2013 Citrix
  18. 18. • Intel DC S3700 (SSD) • When the disks are really fast, none of the technologies catch up. 18 © 2013 Citrix
  19. 19. Measurement Over Different Back Ends • There is another way to look at the data: 1 Throughput (data/time) 19 © 2013 Citrix = Latency (time/data)
  20. 20. • Intel DC S3700 (SSD) • The question now is: where is time being spent? • Compare time spent: • dom0 • blkback • qdisk 20 © 2013 Citrix
  21. 21. Measurement Over Different Back Ends • Inserted trace points using RDTSC ๏ TSC is consistent across cores and domains 6 BD block layer device driver 8 kernel space user space 1. Just before issuing read() 2. On SyS_read() 3. On blkfront’s do_blkif_request() 4. Just before notify_remote_via_irq() 5. On blkback’s xen_blkif_be_int() 6. On blkback’s xen_blkif_schedule() 7. Just before blk_finish_plug() 21 © 2013 Citrix dom0 7 blkback 3 domU VDI block layer device blkfront driver 5 4 9 8. On end_block_io_op() 9. Just before notify_remove_via_irq() 10. On blkif_interrupt() 11. Just before __blk_end_request_all() 12. Just after returning from read() 11 10 syscall / etc() 2 kernel space user space libc 1 12 user process buf fd
  22. 22. Measurement Over Different Back Ends • Initial tests showed there is a “warm up” time • Simply inserting printk()s affect the hot path • Used a trace buffer instead In the kernel, trace_printk() ๏ In user space, hacked up buffer and a signal handler to dump its contents ๏ • Run 100 read requests One immediately after the other (requests were sequential) ๏ Used IO Depth = 1 (only one in flight request at a time) ๏ • Sorted the times, removed the 10 (10%) fastest and slowest runs • Repeated the experiment 10 times and averaged the results 22 © 2013 Citrix
  23. 23. Measurements on 3.11.0 without Persistent Grants Time spent on device Actual overhead of mapping and unmapping and transferring data back to user space at the end 23 © 2013 Citrix
  24. 24. Measurements on 3.11.0 with Persistent Grants Time spent on device Time spent copying data out of persistently granted memory 24 © 2013 Citrix
  25. 25. Measurement Over Different Back Ends • The penalty of copying can be worth taking depending on other factors: Number of dom0 vCPUs ๏ Number of concurrent VMs performing IO ๏ (TLB flushes) (contention on grant tables) • Ideally, blkfront should support both data paths • Administrators can profile their workloads and decide what to provide 25 © 2013 Citrix
  26. 26. Proposals for Improving What else can we do to minimise the overhead?
  27. 27. New Ideas • How can we reduce the processing required to virtualise I/O ? 27 © 2013 Citrix
  28. 28. New Ideas • Persistent Grants Issue: grant mapping (and unmapping) is expensive ๏ Concept: back end keeps grants, front end copies I/O data to those pages ๏ Status: currently implemented in 3.11.0 and already supported in qemu-qdisk ๏ • Indirect I/O Issue: blkif protocol limit requests to 11 segs of 4 KiB per I/O ring ๏ Concept: use segments as indirect mapping of other segments ๏ Status: currently implemented in 3.11.0 ๏ 28 © 2013 Citrix
  29. 29. New Ideas • Avoid TLB flushes altogether (Malcolm Crossley) ๏ Issue: • When unmapping, we need to flush the TLB on all cores • But in the storage data path, the back end doesn’t really access the data Concept: check whether granted pages have been accessed ๏ Status: early prototypes already developed ๏ • Only grant map pages on the fault handler (David Vrabel) ๏ Issue: • Mapping/unmapping is expensive • But in the storage data path, the back end doesn’t really access the data Concept: Have a fault handler for the mapping, triggered only when needed ๏ Status: idea yet being conceived ๏ 29 © 2013 Citrix
  30. 30. New Ideas • Split Rings ๏ Issue: • blkif protocol supports 32 slots per ring for requests or responses • this limits the total amount of outstanding requests (unless multi-page rings are used) • this makes inefficient use of CPU caching (both domains writing to the same page) Concept: use one ring for requests and another for responses ๏ Status: early prototypes already developed ๏ • Multi-queue Approach Issue: Linux’s block layer does not scale well across NUMA nodes ๏ Concept: allow device drivers to register a request queue per core ๏ Status: scheduled for Kernel 3.12 ๏ 30 © 2013 Citrix
  31. 31. e-mail: felipe.franciosi@citrix.com freenode: felipef #xen-api twitter: @franciozzy

×