Xen has made strides in scaling network performance such that it can now saturate 10 Gbit/s NICs and more taking in account aggregate throughput in multi-queued host backends. Grant usage, one of the main bottlenecks associated with paravirtualizated I/O, has seen a number of improvements in the past years. However, 100 Gbit/s network adapters beckon and 25 Gbit/s will replace current 10 Gbit/s adapters in short order. This will require us to revisit the datapath. This presentation discusses ways of overcoming this bottleneck and we describe the work we have been doing in the area without compromising the current grant-based model. We discuss a number of topics in this talk, such as page recycling mechanisms, grants versus copying and trends in networking/drivers. Finally we pitch an optional grant-less model in a way that can be managed by adminstrators and is transparent to guests.

1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
To Grant or Not to Grant
(for the case of Xen network drivers)
João Martins
Principal Software Engineer
Virtualization Team
July 11, 2017

2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2

3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Motivation
3

4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Increasing networking speeds
• 10 Gbit/s is the new 1Gbit/s
– 25/50 Gbit/s NICs increasingly more common on servers1
– The rise of 100 Gbit/s and even crazier packet budgets
• NVDIMM shortening storage latencies
– Exposes virt. backend network processing overhead
• Xen scaling nicely thanks to scalability work on grant tables
– But performance per queue hasn't changed much
[1] http://www.mellanox.com/blog/2016/03/25-is-the-new-10-50-is-the-new-40-100-is-the-new-amazing/
4

5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Xen dealing with increasing networking speeds
• Performance per queue ~15 Gbit/s
• Aggregate throughput improvements in past releases
– Requires a backend with a reasonable amount of VCPUs
• Only bulk data transfer seem to perform well (32K..64K sizes)
• Grants handling represent 50% of Xen backend processing
• Challenge: Reach 100 Gbit/s and with the least amount of vCPUS
5

6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Background
6

7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
netif protocol in a nutshell
●
Two rings (fixed slot size)
●
A ring for control purposes
– Currently for RSS hash keys
●
Separate TX and RX interrupt
●
Guest produces requests, backend
responses once network stack has
finished delivery
●
Data represented as grants
7

8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Grant Tables
8
• Table with round robin recycled 'tickets' visible by Xen
– Owned by granter, table grown on guest request
• Indexes within this table are known as grant refs
• Contains gfn (frame), permissions(flags), domain(dom)
• 512 entries per 4K page
• Two grant table versions and protocols
• Granter populates a ref and fills it in shared ring
• Grantee changes flags to notify frontend of ref usage

9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Grant Tables Handling
9
• Ballooned/Fictitious pages used for mapping foreign memory
• Backends map/unmap/copy grefs
– Priviliged backends can access the whole guest addr space
• Only Xen sees granter/grantee tables
• Xen maintains private state for in-use entries
• Maptrack keeps track of those mapped by grantee (backend)
– Grows with usage, by stealing from other vCPUs
– Grantee is given an handle which are indexes in this table

10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Grants and the I/O path
10
Maptrack
vCPU 0
Maptrack
vCPU 1
Maptrack
vCPU n
Frame 0
0 .. 511
Frame 1
512..1023
Frame N
X..X+512. . .
grant table
Frontend (Linux)Backend (Linux)
Frame 1
512..102
3
Frame N
X..X+512
active grants (private)
. . .Frame 1
512..1023
Frame 0
0 .. 511. . .
Xen
NAPI
(softirq)
Dealloc
(kthread)
guestrx
(kthread)
ref: 500
map:ref=500
handle=10
unmap:handle=10
vifX.Y
xmit
copy: ref=2015 len=84
Pick
Grant
Ref
gfn
writeframe=<gfn>
ref: 500
TX
RX
status: 84
gfn

11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Proposal
11

12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants - Idea
• Backend maps/unmaps a permanent set of grefs
• Initially planned for the linear packet region only
• Frontend copies or ...
– Alternatively, recycles the permanent grefs (Rx-only on Linux)
• Similar to persistent grants except:
– No periodic purging of grants i.e. backend doesn't try to map all incoming grefs
– Guest has more control on managing grants
– The protocol is weighted in favour of guest recycling buffers
12

13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants - Implementation
• Side-channel for co-managing long-lived grants
• Introduce 3 new control ring messages:
– CTRL_TYPE_{ADD,PUT}_GREF_MAPPING
– CTRL_TYPE_GET_GREF_MAPPING_SIZE
• Frontend requests <ring-size> (256) grefs mapping on state Connected
• Alternatively uses a TX|RX slot flag for shortcutting the map
• Backend keeps a hash table of ref ↔ page
13

14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Grants and the I/O path (staging grants)
14
Frame 0
0 .. 511
Frame 1
512..1023
Frame N
X..X+512. . .
grant table
Backend
NAPI
(softirq)
ref: 500
(fetch from
pregranted tbl)
vifX.Y
memcpy: ref=700 len=84
memcpy: ref: 500
TX
RX
status: 84
gfn
Premapped
Frames Table
ref: 500 ro
ref: 700 rw
...

15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants - Page Recycling
●
Having a permanent set of grefs requires either strategies:
1) Use a fixed set of buffers and copy from/to [implemented]
2) Recycle grefs to avoid copy [implemented]
●
Recycling implemented on Intel drivers and Mellanox drivers
●
Rx only; Tx requires copy to the mapped grefs
●
Page reference count tells driver when page is reusable
– Userspace zerocopy RX proposed recently1
[1] “[RFC PATCH 2/2] ixgbe: add af_packet direct copy support”, https://www.spinics.net/lists/netdev/msg417329.html
15

16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants - Page Recycling
●
The trick is to chase our own tail i.e.
– The bigger the ring the longer it takes to consume the oldest entry
– Case for multi-page rings?
●
I/O path to perform as if no grants had to be used!
●
No copy to permanent grant → CPU usage decreased
16

17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants and Fast Packet I/O engines
●
Fast Packet I/O engines (netmap, DPDK, XDP) are known for:
– Preallocation of buffers / Static DMA mappings
– Packet batching
– Minimum amount of metadata per packet (compared to skb or mbufs in Linux/FreeBSD)
●
Express Data Plane (XDP) introduced in Linux 4.9 to compete with DPDK
●
Aiming at improving netif protocol for these cases
17

18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Experimental Setup
18
●
Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (2x10 cores)
●
Assign each RX|TX event channel to a separate CPU
●
Pin Guest/Dom0 CPUs to separate cores, Dom0 has 5 CPUs.
●
Linux v4.10 + Xen 4.10-unstable (commit 3d2010f9ff)
ConnectX-4 100Gbit/s (mlx5)
Xen Baremetal

19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants – Throughput
19

20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants – Rx Page Recycling Efficiency
20

21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants – Packet I/O
21
*DomU → Dom0 (Local, no bridge/NIC involved)

22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Staging grants – Remarks
●
Database being the perfect storm in perf. requirements
●
2.4x better handling OLTP workloads with short payloads
●
CPU utilization on backend decreased from 38 % → 22%
●
XDP (with staging grants) does with 1 core what the stack does with 4!
22

23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
“What IF we didn't have grants at all?”
23

24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Project Zerogrant – Is it that crazy?
●
All powerful Domain-0 can access pretty much the whole gfn space
– For typical Domain-0 setup there's not really a violation of policy
– With Paging and Ballooning it is hard to cope with P2M changes
●
Grant tables are private to DOMID_SELF or Xen
– Need to extend ABI to allow map/unmap foreign grant tables
●
Keeping guest mapped adds up a significant memory requirements to
Domain-0
– struct page overhead (64 bytes i.e. 4GB for a 256 GB guest) plus any added per-page metadata
24

25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Project Zerogrant – Idea for Discussion
●
Map the whole guest on Domain-0
– Restricted to PV backends
●
Map the guests grant table
– Allow a gref_to_pfn(...) semantics on Domain-0
●
Only for the privileged domain
●
Frontend allowing blacklisting certain pfns to be mapped
25
Come discuss on design session “Zerogrant on Xen”, Wed 4:35PM!

26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Zerogrant PoC – Throughput
26
* Unmodified v4.10 guest like Baseline

27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Summary of presented schemes
• Normal grants for heavy
consolidated systems
• Map + unmap as needed
• Add more Domain-0 CPUs
for higher throughput
• Amount of grants mapped
proportional to workload
• Staging grants for
performance
• Backend keeps fixed amount
of grants mapped
• Copy and/or guests
recycling Rx buffers (Guest
choice)
• Balance between I/O
performance and medium
consolidated environments
• No grants for highest
performance
• Exclusively for Domain-0
centric setups
• Grants become “just” a
lookup for guest frame
numbers
• Totally transparent to guests
• Higher memory
requirements per VM
27

28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Conclusions & Future Work
●
V2 design doc/code (XDP included!) targeting Xen 4.10
●
Project zerogrant for extreme performance on PV drivers
●
Other issues identified so far:
– Lack of interrupt coalescing or better batching
– Memory locality per queue (NUMA)
– Infra for properly negotiating features on PV backends
28

29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 29