Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Implementing SR-IOV Failover for
Windows Guests During Migration
Annie Li - Principal Software Engineer, Oracle
Yan Vugenf...
Agenda
• Virtio-win drivers
• Windows guest terminology
• The problem
• Different solutions
• Failover solution with virti...
• Upstream: https://github.com/virtio-win/kvm-guest-
drivers-windows/
• Drivers for the major virtio devices:
• virtio-net...
• WDF drivers for the “simple” devices
• Miniport architecture for network and
storage
• NDIS
• Storport
• Scsiport
VirtIO...
• Supported OS
• Windows XP, Vista, 7, 8, 8.1, 10 (up to
recent builds)
• Widows Server 2003, 2008, 2008R2,
2012, 2012R2, ...
• Send PRs - https://github.com/virtio-
win/kvm-guest-drivers-windows/
pulls
• Code changes should pass WHQL
• We are runn...
Contributors
• And others
NDIS (Network Driver Interface Specification) Architecture
NotifyObject
Network	
configuration	
subsystem
HW	device HW	dev...
• NDIS miniport driver
• Basic driver package:
• INF file – installation description
• SYS file – driver binary
• PDB file...
The Problem – Live Migration and SR-IOV
Overview of SR-IOV Migration Solutions
• Previous efforts and vendor specific HW
solutions
• Hyper-V and Windows
• Linux a...
Previous Efforts
• KVM Forum 2015 Live Migration with SR-IOV
Pass-through - Weidong Han, Huawei
• KVM Forum 2018 - Live Mi...
Overview of Software Solutions
• Windows NIC Teaming
• Windows MUX Intermediate driver
• Hyper-V Solution
Windows NIC Teaming
• Similar to bond in Linux
• Provides failover capability
• Configured through GUI or Powershell
Cmdle...
Windows MUX Intermediate Driver
• Kernel space solution with various models
• One-to-two model for SR-IOV live
migration
Windows MUX Intermediate Driver
TCPIP
VF	miniport	
driver
					
NotifyObject
Open	Adapter1 Open	Adapter2
Network	
configur...
Network Binding of NIC Teaming or MUX
Hyper-V VM Network
• Network virtual service client(NetVSC )
• Synthetic data path
• SR-IOV data path
• Two Installation f...
Hyper-V SR-IOV VF Failover
TCPIP Other	
Protocols
VF	Miniport	
Driver
					Hyper-V	NetVSC	Driver
Hyper-V	NetVSC	
Miniport	...
Network Binding in Hyper-V
Comparison Summary
• MUX Driver model:
• Complicated, New virtual adapter,
Restore offload, NotifyObject.
• Hyper-V model:...
Windows Network of VirtIO and VF
TCPIP Other	Protocols
VF	Miniport	
Driver
NetKVM	
Miniport	Driver
VirtIO	Network	
Device
...
TCPIP Other	Protocols
VF	Miniport	
Driver
					VirtIO	NetKVM	Driver
NotifyObject
Network	
configuration	
subsystem
Virtual...
Protocol Driver in 2-netdev Model
• Behaves like a bridge
• VF adapter is coupled to VirtIO adapter
with the same MAC addr...
Network Binding of VirtIO SR-IOV
Known Issues
• DHCP issue
• Only happens in specific scenario
• Statistics is missing
• NetKVM driver needs to keep the st...
VirtIO and SR-IOV Failover
• VirtIO specification specifics
• Feature bit called
VIRTIO_NET_F_STANDBY. It is
appropriate f...
Installation
• Before
• INF for Miniport
• After
• INF for Miniport
• INF for Protocol
driver definition
and Notify Object...
WHQL Certification
• Before
• Certification of
miniport
• Automatic review
• After
• Two steps
certification
• Automatic r...
Performance
From	VM	to	Remote	Host(NetPerf,	32	vcpus,	32	TCP	Streams,	MTU=9000)
0
25
50
75
100
Default	Default 16k	32k 16k...
Performance
From	Remote	Host	to	VM(NetPerf,	32	vcpus,	32	TCP	Streams,	MTU=9000)
0
25
50
75
100
Default	Default 16k	32k 16k...
Performance
From	VM	to	VM	on	Remote	Host	(NetPerf,	32	vcpus,	32	TCP	Streams,	MTU=9000)
0
25
50
75
100
Default	Default 16k	...
annie.li@oracle.com yan@daynix.com
Links – source code
• virtio-win drivers source code:
• https://github.com/virtio-win/kvm-guest-drivers-
windows
• MiniPor...
Links – download binary drivers
• https://docs.fedoraproject.org/en-US/quick-docs/
creating-windows-virtual-machines-using...
Links – related presentations
• KVM Forum 2015 Live Migration with SR-IOV Pass-through - Weidong Han,
Huawei
• https://www...
Implementing SR-IOv failover for Windows guests during live migration
Upcoming SlideShare
Loading in …5
×

of

Implementing SR-IOv failover for Windows guests during live migration Slide 1 Implementing SR-IOv failover for Windows guests during live migration Slide 2 Implementing SR-IOv failover for Windows guests during live migration Slide 3 Implementing SR-IOv failover for Windows guests during live migration Slide 4 Implementing SR-IOv failover for Windows guests during live migration Slide 5 Implementing SR-IOv failover for Windows guests during live migration Slide 6 Implementing SR-IOv failover for Windows guests during live migration Slide 7 Implementing SR-IOv failover for Windows guests during live migration Slide 8 Implementing SR-IOv failover for Windows guests during live migration Slide 9 Implementing SR-IOv failover for Windows guests during live migration Slide 10 Implementing SR-IOv failover for Windows guests during live migration Slide 11 Implementing SR-IOv failover for Windows guests during live migration Slide 12 Implementing SR-IOv failover for Windows guests during live migration Slide 13 Implementing SR-IOv failover for Windows guests during live migration Slide 14 Implementing SR-IOv failover for Windows guests during live migration Slide 15 Implementing SR-IOv failover for Windows guests during live migration Slide 16 Implementing SR-IOv failover for Windows guests during live migration Slide 17 Implementing SR-IOv failover for Windows guests during live migration Slide 18 Implementing SR-IOv failover for Windows guests during live migration Slide 19 Implementing SR-IOv failover for Windows guests during live migration Slide 20 Implementing SR-IOv failover for Windows guests during live migration Slide 21 Implementing SR-IOv failover for Windows guests during live migration Slide 22 Implementing SR-IOv failover for Windows guests during live migration Slide 23 Implementing SR-IOv failover for Windows guests during live migration Slide 24 Implementing SR-IOv failover for Windows guests during live migration Slide 25 Implementing SR-IOv failover for Windows guests during live migration Slide 26 Implementing SR-IOv failover for Windows guests during live migration Slide 27 Implementing SR-IOv failover for Windows guests during live migration Slide 28 Implementing SR-IOv failover for Windows guests during live migration Slide 29 Implementing SR-IOv failover for Windows guests during live migration Slide 30 Implementing SR-IOv failover for Windows guests during live migration Slide 31 Implementing SR-IOv failover for Windows guests during live migration Slide 32 Implementing SR-IOv failover for Windows guests during live migration Slide 33 Implementing SR-IOv failover for Windows guests during live migration Slide 34 Implementing SR-IOv failover for Windows guests during live migration Slide 35 Implementing SR-IOv failover for Windows guests during live migration Slide 36 Implementing SR-IOv failover for Windows guests during live migration Slide 37
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

Implementing SR-IOv failover for Windows guests during live migration

Download to read offline

Presentation from KVM Forum 2020.

In the past, there were several attempted to enable live migration for VMs that are using SR-IOV NICs. We are going to discuss the recent development based on the SR-IOV failover feature in virtio specification and its implementation for the Windows guests. In this session, Annie Li and Yan Vugenfirer will provide an overview of the failover feature and discuss specifics of the Windows guest implementation.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Implementing SR-IOv failover for Windows guests during live migration

  1. 1. Implementing SR-IOV Failover for Windows Guests During Migration Annie Li - Principal Software Engineer, Oracle Yan Vugenfirer - CEO, Daynix http://github.com/virtio-win/
  2. 2. Agenda • Virtio-win drivers • Windows guest terminology • The problem • Different solutions • Failover solution with virtio-net on Windows guest
  3. 3. • Upstream: https://github.com/virtio-win/kvm-guest- drivers-windows/ • Drivers for the major virtio devices: • virtio-net • virtio-blk, virtio-scsi • virtio-balloon, virtio-serial, virtio-vsock, virtio-input, virtio-rng • Panic, fw-cfg • INF files (pci-serial, sm-bus on Q35) Drivers for Windows
  4. 4. • WDF drivers for the “simple” devices • Miniport architecture for network and storage • NDIS • Storport • Scsiport VirtIO Drivers for Windows
  5. 5. • Supported OS • Windows XP, Vista, 7, 8, 8.1, 10 (up to recent builds) • Widows Server 2003, 2008, 2008R2, 2012, 2012R2, 2016, 2019 VirtIO Drivers for Windows
  6. 6. • Send PRs - https://github.com/virtio- win/kvm-guest-drivers-windows/ pulls • Code changes should pass WHQL • We are running WHQL CI on upstream (HCK-CI) How to Contribute
  7. 7. Contributors • And others
  8. 8. NDIS (Network Driver Interface Specification) Architecture NotifyObject Network configuration subsystem HW device HW device Protocol drivers Protocol Driver Miniport Drivers Network Device Miniport driver Miniport edge Protocol edge Intermediate drivers Protocol driver Intermediate driver Protocol driver Miniport driver NDIS User space
  9. 9. • NDIS miniport driver • Basic driver package: • INF file – installation description • SYS file – driver binary • PDB file – symbols for debugging • CAT file - package digital signature Virtio-net (NetKVM) Driver for Windows
  10. 10. The Problem – Live Migration and SR-IOV
  11. 11. Overview of SR-IOV Migration Solutions • Previous efforts and vendor specific HW solutions • Hyper-V and Windows • Linux and VirtIO
  12. 12. Previous Efforts • KVM Forum 2015 Live Migration with SR-IOV Pass-through - Weidong Han, Huawei • KVM Forum 2018 - Live Migration Support for GPU with SR-IOV - Zheng Xiao, Alibaba Cloud; Jerry Jiang & Ken Xue, AMD • KVM Forum 2020 (parallel session) - Device Keepalive State for Local Live Migration and VMM Fast Restart - Jason Zeng, Intel
  13. 13. Overview of Software Solutions • Windows NIC Teaming • Windows MUX Intermediate driver • Hyper-V Solution
  14. 14. Windows NIC Teaming • Similar to bond in Linux • Provides failover capability • Configured through GUI or Powershell Cmdlets in user space
  15. 15. Windows MUX Intermediate Driver • Kernel space solution with various models • One-to-two model for SR-IOV live migration
  16. 16. Windows MUX Intermediate Driver TCPIP VF miniport driver NotifyObject Open Adapter1 Open Adapter2 Network configuration subsystem VirtIO network device Virtual Function network device Protocol driver Protocol Driver Miniport Driver Network Device Virtual Adapter NetKVM Miniport edge Protocol edge One-to-Two MUX Intermediate Driver Internal Binding Other Protocols ndisndis ndis ndis ndis • Netkvm is Open Source driver code for VirtIO network • VF miniport driver is provided by vendor NDIS(Kernel Space) User space
  17. 17. Network Binding of NIC Teaming or MUX
  18. 18. Hyper-V VM Network • Network virtual service client(NetVSC ) • Synthetic data path • SR-IOV data path • Two Installation files(INF) • Share same driver binary
  19. 19. Hyper-V SR-IOV VF Failover TCPIP Other Protocols VF Miniport Driver Hyper-V NetVSC Driver Hyper-V NetVSC Miniport Driver Hyper-V NetVSC Protocol Driver Miniport Driver 1:1 binding Hyper-V Virtual Network Device Virtual Function Device Virtual Network Device Protocol Driver Protocol Driver ndisvf ndisvf • No Bond/Teaming • No NotifyObject • No new Virtual Adapter NDIS(Kernel space)
  20. 20. Network Binding in Hyper-V
  21. 21. Comparison Summary • MUX Driver model: • Complicated, New virtual adapter, Restore offload, NotifyObject. • Hyper-V model: • Simplified, Appropriate for Hyper-V • 2-netdev model in KVM
  22. 22. Windows Network of VirtIO and VF TCPIP Other Protocols VF Miniport Driver NetKVM Miniport Driver VirtIO Network Device Virtual Function Device Protocol Driver Miniport Driver Virtual Network Device ndis ndis NDIS(Kernel sapce)
  23. 23. TCPIP Other Protocols VF Miniport Driver VirtIO NetKVM Driver NotifyObject Network configuration subsystem Virtual Function Device Protocol Driver Miniport Driver Virtual Network Device 1:1 binding ndis ndis ndis ndis VirtIO SR-IOV VF Failover VirtIO Network Device VirtIO Protocol Driver NetKVM Miniport Driver User Space NDIS in Kernel Space • Netkvm is Open Source driver code for VirtIO network • VF miniport driver is provided by vendor
  24. 24. Protocol Driver in 2-netdev Model • Behaves like a bridge • VF adapter is coupled to VirtIO adapter with the same MAC address • Handling TX/RX network data • Object identifiers(OIDs) are wrapped and forwarded, offloads are propagated
  25. 25. Network Binding of VirtIO SR-IOV
  26. 26. Known Issues • DHCP issue • Only happens in specific scenario • Statistics is missing • NetKVM driver needs to keep the statistics of packets sent to or received from VF driver • Old Windows system • Windows Server 2003 and Windows XP • Need to add VF PNP to Notification Object code or to the registry • Possible race during boot?
  27. 27. VirtIO and SR-IOV Failover • VirtIO specification specifics • Feature bit called VIRTIO_NET_F_STANDBY. It is appropriate for 3-netdev model in Linux, but not for Windows 2-netdev model.
  28. 28. Installation • Before • INF for Miniport • After • INF for Miniport • INF for Protocol driver definition and Notify Object • Not part of the guest tools installer: https://github.com/virtio-win/virtio-win-guest-tools-installer
  29. 29. WHQL Certification • Before • Certification of miniport • Automatic review • After • Two steps certification • Automatic review for miniport • Manual review of protocol driver and the final package
  30. 30. Performance From VM to Remote Host(NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
  31. 31. Performance From Remote Host to VM(NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
  32. 32. Performance From VM to VM on Remote Host (NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
  33. 33. annie.li@oracle.com yan@daynix.com
  34. 34. Links – source code • virtio-win drivers source code: • https://github.com/virtio-win/kvm-guest-drivers- windows • MiniPort and Protocol driver: • https://github.com/virtio-win/kvm-guest-drivers- windows/tree/master/NetKVM • Notification object: • https://github.com/virtio-win/kvm-guest-drivers- windows/tree/master/NetKVM/NotifyObject
  35. 35. Links – download binary drivers • https://docs.fedoraproject.org/en-US/quick-docs/ creating-windows-virtual-machines-using-virtio- drivers/index.html
  36. 36. Links – related presentations • KVM Forum 2015 Live Migration with SR-IOV Pass-through - Weidong Han, Huawei • https://www.youtube.com/watch?v=vnwEnzVp9Zo • https://www.linux-kvm.org/images/9/9a/03x07-Juniper-Weidong_Han- LiveMigrationWithSR-IOVPass-through.pdf • KVM Forum 2018 - Live Migration Support for GPU with SR-IOV - Zheng Xiao, Alibaba Cloud; Jerry Jiang & Ken Xue, AMD • https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Live- Migration-Support-for-GPU-with-SRIOV-Challenges-and-Solution-Zheng-Xiao- Alibaba-Cloud-Jerry-Jiang-Ken-Xue-AMD.pdf • KVM Forum 2020 (parallel session) - Device Keepalive State for Local Live Migration and VMM Fast Restart - Jason Zeng, Intel • https://sched.co/eE3W

Presentation from KVM Forum 2020. In the past, there were several attempted to enable live migration for VMs that are using SR-IOV NICs. We are going to discuss the recent development based on the SR-IOV failover feature in virtio specification and its implementation for the Windows guests. In this session, Annie Li and Yan Vugenfirer will provide an overview of the failover feature and discuss specifics of the Windows guest implementation.

Views

Total views

200

On Slideshare

0

From embeds

0

Number of embeds

4

Actions

Downloads

1

Shares

0

Comments

0

Likes

0

×