RunningFreeBSDonLinuxKVM
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

RunningFreeBSDonLinuxKVM

  • 4,131 views
Uploaded on

AsiaBSDCon 2012...

AsiaBSDCon 2012
Tokyo University of Science, Tokyo, Japan
22-25 March, 2012

Mar 25 17:30 - 18:30 Work-In-Progress session

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
4,131
On Slideshare
4,086
From Embeds
45
Number of Embeds
4

Actions

Shares
Downloads
22
Comments
1
Likes
5

Embeds 45

https://twitter.com 42
https://twimg0-a.akamaihd.net 1
https://si0.twimg.com 1
http://twicli.neocat.jp 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Running FreeBSDon Linux KVM Virtualization ©2003 Project TIMEMACHINE AsiaBSDCon 2012 Tokyo March 25, 2012 Takeshi HASEGAWA [ja] @hasegaw / [en] @hasegaw_en
  • 2. Takeshi HasegawaInterests•  Virtualization (Xen, and KVM)•  Storage•  Snowboarding J In this winter I had great days with my colleges •  Poor at coding. I’m not a software developer.Work Experience•  April 2004- Systems Engineer at Software Research Associates, Inc.•  June 2011- Sales Engineer at Fusion-io 2
  • 3. My works – books and magazines I participated The Best-seller UNIX admin guide with catoonBest-seller Xen book in Japan. characters. More than 15k copies are sold More than 6k copies are sold Our sponsor, Gihyo Books! I discussed the benfit of Linux KVM. And more… 3
  • 4. Running articles on the Web エンジニアなら知っておきたい仮想マシンのしくみ (Now you understand how virtual machine works) http://gihyo.jp/dev/serial/01/vm_work/ 4
  • 5. BSD and myself•  Met FreeBSD 2.2.5 in 1997•  Open Source no matsuri 2000 o  Met JKH and Warner Losh•  2001~2006 This picture is still my treasure! J o  Contracted a dedicated Server with FreeBSD (4.x)•  2010 o  Used V-IMAGE for the network basic session of new-hire training o  Wrote virtio drivers for FreeBSD•  2011-2012 o  Wrote KVM PVClock for FreeBSD IP Networking amiation DEMO Powered by VIMAGE and HTML5 5
  • 6. Running FreeBSDon Linux KVM Virtualization 6
  • 7. Why FreeBSD on Linux KVM?More opportunities to run FreeBSD•  Many service providers are running KVM o  VPS (Virtual Private Server) o  Cloud – such as Sakura Internet (#sacloud)•  Linux KVM is surely handy o  The free-of-charge Type-1 hypervisor o  Guest OS doesn’t require device drivers for the latest hardwareCo-existence of FreeBSD allows leverage of its specialties•  ZFS, BSD Network stacks, jail/vimage, …•  and more! 7
  • 8. However – FreeBSD on Linux KVM runs slowThe biggest Bottlenecks seenin Virtual Machines is: The solution is:•  I/O Bottlenecks •  I/O Para-virtualization –  Disk I/O –  Disk para-virtualization –  Network I/O –  Network para-virtualization seq$write$MB/s$ 90" 80" 70" 60" 50" 40" seq"write"MB/s" 30" 20" 10" 0" Na.ve" FreeBSD"VM"+"SCSI"Emu"Especially FreeBSD is known as“low-performance guest ” on Linux KVM. 8
  • 9. virtio – I/O interface for virtualized guests•  Ring-Buffer Interface definition between Guest and Host Guest•  Independent to hypervisors –  lguest –  Linux KVM virtio virtio virtio virtio –  VirtualBox Ballon Block Block Net –  BHyVe –  Xen last summer, personally I was also working the virtio support on Xen.... virtio virtio virtio virtio Ballon Block Block Net PCI Bus•  Works efficiently Host –  No overhead of H/W emulation –  Less Intel VT/AMD-V world changes –  Less context switches Hardware 9
  • 10. virtio drivers for FreeBSD•  I wrote virtio drivers for FreeBSD – Fall of 2010 o  Rusty Russel, Father of virtio, made a presentation at LinuxCon 2010 “virtio-net guest driver can be implemented by less than 1,000 lines” o  I actually tried on FreeBSD 8.1-R. It took more than 1,000 lines! J o  Finally, I ported: •  virtio interface •  virtio-net (network) •  virtio-blk (block)•  virtio driver for FreeBSD merged to -CURRENT – Winter of 2011 o  Congrats! o  Please note - the merged one is a DIFFERENT work from mine! 10
  • 11. virtio-net increases network performance•  4.7x against rtl8139, 1.3x against e1000 bandwidth improvement! 1.3x 4.7x Test Environment: HP ML115G1, 1.8G 1P2C, PC6400 DRAM (too poor!) 11 Fedora 14 as the Host, and running FreeBSD 8.1-RELEASE as the Guest(2vCPUs)
  • 12. virtio-blk increases block I/O performance … seq$write$MB/s$ 90" 80" 70" 60" 50" 40" seq"write"MB/s" 30" 20" 10" 0" Na.ve" FreeBSD"VM"+"SCSI"Emu" 12
  • 13. virtio-blk increases block I/O performance a bit seq$write$MB/s$ 90" 80" 70" 60" 50" 40" seq"write"MB/s" 30" OOPS! 20" 10" 0" Na.ve" FreeBSD"VM"+"SCSI" FreeBSD"VM"+" Emu" vir.o" 13
  • 14. Hey… what’s happening? 14
  • 15. “Okay I’ll look into that…” 15
  • 16. Latency Distribution – CentOS Guest w/virtioDistribution(%) 91" Linux guest performs native-level performance. Ideal! 81" >79% of IOs are done in <10ms 71" 61" CentOS"5"host" 51" 41" CentOS"VM+Vir;o" 31" 21" 11" 1" 0" 0.5" 1" 1.5" 2" 2.5" 3" 3.5" 4" 4.5" Low latency High 16
  • 17. Latency Distribution – FreeBSD Guest Only 38% of IOs are done in <10ms Distribution(%) 91" 81" FreeBSD guest has some latency 71" bottleneck… 61" FreeBSD"VM+Vir6o" 51" Fedora"14"HDD" 41" FreeBSD"VM+SCSI"Emu" 31" 21" 11" 1" 0" 1" 2" 3" 4" 5" 6" Low latency High 17
  • 18. Latency Distribution – FreeBSD/CentOS ComparisionDistribution(%) 91" 81" Fedora"14"HDD" 71" 61" CentOS5"VM" 51" 41" FreeBSD"VM+Vir>o" 31" 21" FreeBSD"VM+SCSI" 11" Emu" 1" 0" 1" 2" 3" 4" 5" 6" Low latency High 18
  • 19. I/O Characteristics between FreeBSD n’ Linuxfio --filename=/dev/(virtio device) --direct=1 --rw=randwrite --bs=4k --size=2G --numjobs=1 --runtime=10 --group_reporting --name=a CentOS 5.7 Guest on CentOS 5.7Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util vdb 0.00 0.00 0.00 162.00 0.00 0.63 8.00 0.97 5.98 5.98 96.90 vdb 0.00 0.00 0.00 151.49 0.00 0.59 8.00 0.96 6.29 6.31 95.64 vdb 0.00 0.00 0.00 161.00 0.00 0.63 8.00 0.97 6.09 6.04 97.30 vdb 0.00 0.00 0.00 155.00 0.00 0.61 8.00 0.97 6.20 6.26 97.00 vdb 0.00 0.00 0.00 159.00 0.00 0.62 8.00 0.98 6.23 6.16 98.00 FreeBSD 8.1 Guest on Fedora 14Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util dm-6 0.00 0.00 0.00 178.00 0.00 0.35 4.00 0.70 3.88 3.93 70.00 dm-6 0.00 0.00 0.00 160.00 0.00 0.31 4.00 0.73 4.64 4.58 73.30 dm-6 0.00 0.00 0.00 172.00 0.00 0.34 4.00 0.68 3.94 3.98 68.40 dm-6 0.00 0.00 0.00 164.00 0.00 0.32 4.00 0.69 4.20 4.21 69.00 dm-6 0.00 0.00 0.00 168.00 0.00 0.33 4.00 0.75 4.51 4.47 75.10 1.  IO reqs from FreeBSD is intermittent, and the disk is sometimes idle 2.  It seems that Linux queues 2x IO reqs/queue more than FreeBSD 19
  • 20. iostat results in proliant2 (Fedora14)Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util FreeBSD+SCSI Emu dm-6 0.00 0.00 0.00 260.00 0.00 0.51 4.00 0.94 3.63 3.63 94.30 dm-6 0.00 0.00 0.00 264.00 0.00 0.52 4.00 0.94 3.54 3.55 93.80 dm-6 0.00 0.00 0.00 270.00 0.00 0.53 4.00 0.97 3.62 3.60 97.20 dm-6 0.00 0.00 0.00 270.00 0.00 0.53 4.00 0.97 3.58 3.59 96.90 dm-6 0.00 0.00 0.00 264.00 0.00 0.52 4.00 0.97 3.66 3.69 97.40 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util FreeBSD+VirtIO dm-5 0.00 0.00 0.00 266.00 0.00 0.52 4.00 0.99 3.74 3.73 99.10 dm-5 0.00 0.00 0.00 278.00 0.00 0.54 3.97 0.99 3.55 3.56 99.00 dm-5 0.00 0.00 0.00 296.00 0.00 0.58 4.03 1.02 3.46 3.29 97.30 dm-5 0.00 0.00 0.00 272.00 0.00 0.53 4.00 1.03 3.79 3.62 98.40 dm-5 0.00 0.00 0.00 260.00 0.00 0.51 4.00 0.99 3.82 3.81 99.10 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util Linux guest vdb 0.00 0.00 0.00 154.46 0.00 0.60 8.00 3.94 24.20 6.42 99.21 vdb 0.00 0.00 0.00 146.00 0.00 0.57 8.00 3.98 28.44 6.86 100.20 vdb 0.00 0.00 0.00 147.00 0.00 0.57 8.00 3.98 26.01 6.81 100.10 vdb 0.00 0.00 0.00 140.00 0.00 0.55 8.00 4.00 29.05 7.16 100.30 vdb 0.00 0.00 0.00 159.00 0.00 0.62 8.00 3.98 24.55 6.30 100.20 vdb 0.00 0.00 0.00 165.00 0.00 0.64 8.00 3.99 24.22 6.07 100.10 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util Linux native dm-5 0.00 0.00 0.00 259.00 0.00 1.01 8.00 3.97 15.27 3.86 100.00 dm-5 0.00 0.00 0.00 259.00 0.00 1.01 8.00 3.97 15.37 3.86 100.00 dm-5 0.00 0.00 0.00 258.00 0.00 1.01 8.00 3.97 15.50 3.88 100.00 dm-5 0.00 0.00 0.00 250.00 0.00 0.98 8.00 3.99 15.88 4.00 100.00 dm-5 0.00 0.00 0.00 165.00 0.00 0.64 8.00 3.98 16.48 6.06 100.00 20
  • 21. virtio-blk can make FreeBSD Guests faster?No!!Facts•  I/O latency from virtual machine is more than twice of native I/O•  The performance issue is seen in block layer•  virtio-blk surely improved latency distribution (compared to SCSI emu), but it is still ignorable impactThen… what is the bottleneck in the FreeBSD Guests?? o  Interrupt delivery? o  Queueing I/Os to the block device?More deep investigation should be done J 21
  • 22. My another work – KVM Para-virtualized ClockMotivation•  I wanted more reliable timer needed to investigate performace issue of FreeBSD on Linux KVMWhat’s KVM PVClock?•  clocksource which can be found in Linux KVM environment o  “Timecounter” source in FreeBSD world o  More accurate VM time-keeping can be expected•  Inspired by Xen o  Shadows TSC counter of the host processor, and calendar of the host. o  The replica of timer virtualization in Xen o  FreeBSD already has Xen clock support 22
  • 23. The result of the early implementaiton while true; do date; sleep 1; done on FreeBSD Guest The time-slip is less than one second -  The two OSes should be in synchronized theoretically -  No any time-synchronization method required -  tested for 120hours (5 days) while true; do date; sleep 1; done on Linux KVM Host 23
  • 24. Conclusion•  Linux KVM will bring more opportunities to run FreeBSD•  Linux KVM uses virtio for I/O para-virtualizaton•  FreeBSD block layer does not fit well with Linux KVM•  Porting Linux KVM PVClock for FreeBSD•  Any comments? Questions? Suggestions?•  please say hello to me after the session. 24
  • 25. Thank you! 25