- Virtualizing Oracle databases on VMware ESX requires careful configuration and monitoring of memory, CPU, and I/O resources to avoid performance issues. - ESX uses memory overcommitment and techniques like ballooning and swapping to manage physical memory across VMs, which can cause problems if they page Oracle memory like the SGA. - Multiple virtual CPUs on a VM must wait to be scheduled on the physical CPUs, showing up as ready time and impacting performance of CPU-intensive workloads. - Dedicating storage, using raw device mapping (RDM), and tuning I/O can help optimize storage performance for virtualized Oracle databases.

1. Optimize Oracle RDBMS on VMware Guy Harrison Director, R&D Melbourne www.guyharrison.net Guy.harrison@quest.com @guyharrison

2. Introductions

5. Agenda Motivations for Virtualization VMware ESX resource management: Memory CPU IO Paravirtualization (OVM) vs Hardware Assisted Virtualization (ESX)

6. Motivations for Virtualization

7. Resistance to Database virtualization

8. DB virtualization is happening

9. Dba-village.com Mar 2010 May 2009

10. Oracle virtualization is lagging....

11. ESX Memory management

12. Managing ESX memory ESX can “overcommit” memory Sum of all VM physical memory allocations > actual ESX physical memory Memory is critical to Oracle server performance SGA memory to reduce datafile IO PGA memory to reduce sort & hash IO ESX uses four methods to share memory: Memory Page Sharing “Ballooning” ESX swapping Memory compression DBA needs to carefully configure to avoid disaster

13. Configuring VM memory VMs Compete for memory in this range Relative Memory Priority for this VM Maximum memory for the VM (dynamic) Minimum Memory for this VM

14. Monitoring VM memory

15. ESX and VM memory ESX Swap ESX swap VM ESX virtual memory Effective VM physical memory ESX physical memory VM virtual memory

16. ESX Ballooning ESX Swap ESX swap Vmmemctl “balloon” VM ESX virtual memory Apparent VM physical memory Effective VM physical memory ESX physical memory VM Swap VM Swap

17. ESX Ballooning As memory grows, ESX balloon driver (vmmemctl) forces VM to page out memory to VM swapfile

18. ESX Ballooning Inside the VM, paging to the swapfile is observed. The guest OS will determine which pages are paged out If LOCK_SGA=TRUE, then the SGA should not be paged.

19. ESX Swapping ESX Swap ESX swap ESX virtual memory VM Effective VM physical memory ESX physical memory VM virtual memory

20. ESX Swapping ESX Swap ESX swap VM Apparent VM physical memory Effective VM physical memory ESX virtual memory ESX physical memory

21. ESX Swapping ESX swaps out VM memory to ESXswapfile

22. ESX Swapping Within the VM, swapping cannot be detected. ESX will determine which memory pages go to disk Particularly occurs when VMware tools are not installed Even if LOCK_SGA=TRUE, SGA memory might be on disk

23. Avoiding Ballooning and swapping memory reservations help avoid ballooning or ESX swapping

24. Other VMware memory management thoughts Memory page sharing Multiple VMs can share an identical page of memory (Oracle code pages, etc) Modern chipsets reduce memory management overhead Multiple hardware page layers Memory compression (new in ESX 4.1) Pages are compressed and written to cache rather than to disk Swapping is more expensive than ballooning Slower to restore memory OS and Oracle get no choice about what gets paged “Double paging” can occur – guest and ESX both page a block of memory

25. Ballooning vs. Swapping Swingbench workload running on Oracle database – from VMWare whitepaper: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf

26. VMware memory recommendations Paging or swapping of PGA or SGA is almost always a Very Bad Thingtm. Use memory reservations to avoid swapping Install VMware tools to minimize swapping Ballooning >> swapping (double paging, paging SGA, high swap-in latency, etc) Set Memory reservation = PGA+SGA+process Overhead Be realistic about memory requirements: In physical machines, we are used to using all available memory In VM, use only the memory you need, freeing up memory for other VMs Oracle advisories (or Spotlight) can show you how much memory is needed Reduce VM reservation and Oracle memory targets in tandem to release memory

28. ESX CPU management

29. VMware CPU management If more virtual CPUs than ESX CPUs, then VCPUs must sometimes wait. Time “stops” inside the VM when this occurs For multi-CPU VMs, it’s (nearly) all or nothing). A VCPU can be in one of three states: Associated with an ESX CPU but idle Associated with an ESX CPU and executing instructions Waiting for ESX CPU to become available Shares and reservations determine which VM wins access to the ESX CPUs

30. Configuring VM CPU VMs compete for CPU in this range Shares determine relative CPU allocated when competing

31. CPU utilization VM “CPU Ready” is the amount of time VM spends waiting on ESX for CPU Inside the VM, CPU stats can be misleading

32. SMP for vCPUs ESX usually has to schedule all vCPUs for a VM simultaneously The more CPUs the harder this is Some CPU is also needed for ESX More is therefore not always better (Thanks to Carl Bradshaw for letting me reprint this diagram from his Oracle on VMWare whitepaper)

33. ESX CPU performance comparisons VT enabled Vs 2 core 1.8 GHz physical machine

34. Programmatic performance

35. Programmatic performance (2)

36. ESX CPU recommendations Use up to date chipsets and ESX software Allocate as few VCPUs as possible to each VM Use reservations and shares to prioritise access to ESX CPU Performance of CPU critical workloads may be disappointing on older hardware Monitor ESX Ready time to determine the “penalty” of competing with other virtual machines

37. ESX IO managment

38. Typical VMWare disk configuration

39. Disk Resource Allocation Disk shares can be used to prioritize IO bandwidth. However, ESX often does not know underlying storage architecture

40. Performant VMware disk configuration

41. Optimal configuration See “Oracle Database Scalability in VMware® ESX” at www.vmware.com/oracle Each virtual disk directly mapped via RDM to dedicated RAID 0 (+1) group 41 Spindles!

42. ESX IO recomendations Follow normal best practice for physical disks Avoid sharing disk workloads Dedicated datastores using VMFS Align virtual disks to physical disks? Consider Raw Device Mapping (RDM) If you can’t optimize IO, avoid IO: Tune, tune, tune SQL Prefer indexed paths Memory configuration Don’t forget about temp IO (sorts, hash joins)

43. Shameless plugs

46. Paravirtualizationvs Hardware Virtualization

47. Paravirtualizationvs “Hardware Virtualization” Virtualization is not emulation.... Where-ever possible, Hypervisor runs native code from OS against underlying hardware Because a virtualized operating system is running outside privileged x86 “ring 0”, direct calls to hardware need special handling. The three main approaches are: Full Virtualization (VMWare on older hardware) ParaVirtualization (Xen, Oracle VM) Hardware Assisted Virtualization (Intel VT, AMD-V)

48. Full virtualization Hardware calls from the Guest are handled by the hypervisor by: Catching the calls as they occur at run time Re-writing the VM image at load time (binary translation) Requires no special hardware Supports any guest OS Relatively Poor performance Used by ESX on older chip-sets Guest OS Hypervisor Ring 0 Hardware

49. Hardware Assisted virtualization Intel VT and AMD-V chips add a “root mode”. Guest can issue instructions from non-root Ring 0. CPU can divert these to hypervisor No changes to OS required Good performance Requires modern chipsets Root Mode Non-Root Mode Guest Hypervisor Ring 0 Hardware

51. Hypercalls are handled by a special VM (dom0 in Xen/OVM)

52. Good performance but requires modified guest OS

53. Xen can use either paravirtualization or hardware assistGuest (domU) Ring 0 Guest (dom0) Hypervisor Hardware

54. RAC and ESX

55. Paravirtualization, ESX and RAC Prior to 11.2.0.2, Oracle relied on paravirtualized kernels to maintain time synchronization for RAC clusters. From 11.2.0.2 Oracle uses Cluster Time Synchronization Service (CTSS) to maintain clock sync, and this works on ESX Therefore, Oracle supports RAC on Vmware ESX only from 11.2.0.2 onwards See Oracle MySupport Note 249212.1

56. References Latest version of this presentation: http://www.slideshare.net/gharriso/optimize-oracle-on-vmware-5271530 My blog (www.guyharrison.net ): http://guyharrison.squarespace.com/blog/2010/2/22/memory-management-for-oracle-databases-on-vmware-esx.html http://guyharrison.squarespace.com/blog/2010/4/9/esx-cpu-optimization-for-oracle-databases.html http://guyharrison.squarespace.com/blog/2010/7/12/stolen-cpu-on-xen-based-virtual-machines.html http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf http://www.vmware.com/files/pdf/Oracle_Databases_on_vSphere_Deployment_Tips.pdf

58. End of Presentation