On X86 systems, using an Unbreakable Enterprise Kernel (UEK) is recommended over other enterprise distributions as it provides better hardware support, security patches, and testing from the larger Linux community. Key configuration recommendations include enabling maximum CPU performance in BIOS, using memory types validated by Oracle, ensuring proper NUMA and CPU frequency settings, and installing only Oracle-validated packages to avoid issues. Monitoring tools like top, iostat, sar and ksar help identify any CPU, memory, disk or I/O bottlenecks.

1. On X86 systems
Baruch Osoveskiy Senior Consultant Brillix

2. Who Am I
• Senior Consultant in brillix
• Unix-Linux System Admin since 1997
• DBA oracle and MySQL since 2000
• Linux and security Consultant
• blogger in ildba.co.il

3. • Enterprise Distributions
• Kernel
• Packages
• Drivers

4. ◦ Use Unbreakable Enterprise Kernel (UEK), Redhat ,
Suse Whey ?
 Better optimized for large systems and workloads
 Better hardware support
 Modern Linux features
 Patches required for correct Oracle product operation

5. • Bug Fixes from upstream
• Most Bug Fixes originate Upstream and are backported to Enterprise Distributions
• More code change from upstream means more time before patches are backported -- if it’s even possible to do so
• More time for security patches to be backported to the Enterprise versions
• Bug Fixes in Upstream apply cleanly to UEK
• Better testing
• Code is tested by the whole Linux community, not dependent on one OS vendor and their customers
• You run the same code that’s in upstream
• No backporting/scaffolding to use the latest Linux kernel features (i.e. NFSv4, TCP fast open, etc)
• Better contributions
• Largest amount of Developers and Company Contributions
• Major Backports not required to provide cutting edge features
• New features seamlessly used by Oracle products

6. • Updates contain critical security and bug fixes.
• Most Enterprise Distribution updates contain new features.
• “Security Errata” can contain new features
• Updates often contain 1000s of lines of new code from upstream.
• Includes features, bug fixes, enhancements and other tweaks
• Install Only security and bug fixes to avoid down time from new and
untested features
• The DB server is not your Laptop do not install unknown/new software.

7. • Use Oracle validated
• http://www.oracle.com/technetwork/server-storage/linux/validated-configurations-
085828.html
• Install only base + Oracle validated packages
• Do not install games, applications .. on the production servers

8. #cd /etc/yum.repos.d
# wget http://public-yum.oracle.com/public-yum-ol6.repo
# yum list
#yum install oracle-rdbms-server-11gR2-preinstall

9. • BIOS
• CPU
• Memory TYPE , swap
• Disk (Disks in the more the better )
• Virtualization

10. Motherboard
Handle 0x0003, DMI type 2, 16 bytes
Base Board Information
Manufacturer: Intel
Product Name: S5000PAL0
Processor
Processor Information
Version: Intel(R) Xeon(R) CPU X5355
Memory
Handle 0x0034, DMI type 17, 27 bytes
Memory Device
Data Width: 64 bits
Size: 2048 MB
Form Factor: DIMM
Set: 1
Locator: ONBOARD DIMM_A1
Bank Locator: Not Specified
Type: DDR2
Type Detail: Synchronous
Speed: 667 MHz (1.5 ns)
• Motherboard
4 Memory Channels (S5000PAL0) 8 Slots
(A1/A2/B1/B2/D1/D2)Channels
• CPU
Intel ClovertownCPUs
1333Mhz (Dual Independent FSB)
Bandwidth 10666 MB/s per FSB
21 GB/s Maximum FSB Bandwdith
• Memory
Memory DDR2 667 = PC2-5300
4 Memory Channels at 5.3GB/s each
Memory Bandwidth of 21 GB/s from all 4
channels
16GB memory in total
http://ark.intel.com/

11. • Always check for appropriate BIOS settings Look out for:
• CPU features
• Enable Maximum Performance in the BIOS
• Memory
• Enable numa
• Power Management

12. • will give you 35% better performance ( Test On OLTP).
• SMT Simultaneous Multi-Threading
• Run 2 threads at the same time per core
• Do I have HT ?
• Ensuring that HT is enabled at the BIOS.
• grep -e "model name" /proc/cpuinfo
• http://ark.intel.com/
• Do not Enable TH on I/O bound server it only will make it worse.

13.  cpufreq
you can dynamically scale processor frequencies through the CPUfreq subsystem.
◦ Enable Maximum Performance in the BIOS
◦ /sys/devices/system/cpu/cpu<n>/cpufreq/scaling_governor
◦ On Redhat 5.x default is performance
◦ On Redhat 6.x default is normal
◦ echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

14. sudo modprobe cpufreq_conservative
sudo modprobe cpufreq_ondemand
sudo modprobe cpufreq_powersave
sudo modprobe cpufreq_stats
sudo modprobe cpufreq_userspace
/etc/init.d/cpuspeed status
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
grep -i mhz /proc/cpuinfo
cpu MHz : 2367.330

15. • Load of more then n on n cpu server is bad
• Load of n cpu on n cpu server is good
= load of 1.00
= load of 0.50
= load of 1.70

16. • Use Load to find if the server is CPU bound
• one, five, and fifteen minute averages
• In LINUX CPU bound can impact I/O

17.  cat /proc/meminfo
◦ Free = Cached + Free
◦ All free space on Linux is used for pagecache.
◦ This behavior can be controlled by cgroups.
◦ PageTables large use HugePages.
 /dev/shm
◦ implementation of traditional shared memory (ramfs )
◦ Used by Automatic Memory Management ( AMM MEMORY_TARGET )
◦ Not working with Hugepages ID 1134002.1

18. • Oracle will recognize NUMA systems and adjust memory and scheduling operations
accordingly and NUMA technology allows for faster communication between
distributed memory in a multi-processor server. ID 759565.1
• !! Disabling or enabling NUMA Will change application performance. !!
• 8 sockets and beyond may see gains of approximately 5%
• Enbale on bios level and in grub.conf remove numa=off
• _enable_NUMA_optimization=TRUE (look for bugs on your version before enable)
• dmesg | grep -i numa
NUMA: Initialized distance table, cnt=2
NUMA: Node 0 [0,c0000000) + [100000000,1040000000) -> [0,1040000000)
pci_bus 0000:00: on NUMA node 0 (pxm 0)
pci_bus 0000:80: on NUMA node 1 (pxm 1)

19. • Without HugePages the memory of the is divided into 4K pages
• Using HugePages the page size is increased to 2MB (configurable to 1G )
• HugePages reducing the total number of pages to be managed by the
kernel
• reducing the amount of memory required to hold the page table in memory.

20. • Use Hugepages Oracle Doc ID 749851.1
• Reduce footprint of individual Oracle database connections.
• Increase performance and scalability with fewer tlb misses.
• Requires manual tuning after SGA changes, and does not work with AMM
(/dev/shm).

21. Without Hugepages
o 200 Connections to a
12.9GB SGA
o Before DB Startup
Pagetables: 7400 kB
o After DB Startup Pagetables:
652900 kB
o After 200 PQ Slave run query
o Pagetables: 6189248k
o Time to complete:
00:10:23.60
With Hugepages
o 200 Connections to a 12.9GB
SGA
o Before DB Startup PageTables:
7748 kB
o After DB Startup Pagetables:
21288 kB
o After 200 PQ slaves run query
o Pagetables: 80564 kB
o Time to complete: 00:00:18.77

22.  Use Hugepages with VMs for non-swappable, shared pagetables.
 Hugepages must allocated in the guest VM and the hypervisor
 Oracle VM 3.2.6 contains support for pv-hugepages

23.  What about Swap?
◦ Modern Linux distributions Do not use Swap (swappiness is very low )
◦ Swap is for OS services only. I do not recommend swap = ram.
◦ Check vmstat output: ensure swap
◦ Do not use Swap as memory – buy more memory
◦ If you have free memory
 echo 10 > /proc/sys/vm/swappiness
 vm.swappiness in /etc/sysctl.conf

24. • Disks – the more the better
• Do not mix.
• Use RAID
• Use Hardware RAID
• RAID 1+0 is best for write performance (logs).
• RAID 5 is best for read performance.
RAID Level
Total array
capacity
Fault tolerance Read speed (4k) Write speed (4k)
RAID-1+0
500GB x 4 disks
1000 GB 1 disk 2X 2X
RAID-5
500GB x 3 disks
1000 GB 1 disk 3X
Speed of a RAID 5
depends on
controller

25. • High “log file sync” event time .
• Do Not Use Raid 5 on Redo Logs (low write performance).
• Upgrading the CPU enabled more throughput increase for redo (LGWR also requires CPU)
• reducing the overall number of commits by batching transactions can have a very beneficial effect.
• See if any of the processing can use the COMMIT NOWAIT option.
• See if any activity can safely be done with NOLOGGING / UNRECOVERABLE options.
• Enlarge the redologs so the logs switch between 15 to 20 minutes.
• ID 34592.1

26.  On Linux Use ASM (Block/RAW Device, O_DIRECT )
 Raw Devices deprecated by OUI for Oracle 11.2 ID 357492.1
 Raw Devices may still bring benefits for intensive redo and large redo log
files
 Use udev or asmlib to Control Devices

27.  If using file system Bypass journaling when you create a file system., use EXT-2 or
EXT-4 with journaling turned off,
 journaling turned off eliminates double writes.
 “noatime” option eliminates the need for the system to create writes to the file
system when objects are only being read.
 To Creaet partition and to disable DOS compatibility
fdisk -c -u /dev/sda1
 To turn off journaling, execute:
tune4fs -O ^has_journal /dev/sda1
mount -t ext4 -o noatime /dev/sda1 /oradata

28. Device w/s wMB/s avgrq-sz avqqu-sz avwait svctm %util
sdb1 21357.33 167.86 16.10 1.51 0.07 0.02 44.53
Device w/s wMB/s avgrq-sz avqqu-sz avwait svctm %util
sdd1 3343.00 130.68 80.06 3.25 0.97 0.25 83.97
SSD
HDD
iostat information recorded during the ASM tests SSD/RAW, HDD/RAW, 50GB over a 5 minute period
the redo on the 8 x SSD drives is writing 1.28X more data per second and doing 6.4X the writes/second
although the avgrq-sz shows that the HDD configuration is writing more data for each operation.
However, the avwait, svctm and %util show the the HDD configuration is busier and responding slower.

29. • Top 5 Timed Events (AWR) looked as follows:
Event Waits Time(s) Avg wait (ms) % DB time Wait Class
DB CPU 19,832 78.42
log file sync 6,700,242 4,059 1 16.05 Commit
Event Waits Time(s) Avg wait (ms) % DB time Wait Class
DB CPU 14,255 52.53
log file sync 5,366,376 12,709 2 46.83 Commit
SSD
HDD

30. • DB Smart Flash Cache is new (11.2) extension for buffer cache area.
• extension to the SGA as L2 cache
ID 1317950.1
db_flash_cache_file = <+FLASH/filename>
db_flash_cache_size = <flash pool size>
alter [table|index object_name] storage (flash_cache keep);

32. • Look for high I/O wait (%wa in top, await iostat)
• Look at %util for disk saturation.
• In the AWR most of DB Time is I/O.

33.  Virtualization performance is proportional to native performance
 VM Drivers Vs Native Drivers Have ~16% Overhead

34. • top
• iostat –Nx 1 100
• Sar
• Ksar
• Oracle Orion Calibration Tool
http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#PFG
RF95244

35. • From Redhat 6.x (6.2 best) and EUK 3
• cgroups: Control Groups for Linux Containers
• Provide fine grained control over system resources
• Can be used to throttle page cache use by backup
processes - Often the reason why systems are slower
after overnight backups

36. cgroup

37. Cgroup How To Use
yum install libcgroup
/etc/init.d/cgconfig start
/etc/cgconfig.conf
mount {
cpu = /cgroup/cpu;
memory = /cgroup/memory;
}
group http {
memory {
memory.limit_in_bytes = 10M;
}

38. Thank You!
Baruch Osoveskiy
054-4746164
baruch@brillix.co.il