2. Oracle Open World 2009 2
Agenda
• Introduction to running Oracle Databases on Sun
SPARC systems – Allan Packer
• Best Practices for running Oracle Databases on Sun
SPARC systems – Bob Sneed
3. Oracle Open World 2009 3
Running Oracle DB on Sun SPARC
• Database Performance – the Data Stall Problem
• Background to Sun's SPARC Systems
• Oracle on CMT and M-Series Servers
• Optimizing Oracle Performance on SPARC
4. Oracle Open World 2009 4
The Dark Underbelly of DB Workloads:
Data Stalls
• Data stalls dominate execution time
> 46%-64% of execution time wasted on data stalls
> True for both OLTP and DSS database workloads
> Even more execution time wasted on instruction stalls
– Especially for OLTP, where you waste a further 10%-15%
• This problem applies to typical multi-core chips
> Designed with sophisticated out-of-order cores and
aggressive speculation
– E.g. IBM Power, Intel Core Duo
Source: “Database Servers on Chip Multiprocessors: Limitations and Opportunities”
September 2006, Carnegie Mellon University
5. Oracle Open World 2009 5
Everything Changes with Sun's
CoolThreads CMT Systems
• Data stalls drop to 13%
> 76%-80% of execution time spent on useful computation
> True for both OLTP and DSS database workloads
> Instruction stalls are negligible
> END RESULT: 70% higher throughput than traditional
systems
• BUT: not good for unsaturated workloads
> Single-threaded workloads perform 41% worse
• So CMT is a win, provided you keep the CPUs busy
Source: “Database Servers on Chip Multiprocessors: Limitations and Opportunities”
September 2006, Carnegie Mellon University
6. Oracle Open World 2009 6
A Simple Analogy: Cars are Good!
Designed for
speed and for
the experience
Plenty of
horsepower
Gets you there
fastest
Ideal for 1 or 2
passengers
But – not so
great with lots of
passengers
7. Oracle Open World 2009 7
Buses are Good, Too!
Designed for
efficiency
Not flashy, but
gets you there
promptly
Excellent
fuel economy per
passenger
Carries many
more
passengers per
trip than a car
But – not so
great when
lightly loaded
8. Oracle Open World 2009 8
How Many Application Threads
Does it Take to Consume a Server?
• Making effective use of CPU strands/threads requires
an adequate supply of application processes/threads
• The specifics will depend on the CPU implementation
1 2 4 8 16 32 64 128 256
1 100% 50% 25% 13% 6% 3% 2% 1% 0%
2 100% 100% 50% 25% 13% 6% 3% 2% 1%
4 100% 100% 100% 50% 25% 13% 6% 3% 2%
8 100% 100% 100% 100% 50% 25% 13% 6% 3%
16 100% 100% 100% 100% 100% 50% 25% 13% 6%
32 100% 100% 100% 100% 100% 100% 50% 25% 13%
64 100% 100% 100% 100% 100% 100% 100% 50% 25%
128 100% 100% 100% 100% 100% 100% 100% 100% 50%
256 100% 100% 100% 100% 100% 100% 100% 100% 100%
CPU Count (Hardware Strands/Threads)
Active Application
Processes and/or
Software Threads
Maximum Potential
CPU Utilization
10. Oracle Open World 2009 10
CMT Design – True System on a Chip
• UltraSPARC-T2 – all server functions on one chip
> 8 cores, each with
– 2 Integer pipelines
– 8 hardware strands/threads (4 per pipeline) – total 64
– Solaris and Oracle see 64 CPUs
– Floating Point Unit (FPU)
– Crypto unit, delivering embedded wire-speed cryptography
> Integrated Memory Management Unit (MMU)
> Integrated L1, L2 caches
> Integrated networking
> Integrated PCI-Express
11. Oracle Open World 2009 11
CMT – Multiple Strands Per Core
• 8 hardware strands per core
> Implemented with unique registers per strand
> 2 integer pipelines
> A different strand is scheduled per clock cycle
– Strand skipped if stalled or idle
• Simplified pipeline
> Single strand performance on a dedicated
core is lower than other SPARC cores
> Designed for throughput
• Very efficient, perform well
> Stalled strands efficiently skipped for executable strands
S2
S0
S3
S1
IdleIdleIdleIdle
Idle Idle
Stall
Stall Stall
Stall
Stall
StallRun
Run
Run
Run
Clock Cycles
T1 T2 T3 T4
Strands
12. Oracle Open World 2009 12
CMT Design Points
• Benefits
> High throughput with lots of database connections
> Low power utilization
> Small footprint
• Trade-offs
> Single-user performance less than traditional CPU
architectures
> Single-threaded jobs usually run slower
13. Oracle Open World 2009 13
Oracle Performance Expectations
• Most OLTP environments run very well with CMT
• Understand response time components
> DB CPU usually around 20% of response time
> IO, Network, and application tier makes up the rest
• Bad code is a bottleneck on any server
> CPU-bound bad code bites you sooner on CMT servers
• Serial Batch jobs run slower with CMT
> Check the usual suspects
– Execution Plans, Indexes, Business Logic
> Divide and Conquer
– Increase Concurrency and Parallelism
14. Oracle Open World 2009 14
Running Oracle on CMT
• Single-threaded operations should be tuned
• Parallelism
> The key to achieving optimal throughput
– For DSS and Batch workloads
– For Create Index, RMAN backups, Analyze
> Requires Enterprise Edition
• Concurrency
> The key to achieving optimal throughput
– For OLTP workloads
– For Data Load, Import/Export
• Can often use a combination of both
15. Oracle Open World 2009 15
SPARC Enterprise M-Series Servers
• Mainframe-class reliability
• Highly scalable performance
• Advanced virtualization
• Seamless scalability
• Easy upgradability
• Balanced SMP design
• Price/performance targeted
• Competitive RAS and S/W
• Dense rack mount package
• Highest availability
• Highest absolute performance
• Sophisticated resource control
• Highest scalability
• Entry level enterprise system
• Mainframe-class RAS
• Dense rack mount package
M4000 and M5000
M8000 and M9000
M3000
16. Oracle Open World 2009 16
M-Series Compared to CMT
• What's the Same
> Large CPU counts (potentially)
– Need to increase concurrency and parallelism, as for CMT
– Binary compatibility
• What's Different
> Much better Single-Threaded Performance
> More Scalable
– Much greater capacity for CPU, memory, I/O (storage, network)
> Better RAS
– Reliability, Availability, Serviceability
17. Oracle Open World 2009 17
Oracle on Sun M-Series
• Dynamic Reconfiguration
> Add or remove system boards without Oracle instance
downtime
– Leverages Oracle Dynamic SGA and Solaris DISM
> 5th
generation feature on SPARC Enterprise Servers
• Mixed processor support
> Sun M-series supports mixed speed and mixed
generation CPUs
– Allows customers to implement rolling CPU upgrades without
impacting Oracle instance uptime
– Delivers greater ROI because customers not required to
upgrade all CPUs
18. Oracle Open World 2009 18
Oracle on Sun M-Series
• True memory mirroring
> Supported on SPARC Enterprise M4000-M9000
> Domain services can survive up to 50% loss of memory
without interruption of Oracle services
> Dynamic Reconfiguration can be used to repair memory
boards, again without interruption of Oracle services
• Mixed memory densities with no memory speed
reduction
> Industry practice is to require memory speed to be
reduced to lowest speed when mixing memory capacities
• Current PCIe Technology
> Up to 4GB/sec per card
19. Oracle Open World 2009 19
Oracle on SPARC Servers – Summary
• Sun's CMT Server Advantages
> Low power consumption
> Maximum rack computational density
> Logical Domains (LDoms) powerful and bundled
• Sun's M-Series Server Advantages
> Outstanding scalability
> Enterprise-class RAS features
> Flexible domaining
• Solaris Containers run on both CMT and M-Series
> Secure virtual OS instances, negligible perf overhead
> Recognized by Oracle for licensing purposes
20. Oracle Open World 2009 20
Oracle on Sun Storage
• Sun Storage F5100 Flash Array
> Unprecedented performance and scalability
> Up to 1.92TB capacity
> 1.6m read IOPS, 1.2m write IOPS, 12GB/s throughput
• Sun Flash Accelerator F20 PCIe Card
> Low profile PCIe form factor
> 96GB capacity
> 100k read IOPS, 87k write IOPS, 1.1GB/s throughput
> Windows, Solaris, OpenSolaris, Linux
21. Oracle Open World 2009 21
Oracle Database 11gR2 Flash Cache
• New SGA feature in 11gR2
on Solaris
• Expands DB buffer cache
beyond main memory to flash
> ASM, file system file, raw
device,...
> Best with read-mostly and
read-only workloads
• Ideal with Sun Storage F5100
Flash Array, Sun Flash
Accelerator F20 PCIe Card 0x
1x
2x
3x
4x
5x
Oracle Database 11gR2 Flash Cache
Sun SPARC Enterprise M5000, Sun Storage F5100 Flash Array
No Flash Cache
Flash Cache
Response Time
DB Flash Cache Size
TransactionResponseTimeImprovement
Throughput
22. Oracle Open World 2009 22
Oracle on Sun Storage
• Sun 7000 Unified Storage Arrays
> Hybrid Storage Pool – unique innovation with Flash
acceleration
> Analytics – accessible and comprehensive insight into
Oracle data access patterns and behavior
> Support for Oracle DB, NFS, iSCSI, Oracle Direct NFS,...
> Leverages the end-to-end checksum integrity of ZFS
– For data and metadata
23. Oracle Open World 2009 23
Oracle on Sun Storage
• Sun's Range of Storage Arrays
> Supports an extremely scalable stack
– Running on 1u x86 through high-end SPARC servers
> Rich client protocol support
– Fibre Channel
– Infiniband files or blocks
– 10Gb Ethernet
– NFSv3 and NFSv4
– ZFS with local disks
– Note: tuning required for optimal Oracle performance on ZFS
24. Oracle Open World 2009 24
Oracle on Solaris
• Solaris 10 is the #1 UNIX platform for Oracle
deployments
• Solaris benefits from many years of performance
enhancements designed to benefit Oracle
• Solaris offers outstanding
> Scalability, to hundreds of CPUs
> Observability (e.g. Dtrace)
> Virtualization (Containers/zones, no additional cost)
> File System technology (ZFS – no limits)
> Robustness, Security, Application support,...
25. Oracle Open World 2009 25
Configuring Oracle for SPARC
Servers
• Take maximum possible advantage of
> Parallelism
> Concurrency
26. Oracle Open World 2009 26
Oracle Data Load on Solaris
• Parallel Data Loading option of SQL*Loader
> Allows multiple processes to load data into the same
table
> Create multiple input files
> Create a control file for each input file
> Example of 4 parallel loads
sqlload scott/tiger CONTROL=control1.ctl
DIRECT=TRUE PARALLEL=TRUE
sqlload scott/tiger CONTROL=control2.ctl
DIRECT=TRUE PARALLEL=TRUE
sqlload scott/tiger CONTROL=control3.ctl
DIRECT=TRUE PARALLEL=TRUE
sqlload scott/tiger CONTROL=control3.ctl
DIRECT=TRUE PARALLEL=TRUE
27. Oracle Open World 2009 27
Creating Oracle Indexes on Solaris
• Use the parallel option when creating indexes
• The “unrecoverable” option eliminates redo
• “compute statistics” can be done at the same time
> Default in Oracle11g
• Example
create index CUSTOMER_INDEX1 on CUSTOMER
(CUST_ID, REGION)
parallel(degree 32)
unrecoverable
compute statistics;
28. Oracle Open World 2009 28
Oracle RMAN Backups on Solaris
• Use multiple channels and parallelism to maximize
I/O throughput
> Create multiple backup channels for RMAN
> Tell RMAN how to connect to the channels
> Configure the degree of parallelism
– RMAN> CONFIGURE DEVICE TYPE DISK BACKUP TYPE
TO BACKUPSET PARALLELISM 20;
> Issue the backup command
> Throughput improvements can exceed 15x on CMT
• For an example, refer to
http://blogs.sun.com/glennf/entry/parallelizing_oracle_backup_with_rman
29. Oracle Open World 2009 29
Running OLTP Workloads on Solaris
• Active shadow processes can be scheduled onto
available CPUs by Solaris
• The key: an adequate supply of active users
> As user load increases, response times degrade much
more gracefully on CMT than for traditional systems
> Lightly loaded systems are not optimal for CMT
– Traditional systems and Sun's M-Series deliver faster response
times on lightly loaded systems
30. Oracle Open World 2009 30
Running Batch Workloads on Solaris
• Important candidates for parallelism
> Single-threaded batch jobs perform relatively poorly on
CMT
• Oracle Applications
> Increase the number of Concurrent Managers
> adadmin
– Specify number of workers
– Specify parallelism for maintenance functions
31. Oracle Open World 2009 31
Running DSS Workloads on Solaris
• Parallel Query Option (PQO)
• Example: table scan
> Pre-defining a parallel hint for a table
alter table CUSTOMER parallel (degree 32);
select NAME, ADDRESS, BALANCE_OWING
from CUSTOMER;
> Using a hint to override any pre-defined table parallelism
select /*+ PARALLEL(customer, 32) */
NAME, ADDRESS, BALANCE_OWING
from CUSTOMER;
32. Oracle Open World 2009 32
Running DSS Workloads on Solaris
• Parallel DML
• Examples
> Create as select
alter session enable parallel dml;
create table customer
parallel (degree 32)
as select /*+ parallel (temp_table, 32) */
from temp_table;
alter session disable parallel dml;
> Insert as select
alter session enable parallel dml;
insert /* parallel(customer, 32) */ into customer
select * from temp_table;
alter session disable parallel dml;
33. Oracle Open World 2009 33
For More Information
• Check out the Sun Performance Wiki
> For Sun Performance information generally
– Servers, Storage, and Software
> For Oracle on Sun Performance information in particular
http://wikis.sun.com/display/performance
34. Oracle Open World 2009 34
Agenda
• Introduction to running Oracle Databases on Sun
SPARC systems – Allan Packer
• Best Practices for running Oracle Databases on Sun
SPARC systems – Bob Sneed
35. Oracle Open World 2009 35
Best Practices Categories
• Avoiding known performance bugs
> Some are surprisingly high-impact
> Architecturally-dependent factors are not widely-known
• Copying proven patterns, avoiding known antipatterns
> Filesystem and network choices
> “Big knob” tunables
> Platform- or application-specific settings (some not widely-known)
• In general, being disciplined about performance
> Holistically, including the application code, solution architectures,
operational plans, backup and recovery, business continuance
> Methods, tools, interdisciplinary and interdepartmental alignment,
frameworks, metrics, processes
NOTE: While there is no clear objective line between “Best Practices” and “tuning” - the
distinction is much like how “just do it” relates to “optimization”. Both may be needed to
meet system performance objectives.
36. Oracle Open World 2009 36
Best Practice Topics
• Topics discussed here are the most-frequently-
recurring issues in the global support of Oracle
performance on the Sun Solaris OS
> These are the top 'escalation generators'
> Remedies are often difficult to deploy; configuration
management, logistics, and 'proof points' often get in
the way ...
37. Oracle Open World 2009 37
Oracle Performance Bugs & Patches
• 10.2.0.4 is a Good Place to be, with these fixes
(among others) ...
> Bug #6814520 - Compute efficiency of block checksum code on
SPARC was very poor
> Bug #5752399 - For enabling direct I/O on certain filesytems,
filesystemio_options=setall was broken
• 10.2.0.4 has point patches available for these ...
> Excessive client attach latency & high cost of session initiation
– Bug #8737425 - Excessive close() calls on new session creation
> Scalability/consolidation issue: kstat overhead excessive with
many instances running
– Patch #8531434 – "KSTAT CALLS BY MMNL/CJQ0 INCUR HIGH SYS ...”
> RE: Oracle NUMA features, see Metalink #759565.1
– Patch #8199533 - Disables NUMA features as new default
38. Oracle Open World 2009 38
Filesystem/Feature Selection
• Performance Best Practice: use an unbuffered,
concurrent I/O stack and consistent contiguous on-
disk data placement
> Solutions with these characteristics perform similarly
– RAW
– UFS direct I/O
– QFS with samaio or qwrites
– ASM
– VxFS with ODM, QIO, or direct with 'cio' (VxFS has new 'cio'
option; ODM no longer an extra fee)
– NFS with direct I/O (Of course, performance depends greatly
with NFS server factors and network factors)
> Each solution has its own footnotes
– See: http://blogs.sun.com/bobs
39. Oracle Open World 2009 39
Buffered Filesystem I/O
Where buffered I/O is used, recent performance fixes
are often crucial ...
> With Solaris x64, new VPM cache management is vastly
more-efficient than old segmap mechanism
– CR 6256083 "Need a lightweight file page mapping mechanism
to substitute segmap”; Solaris S10U6 10/08
– ERRATA: Technical Instruction Solution 229118 "Segmap
tuning for file system performance on Solaris[TM], Solaris[TM]
x64"-- QFS 5.0 now does leverage VPM
> With VxFS, new VMODSORT improvements requires
upgrade/patch plus explicit enabling in /etc/system
– See also: SunAlert 201248 - VMODSORT article
– set vxfs:vx_vmodsort=1 * enabler in /etc/system
40. Oracle Open World 2009 40
Control LGWR Priority on Purpose
• Run LGWR FX 60, even when FSS is in use
> Otherwise, TS scheduling will reduce its priority when it
gets busy; exactly opposite what one would want!
• Other background processes may also benefit from
special scheduling, but LGWR is uniquely critical-
path to all transactions
• This is especially important on systems with high
aggregate CPU utilization or systems with slower
single-thread speeds
41. Oracle Open World 2009 41
PGA/UGA Page-Size Controls
• With Automatic Memory Management (AMM; when
pga_aggregate_target is used), 'realfree heap' is
engaged by default, but with default OS page size
> OS overhead from frequent re-mapping of small pages can
become high
• One of these often yields favorable performance
impact
> _realfree_heap_pagesize_hint=4194304 for SPARC (2097152 on
x64) – causes mmap'd realfree heap to use large pages
> _use_realfree_heap=false - causes local heap and malloc() to be
used instead of realfree heap
42. Oracle Open World 2009 42
PGA/UGA Page-Size Controls
• Observability
> Use pmap -sax to observe pages sizes used
> Measuring TLB maintenance overhead depends ...
– On M-Series and older SPARC CPUs, trapstat -T
– On SPARC CMT CPUs and recent x64 CPUs, 'hardware table
walk' decreases cost but obscures observability; counters may
be visible via cpustat
• There's a third rarely-used option to consider ...
> With Oracle Shared Server (MTS), PGA allocated from
SGA will leverage the ISM or DISM performance
advantages of the SGA
43. Oracle Open World 2009 43
RAC: Use Jumbo Frames!
• Jumbo frames delivers multiple benefits
> Reduces GC latency
> Reduces CPU utilization, %sys, and hardware interrupts
> Avoids scaling challenges of IP packet reassembly
> Radically improves scaling of RAC on CMT CPUs
• Implementation notes
> Oracle specifies dedicated interconnects for RAC; jumbo
frames should not be an 'infrastructure project'
> Symantec currently recommend MTU of 1500 with their
LLT product; set with 'lltconfig' or in /etc/llttab
• See also
> http://blogs.sun.com/glennf/entry/jumbo_frames_with_oracle_rac
44. Oracle Open World 2009 44
RAC: Reduce LMS Daemon Count
• Default values are generally too large on SMP
systems, especially CMT and NUMA systems
> Too many LMSD causes interconnect scaling issues,
excessive CPU usage, and GC congestion symptoms
> LMSD work increases as their count increases, so more
is usually not better
> Some of the busiest RAC clusters in the world only use
2-4 LMSD per node
> With consolidation projects, the systemwide total must
not be excessive
45. Oracle Open World 2009 45
RAC: More on LMS Daemons
• From “Oracle RAC Performance Experts Reveal
All” - by Barb Lundhild and Michael Zoll at Oracle:
“Fewer and busier LMS processes may be more
efficient. i.e. monitor their CPU utilization”
> Use prstat -mL to observe
• Best-possible LMSD CPU QoS can be assured by
running them in nointr psrsets
> This is a proven method of optimization, but somewhat
esoteric and cumbersome to administer
> This is often called 'interrupt fencing'
46. Oracle Open World 2009 46
Oracle CPU_COUNT on Solaris
• Oracle automatically detects the number of CPUs in
the system, or current processor set (if one is in
use), and sets CPU_COUNT accordingly
• CPU_COUNT determines
> Minimum SGA size
– Larger counts require bigger SGAs
> The size of various internal Oracle structures
– Can have an impact if you dynamically increase/decrease the
number of CPUs in a domain or processor set
• CPU_COUNT pre-dates modern architectures
> Virtual CPUs and highly-thread CPUs not anticipated
> Defaults often inappropriate on highly-threaded systems
and consolidation platforms
47. Oracle Open World 2009 47
Oracle CPU_COUNT on Solaris
• PLEASE NOTE: Most CPU concurrency with Oracle
derives from the population of client processes, not
the setting of CPU_COUNT
> CPU_COUNT does not constrain CPUs from being used
• CPU_COUNT should be set thoughtfully, depending
on the context ...
> Are CMT or VMT CPUs being used?
> Can PQ jobs take over the system?
> Are multiple instances sharing the same system?
> There is not single right answer, but the defaults are
generally too high!
48. Oracle Open World 2009 48
Concurrency and Scaling
Keep in mind, Oracle's compute concurrency comes
mostly from the client process population ...
> Ideally, large populations are pooled to persistent
connections at the application-server tier
> Alternately, Oracle Shared Server (MTS) can be used to
reduce cost of high-rate client attach/detach
> Parallel Query Option (PQO) - useful in many cases for long-
running queries
– Picking the ideal Degree of Parallelism (DOP) depends on system
architecture, chip architecture, and how much one is sharing
resources with competing workload elements
– With virtualization, analysis of competition can be challenging
> The number of clients that can happily share a CPU depends
on the CPU power and the relative compute content of the
client's transactions
49. Oracle Open World 2009 49
M-Series Considerations
• Consider the Architecture
> PERFORMANCE CONSIDERATIONS FOR DEVELOPERS
UTILIZING SUN SPARC ENTERPRISE® M-SERIES
SERVERS [PDF]
• Multithreading details vary between CPU chips
> http://developers.sun.com/solaris/articles/chip_multi_thread.html
> http://developers.sun.com/solaris/articles/hardware_counters.html
> http://blogs.sun.com/morganic/resource/paralleltermsarticle.html
• Cache-to-cache copies on some CPUs can be
expensive; process migration and data placement
can be important
50. Oracle Open World 2009 50
T-Series (CMT) Considerations
• Mainly, a bottleneck thread will bottleneck sooner
due to the single-thread performance
> The value of controlling priorities and eliminating
inefficiencies is amplified
> Also consider application-critical non-Oracle threads (eg:
"dispatcher", "logger", "reader" ...)
• Out-of-box defaults did not foresee CMT
> Limit cpu_count (Oracle may not startup on high thread-
count systems otherwise)
> Rules of thumb ...
– set cpu_count <= 32 absent a specific requirement
– set cpu_count <= number_of_cores
51. Oracle Open World 2009 51
More T-Series (CMT) Considerations
• Due to the reduced single-thread speed of a CMT
hardware thread ...
• Best Practices are more important on CMT systems!
> Getting it all right pays compound dividends!
All of the above!
52. Oracle Open World 2009 52
Resources
• Cool Tools
> http://cooltools.sunsource.net/
• Blueprints
> http://wikis.sun.com/display/BluePrints/Main
• CoolThreads Experts Q&A Portal
> http://www.sun.com/secure/servers/coolthreads/tnb/contact.jsp
• Performance Wiki
> http://wikis.sun.com/display/performance
• Blogroll
> Glenn Fawcett - http://blogs.sun.com/glennf
> Denis Sheahan - http://blogs.sun.com/deniss
> Tanel Poder - http://blog.tanelpoder.com
> Greg Rahn - http://structureddata.org
> Bob Sneed - http://blogs.sun.com/bobs (paused)
> Allan Packer - http://blogs.sun.com/allanp
53. Oracle Open World 2009 53
Call To Action
• Visit the Sun booth in Moscone South #1101
> See the Sun Storage and Server portfolio in person
> View Sun Oracle solutions that bring Extreme
Innovation to the Enterprise
> Talk to Sun experts and leave with answers
> Get briefed on next-gen Sun Storage and Servers
under NDA
• After the show...
> Visit Sun.COM for Sun BluePrints, White Papers,
Videos, and more!