Oracle database maximum performance on Exadata machine

1. ⛳ Oracle database
Maximum performance
on Exadata.
A total review of features
Hello everyone,
My name is Alireza Kamrani and I have collected some useful informations about abilities of
implementing Oracle database on Exadata and explaining its features to achieve maximum
performance, I am sharing this article with you in this post and I hope this will be useful for you.
⛳ Overview of Exadata Machine:
Oracle Exadata Database Machine (Exadata) is an engineered system that delivers the ideal
platform for running Oracle Database workloads at an optimal cost. Its scale-out architecture
integrates the latest processors, intelligent storage servers, cutting-edge caching technologies,
and advanced networking features. Customers can harness unique hardware and software
optimizations of Exadata by running Oracle Autonomous Database and Oracle Exadata Database
Service on dedicated Exadata Cloud Infrastructure on Oracle Cloud Infrastructure (OCI) and in
Microsoft Azure data centers through the Oracle Database@Azure o
ff
ering. Additionally, through
Oracle Exadata Cloud@Customer, which uses the same architecture and infrastructure as Oracle
Exadata Database Machine, customers can bring the performance and economics of Oracle
Exadata Database Service and the fully managed Oracle Autonomous Database into their data
centers, thereby assisting in meeting data residency or application architecture requirements.
Lastly, both the Zero Data Loss Recovery Appliance and Zero Data Loss Autonomous Recovery
Service utilize Exadata hardware and software; they simplify Oracle Database backup processes,
accelerate database recovery, and increase protection against cyberattacks.
Here I represent some new features on Exadata:
🚩 Persistent Storage Index
Storage indexes help Oracle Exadata System Softwareto avoid unnecessary I/O operations. By
avoiding unnecessary I/O, a
ff
ected operations perform better. Also, overall system throughput
increases because the saved I/O bandwidth can be used to perform other work.
Oracle Exadata System Software creates and maintains a storage index in shared memory on
Exadata Storage Server. For each storage region, the storage index automatically tracks minimum
and maximum column values and set membership. Storage index functionality works
transparently with no user input. There is no need to create, drop, or tune the storage index.
Starting with Oracle Exadata System Software release21.2.0, the storage index can persist across
cell restarts and upgrades. When the feature is enabled, the storage index shared memory
segments are written to the storage server system disks during a graceful shutdown, and these
are automatically restored during restart. In the event of a cell server crash or o
ffl
oad server crash,

2. the storage index shared memory segments are retained by the operating system and are
immediately available when the software service restarts.
🚩 Persistent Columnar Cache
The columnar cache is a portion of Exadata Smart Flash Cache that contains data in columnar
format. Data in Exadata Smart Flash Cache is automatically converted to columnar format and
stored in the columnar cache to aid query performance.
Before Oracle Exadata System Software release 21.2.0, metadata for managing the columnar
cache is lost when the cell fails or shuts down. Consequently, the columnar cache must be rebuilt
following each cell restart.
Starting with Oracle Exadata System Software release21.2.0, the columnar cache can persist
across cell restarts and upgrades. This feature saves resources associated with rebuilding the
cache and enables more consistent query performance immediately after a cell restart.
🚩 IORM Cluster Plan
Oracle Exadata System Software release 21.2.0extends I/O Resource Management (IORM) to
include the cluster plan. The IORM cluster plan speci
fi
es how storage server resources are
allocated across multiple Oracle Grid Infrastructure clusters. Each directive in a cluster plan
speci
fi
es an allocation to an Oracle Grid Infrastructure cluster, rather than an individual database
or consumer group.
For example, consider a system hosting two clusters;clusterA and clusterB. Now, imagine a
cluster plan with share-based resource allocation where clusterAhas three shares, and clusterB
has one share. In that case, and in the absence of any other IORM plans, all of the databases
running in clusterA share 75% of the I/O resources, while the databases in clusterBshare the
remaining 25%.
🚩 Options to Preserve Redundancy while Dropping a
Physical Disk
Oracle Exadata System Software release 21.2.0introduces the MAINTAIN REDUNDANCY and
NOWAIToptions to the CELLCLI ALTER PHYSICALDISK DROP FOR REPLACEMENT command.
The MAINTAIN REDUNDANCY option streamlines Exadata Storage Server administration by
automatically rebalancing data before dropping the corresponding ASM disks. In previous
releases, maintaining redundancy required additional administrator intervention to manually drop
the corresponding ASM disks and rebalance the data prior to using the ALTER PHYSICALDISK
DROP FOR REPLACEMENT command.
The NOWAIT option allows the ALTER PHYSICALDISK command to complete immediately while
the DROP FOR REPLACEMENT operation runs asynchronously in the background.
🚩 Smart Scan Metadata Sharing
During Smart Scan operations, Oracle Database sends query metadata to the Exadata Storage
Servers. The query metadata contains information like predicated columns, projected columns,
predicates, database session information, bloom
fi
lters, and so on, which is required to o
ffl
oad the
query to the storage servers. Historically, during parallel query execution, each Oracle Database
parallel query server process sends serialized and compressed query metadata to the storage
server, where it is uncompressed and deserialized before use. Furthermore, for each parallel
query, a signi
fi
cant portion of the metadata is common to all of the parallel query server
processes.
🚩 Faster Upgrades with ILOM Pre-Staging
The upgrade process for Oracle Exadata System Software typically includes a
fi
rmware upgrade
for Integrated Lights Out Manager (ILOM). Historically, the ILOM
fi
rmware upgrade was performed

3. as a serial process, taking approximately 30% of the total software upgrade time on each physical
server (database server or storage server).
🚩 Automatic Recovery from Disk Controller Cache
Failure
High Capacity (HC) and Extended (XT) Oracle Exadata Storage Server models have a disk
controller that contains a write-back cache, which is separate and distinct from the write-back
fl
ash cache.
A multiple-bit ECC error on the disk controller cache can crash the server, resulting in the loss of
un
fl
ushed data in the disk controller cache and leaving stale data on the disks. If not handled
carefully, such errors are severe and can result in data loss.
Oracle Exadata System Software release 21.2.0supports automatic recovery following a disk
controller cache failure. Where possible, this feature automatically detects the problem and copies
data from other mirrors to recover all of the disks in the failed server.
🚩 Oracle ACFS I/O Caching in Flash Cache
To optimize I/O e
ffi
ciency, small sequential writes to Oracle Advanced Cluster File System (Oracle
ACFS) are automatically combined into 1 MB chunks. However, before Oracle Exadata System
Software release 21.2.0, these larger writes would bypass the cache and go directly to disk.
Commencing with Oracle Exadata System Softwarerelease 21.2.0, all Oracle ACFS I/O is eligible
for caching in Exadata Smart Flash Cache.
This feature enhances Oracle ACFS performance, especially for applications such as Oracle
GoldenGatetrail
fi
les, having an I/O pro
fi
le dominated by small sequential writes and fast
subsequent reads.
🚩 Deprecation of USB Images
Commencing with the September 2022 Oracle Exadata System Software release updates
(versions 21.2.16 and 22.1.3), USB images are deprecated.
After the deprecation, USB images are no longer available for imaging or re-imaging an Oracle
Exadata Database Machine server and are not included in the Secure Eraser package
(secureeraser_label.zip). Instead of USB image
fi
les, use the PXE option with ISO image
fi
les.
🚩 Deprecation of RAM Cache
Commencing with the September 2022 Oracle Exadata System Software release updates
(versions 21.2.16 and 22.1.3), the cell RAM Cache feature is deprecated.
Existing RAM Cache installations continue to operate as they did in previous releases. However,
new RAM Cache creation requests are not allowed after the deprecation.
The cell RAM Cache feature was introduced in Oracle Exadata System Software version 18.1.0
⛳ Exadata scalability features:
Scale-out architecture enables easy growth
Exadata Database Machine X10M’s scalable design lets customers size initial deployments to
meet current needs and add incremental database and storage servers to support new or
increasing requirements.
Scalable database processing runs demanding
workloads
Customers can utilize as few as 16 licensed CPU cores on Exadata Database Machine X10M or
scale up consumption to thousands of cores to meet the needs of demanding workloads.

4. Consolidating databases on fewer systems and managing them together reduces database
infrastructure and management complexity by letting more resources be managed as a single
unit.
Scalable storage for large databases and data
warehouses
Exadata Database Machine X10M’s high-capacity storage server has 22% more disk storage
capacity than the previous generation system, supporting up to 1.2 PB of usable, triple-mirrored
storage and more than 1 TB/second of analytics throughput in a single rack. Multirack scalability
helps meet the database storage and analytics needs of large organizations.
🚩 All-
fl
ash storage scaling accelerates latency-
sensitive workloads
All-
fl
ash Exadata X10M Extreme Flash storage servers provide up to 630 TB of usable, triple-
mirrored storage capacity, a 2.4X increase over previous generations. In addition, the new
Extreme Flash storage servers increase Exadata Smart Flash Cache capacity to 27.2 TB, further
accelerating Exadata Smart Software features.
🚩 Scalability improves database consolidation
Exadata Database Machine X10M provides high compute and storage density, letting
organizations consolidate their Oracle Database workloads onto smaller con
fi
gurations that cost
less and use less data center space.
🚩 Oracle Linux advantages for Oracle Database on
Exadata
Before focusing on Exadata, it's important to note that Oracle provides a choice between two
Linux kernels—the Red Hat Compatible Kernel (RHCK) and Oracle Linux Unbreakable Enterprise
Kernel (UEK)—and that UEK is meticulously tailored to provide high performance for Oracle
Database and Exadata. Oracle's Exadata, Database, and Linux engineering teams collaborate
closely to enhance the performance of UEK. This includes tuning system calls and C library
interfaces and integrating cutting-edge features, all leading to extraordinary query processing
times and accelerated applications.

5. 🚩 Oracle Linux with UEK is optimized for Exadata
Exadata is exclusively powered by Oracle Linux with UEK. Exadata systems boast a distinctive
feature—the operating system (OS) installed on them is a tailored, pared-down version of the
standard Oracle Linux distribution with UEK. The minimized UEK, a mere 80MB in size, and the
operating system includes only the necessary packages (approximately 1070 packages), device
drivers, and features required to run Oracle Database on Exadata infrastructure and Exadata
System Software (ESS). Ultimately, this results in a smaller installation footprint, fewer updates,
and reduced potential attack surface.

6. 🚩 Exadata System Software (ESS)
Oracle Exadata Storage Server is a highly optimized storage server that runs Oracle Exadata
System Software to store and access Oracle Database data.
With traditional storage, data is transferred to the database server for processing. In contrast,
Oracle Exadata System Software provides database-aware storage services, such as the ability to
o
ffl
oad SQL and other database processing from the database server, while remaining transparent
to the SQL processing and database applications. Oracle Exadata storage servers process data
at the storage level, and pass only what is needed to the database servers.
Oracle Exadata System Software is installed on both the storage servers and the database
servers. Oracle Exadata System Software o
ffl
oads some SQL processing from the database
server to the storage servers. Oracle Exadata System Software enables function shipping
between the database instance and the underlying storage, in addition to traditional data
shipping. Function shipping greatly reduces the amount of data processing that must be done by
the database server. Eliminating data transfers and database server workload can greatly bene
fi
t
query processing operations that often become bandwidth constrained. Eliminating data transfers
can also provide a signi
fi
cant bene
fi
t to online transaction processing (OLTP) systems that include
large batch and report processing operations.
The hardware components of Oracle Exadata Storage Server are carefully chosen to match the
needs of high-performance processing. The Oracle Exadata System Software is optimized to take
maximum advantage of the hardware components. Each storage server delivers outstanding
processing bandwidth for data stored on disk, often several times better than traditional solutions.
Oracle Exadata storage servers use state-of-the-art RDMA Network Fabric interconnections
between servers and storage. Each RDMA Network Fabric link provides bandwidth of 40 Gb/s for
In
fi
niBand Network Fabric or 100 Gb/s for RoCE Network Fabric. Additionally, the interconnection
protocol uses direct data placement, also referred to as direct memory access (DMA), to ensure
low CPU overhead by directly moving data from the wire to database bu
ff
ers with no extra copies.
The RDMA Network Fabric has the
fl
exibility of a LAN network with the e
ffi
ciency of a storage
area network (SAN). With an RDMA Network Fabric network, Oracle Exadata eliminates network
bottlenecks that could reduce performance. This RDMA Network Fabric network also provides a
high performance cluster interconnection for Oracle Real Application Clusters (Oracle RAC)
servers.
Each storage server contains persistent storage media, which may be a mixture of hard disk
drives (HDDs) and
fl
ash devices. There are several con
fi
gurations of storage server each
con
fi
gured to maximize some aspect of storage based on your requirements.
For example, Exadata takes advantage of Remote Direct Memory Access (RDMA) to facilitate
direct memory access from the database servers to the storage servers. The required kernel and
system packages for this compelling feature are not merely included in Oracle Linux with UEK for
Exadata use; they are carefully integrated, rigorously tested, and delivered to help ensure superior
performance and security. Exadata's singular focus on being the ideal platform for Oracle
Database, combined with co-engineering with the Oracle Linux team, translates to exceptional
performance, improved upgrade time, and hardened system security.
🚩 Superior data throughput and scalability
Exadata utilizes an RDMA over Converged Ethernet (RoCE) network fabric to connect each
database server to every storage server. This network provides low latency and ample bandwidth
for rapid data access and transfer rates. Beyond speed, the RoCE fabric includes capabilities
such as zero packet loss messaging, direct data access without CPU involvement, and KVM-
based virtualization security.
As database workloads grow, the RoCE network fabric plays a central role in Exadata's scale-out
architecture. Customers can add additional databases and/or storage servers online to increase
performance and capacity. Should the demand exceed the capacity of a single physical rack,
multiple racks can be interconnected without downtime to meet the performance, scalability, and
availability needs of any environment.
RDMA is an integral part of the high-performance architecture of Exadata and has been
continuously enhanced with each successive generation of Exadata, underpinning Exadata-only
technologies such as Exadata RDMA Memory (XRMEM) Data Accelerator and Exafusion Direct-

7. to-Wire Protocol. Moreover, it optimizes the e
ffi
ciency of complex, mission-critical OLTP and
analytics workloads, as well as modern applications using JSON, blockchain, spatial, graph, and
vector processing within Oracle Database.
To fully leverage Oracle Linux with the minimized UEK and harness the comprehensive features
for which Exadata is renowned, specialized software known as ESS is installed and
operated on all database and storage servers within the Database Machine. ESS enables a range
of features such as XRMEM Data Accelerator, Exadata Smart Flash Cache (Flash Cache), Smart
Scan, Storage Indexes, Hybrid Columnar Compression, Database In-Memory Columnar Cache on
Storage Servers, and many more.
RDMA (Remote Direct Memory Access) allows one computer to directly access data from
another without operating system or CPU involvement, and has high bandwidth and low latency.
The network card directly reads from and writes to memory with no extra copying or bu
ff
ering and
with very low latency. RDMA is an integral part of the Exadata high-performance architecture, and
has been tuned and enhanced over the past decade, underpinning several Exadata-only
technologies such as Exafusion Direct-to-Wire Protocol.
Exadata RDMA Memory (XRMEM) in the Exadata Storage Servers is leveraged as a shared read
accelerator. The XRMEM Data Accelerator is a memory cache tier in front of Flash Cache,
enabling orders of magnitude lower latency accessing remotely stored data. By utilizing RDMA to
access memory remotely, XRMEM Data Accelerator bypasses the network and I/O stack,
eliminating expensive CPU interrupts and context switches, and reducing latency by more than
10x, from 200 µs to less than 17 µs. Smart Exadata System Software also ensures data is
mirrored across storage servers, which provides additional fault tolerance. Exadata's unique end-
to-end integration between Oracle Database and Exadata Storage Servers automatically caches
the hottest data blocks e
ffi
ciently between the bu
ff
er cache in database servers and XRMEM and
Flash Cache in storage servers. XRMEM is a shared storage tier across all of the storage nodes,
which means the aggregate performance of this cache can be dynamically used by any database
instance on any database server. This is a signi
fi
cant advantage over general-purpose storage
architectures, which preclude sharing storage resources across database instances.
Exafusion Direct-to-Wire Protocol is a database optimization for Exadata that allows Oracle
Real Application Cluster (RAC) Databases to send and recieve messages directly between
instances circumventing the local operating system and network stack. Exafusion leads to
response time improvements and greater database scalability by reducing the processing
requirements of these inter-instance messages.
In an Oracle RAC envirnoment, each database instance caches data blocks in its System Global
Area (SGA) to satisfy queries and transactions. If another instance in the cluster requires that
block, it is passed between instances.
After one instance caches data, any other instance within the same cluster database can acquire
a block image from another instance in the same database faster than by reading the block from
disk. Cache Fusion moves current blocks between instances rather than re-reading the blocks
from disk. When a consistent block is needed or a changed block is required on another instance,
Cache Fusion transfers the block image directly between the a
ff
ected instances. Oracle RAC uses
the private interconnect for interinstance communication and block transfers.
Exafusion Direct-to-Wire Protocol is an optimization to the way Oracle RAC implements Cache
Fusion on Exadata, enabling Oracle Database 12.1.0.2 (BP13) and higher to directly interface with

8. the RDMA over Converged Ethernet (RoCE) network cards bypassing the Operating System
kernel, and network software thereby improving performance for OLTP workloads signi
fi
cantly
🚩 Lower OLTP I/O latency from data caching in remote
memory
The XRMEM Data Accelerator is an automatically managed, shared read-accelerator memory
cache tier in front of Flash Cache, which enables orders of magnitude lower latency when
accessing remotely stored data. Oracle Database uses RDMA to remotely access the XRMEM
Data Accelerator memory, bypassing the network and I/O stack on both the database and storage
servers. This eliminates costly CPU interrupts and context switches and reduces latency by more
than 10X, from 200 μs to as low as 17 μs. Exadata's end-to-end integration between Oracle
Database and Exadata Storage Servers automatically caches the hottest data e
ffi
ciently between
the database bu
ff
er cache and Flash Cache in the storage servers, thereby increasing the e
ffi
cacy
of RDMA-enabled Oracle Linux with UEK.

9. 🚩 Superior performance produced by intelligent data
management
Exadata Storage Servers incorporate PCIe Flash Cards, which function as a caching tier between
Exadata RDMA Memory and persistent storage, commonly referred to as Exadata Smart Flash
Cache. This caching mechanism is used in conjunction with the XRMEM Data Accelerator to
automatically cache frequently accessed data, while less frequently accessed data remains on
disk. This approach combines the high I/O rates and fast response times of
fl
ash with the large
capacity and cost e
ffi
ciency of disk. Exadata possesses a unique understanding of database
workloads and can intelligently avoid caching data that negatively impacts overall performance.
Smart Scan, also known as SQL O
ffl
oad to Storage and exclusive to Exadata, pushes SQL query
processing down to the storage servers. It performs tasks such as
fi
ltering, column selection,

10. joins, and aggregation within the storage tier, eliminating the need to bring all queried data to the
database tier to begin with. Conventional storage arrays only deliver a fraction of the aggregate
bandwidth available from their
fl
ash drives due to the network link between the database server
and storage becoming a signi
fi
cant bottleneck.
A key database format-aware storage server optimization is the Storage Index. When smart scans
run on storage servers, they populate an in-memory metadata cache that records the ranges of
column values stored within each of the data units accessed by the scan. This metadata can be
used by future smart scans to determine, based on their
fi
lter predicates, whether to visit or skip a
certain data unit. As a result, there can be a signi
fi
cant reduction in the number of storage I/O
operations issued by the smart scan, depending on the selectivity of the
fi
lter.
🚩 Reduced overheads through optimized
communication
Exafusion is the next-generation networking protocol uniquely available on Oracle Exadata.
Oracle Database utilizes Exafusion to eliminate context switching and minimize OS overhead
through direct-to-wire messaging, yielding a 3X improvement in latency compared to non-Exadata
platforms relying on traditional messaging models. Additionally, there is a substantial reduction in
CPU utilization associated with inter-instance communications. Overall, this acceleration in
messaging directly bene
fi
ts application performance, particularly for those deployed on Oracle
Real Application Clusters (RAC).
When deployed on Exadata, Oracle Database maintains an In-Memory Commit Cache. Each
Oracle RAC instance maintains a cache of local transactions and commit state, which remote
instances can read using RDMA. This commit cache facilitates bulk transaction state lookup,
which reduces single transaction lookup messages and substantially improves both query and
DML processing time.
🚩 Enhanced high-availability
In addition to the Maximum Availablity Architecturefeatures and best practices that Exadata is
built on, Exadata Instant Failure Detection distinctively enhances Oracle Database availability. By
leveraging RDMA, instead of TCP/IP heartbeat messages, it rapidly detects—sub-second
detection—any failures in a database or storage server. Without Exadata Instant Failure Detection,
server failure detection typically takes a minute or more, during which time the application may
experience brownouts or inconsistent performance.
🚩 Database consolidation on Exadata
Isolation is crucial in hosted and shared service provider and DevTest environments. Consolidated
environments running on Exadata X10M may also use KVM-based Virtual Machines (VM Guests)
and Secure RDMA Fabric Isolation for strong isolation between workloads. When using
virtualization, Exadata can securely deploy multiple VM clusters—running either the same or
di
ff
erent Exadata software, grid infrastructure, or database versions—all on the same set of
database servers.
🚩 Elevated security without interruption
Exadata database servers leverage the Oracle Ksplice zero-downtime capability of Oracle Linux
to apply security patches while the OS stays online, meaning that no workloads or systems need
to be interrupted. For the Exadata Database Machine, customers can use Ksplice to apply
updates to the kernel on database servers. For Exadata Cloud@Customer, when possible,
updates deployed by Oracle are applied to running systems without downtime through the use of
Ksplice. For other updates that require a component restart, Oracle performs the component
restart in a rolling fashion to help ensure service availability during the update process.
Exadata's Linux con
fi
guration exceeds 90% on the Defense Information Systems Agency
(DISA) Security Technical Implementation Guide (STIG)Security Content Automation Protocol
(SCAP)benchmark right out of the factory. Coupled with Exadata-optimized Linux security
features such as Secure Boot, Advanced Intrusion Detection Environment (AIDE), Security

11. Enhanced Linux (SELinux), LDAP, Kerberos support, Federal Information Processing Standard
(FIPS) 140-2 support, Secure Computing (seccomp) on Storage Servers, and the full suite of
Oracle Database security features, Exadata delivers end-to-end data protection without
compromising performance or availability.
🚩 Fueling business success worldwide
Today, thousands of businesses spanning industries such as retail,
fi
nance, and
telecommunications are running their most critical and demanding database workloads on
Exadata, powered by Oracle Linux. Irrespective of the workload—whether it's OLTP, data
warehousing, or machine learning—the dynamic combination of Exadata, Oracle Database,
and Oracle Linux, empowers our customers globally to handle mission-critical workloads in a
deployment model tailored to their requirements.
Sincerely,
Alireza Kamrani
Oracle technical Consultant.