This document discusses migrating a legacy database platform to an Oracle Exadata platform to realize consolidation savings and modernize the database environment. It provides background on the existing legacy environment, alternatives considered, factors in selecting Exadata, planning and operational considerations for the Exadata migration, lessons learned, and references. The key outcome was migrating from a 5-node Oracle RAC environment on aging hardware to a quarter rack Exadata configuration, which significantly improved performance.

1. Exadata Migration
How to modernize your database platform to realize consolidation savings
Chris Bradham

2. About Applied Engineering Management
Applied Engineering Management (AEM) Corporation Founded in 1986 as a
100% woman-owned business
More than 25 years of profitable growth
Headquartered in Chantilly, VA with offices located in major metropolitan
areas including; Los Angeles, San Antonio, and Jacksonville
Diversified client base including Fortune 500 and major government
agencies
Industry recognized awards and certifications for performance, capability
and delivery

3. Chris Bradham
•Oracle DBA experience 1997 to present (Oracle 7 to 11.2)
•Replication, Materialized Views, Data Guard, GoldenGate, Exadata, RAC
•Part-time Instructor George Mason University (OCA/OCP)
•Oracle Certified Exadata Implementation Specialist, Oracle Certified
Professional (11g), Oracle Performance Tuning Certified(11g), ITIL
Foundation, Security+
•Oracle Application Server, SAN Administration, Linux Administration,
VMWare, IBM / Sun Blade Administration
chris.bradham@aemcorp.com

4. What’s being covered?
•Technology Refresh
•Legacy Environment / Options
•Exadata Components
•Operational Considerations
•Migration Considerations
•Results of Migration
•Lessons Learned
•References
•Q & A

5. Disclaimer
The US Navy, Commander Navy Installations Command (CNIC), nor
any related government organization in no way endorse Oracle
Corporation or the Oracle Exadata Database Machine. The opinions
expressed during the presentation are solely those of the presenter
and in no way reflects those of the government or any government
interest.

6. Background Information
Global multi-service DoD Web-based Housing application
Over 300 schemas
750 Gb of data
4,300 Active Users
4.2 million annual log ins
4,500 Reports Generated Per Day
AEM Corporation responsible for Hosting / Operations & Maintenance / Technology Refresh

7. Legacy Configuration
Physical Application Servers
4 Gbps Fibre Channel
Clarion Disk Array (10k / 15k RPM)
5 Node RAC Servers
Oracle 10gR2 RAC install

8. Pre-Tech Refresh Issues
Legacy hardware over six years old
•Patches (5 nodes, slower machines)
•Deployments, data updates time consuming
•Large or complex reports often hang
•Node evictions due to network / disk speed issues
•Oracle 10.2.0.4 Support ended 6/31/11
Data Warehouse delay due to performance requirements
(Oracle Streams attempt)

9. Alternative 1 : Based on Legacy Solution
Virtualized application servers
Network bonding
8 Gbps Fibre Channel
EMC Disk Array
5 Node RAC Servers
Oracle 11gR2 RAC install

10. Alternative 2 : Based on Exadata Solution
 Virtualized application servers
 Network bonding
 40 Gbps backbone
 Oracle Storage Servers
 2 node Quarter Rack
 Oracle 11gR2 RAC preconfigured
Surprise, we chose Exadata!

11. Exadata Selection Points
•Licensing fees made Exadata the low cost solution
•Total database hardware solution
o2 nodes vs. 5 nodes
oNumber of Vendors
•Patch Strategy
•Sizable and expandable
•Oracle vested to help DoD succeed
•Storage Indexes / Smart Scan / Smart Flash Cache

12. Throughput Gb/Second
80.0
75
70.0
60.0
50.0 2 Gbps Fibre Channel x2
4 Gbps Fibre Channel x2
40.0
8 Gbps Fibre Channel x2
30.0 37
Exadata 1/4 - Disk
20.0 25
Exadata 1/2 - Disk
10.0 12.5 16 Exadata Full - Disk
0.4 0.8 5.4
1.6 Exadata 1/4 - Disk & Flash
0.0
Exadata 1/2 - Disk & Flash
Exadata Full - Disk & Flash

13. Exadata Planning/Considerations
•Pre-configuration customer worksheet (DBM Configuration spreadsheet)
•Initial Setup Activities
•Network Configuration / Software Installation
•Procurement considerations (2 Quarter Racks)
•Space Choices
•High Capacity (3 Tb / 7.2 K RPM)
•High Performance ( 600 Gb / 15k RPM)
•Space Allocation (ASM Disk Group layout)
•78% DATA, 20% RECO, 2% DBFS_DG
o25 Tb <- Production (Triple Mirror)
o8 Tb <- Test (Mirror)

14. Tech Refresh Challenges
•100% hardware replacement and Data Center move
•Narrow window for outage to avoid negative impact on end users
•Performance of system, database growth, and network bandwidth
•Exadata unproven in DoD space at the time (Security)
•Upgrading Database versions (data/code/reports)
Lots of change, what if issues surface???

15. Summary of Choices (X2)
X2-8 Full X2-2 Full X2-2 Half X2-2 Quarter
Initial / Licensing Cost $$$ $$$ $$ $
Database Servers 2 8 4 2
Database Grid Cores 160 96 48 24
Database Grid Memory (GB) 4096 768 (max 1152) 384 (max 576) 192 (max 288)
InfiniBand switches 3 3 3 2
Ethernet switch 1 1 1 1
KVM No Yes Yes Yes
Exadata Storage Servers 14 14 7 3
Storage Grid CPU Cores 168 168 84 36
Raw Flash Capacity 5.3 TB 5.3 TB 2.6 TB 1.1 TB
Raw Storage High Perf 100 TB 100 TB 50 TB 21.6 TB
Capacity High Cap 504 TB 504 TB 252 TB 108 TB
Usable mirrored High Perf 45 TB 45 TB 22.5 TB 9.5 TB
capacity High Cap 224 TB 224 TB 112 TB 48 TB
Usable Triple High Perf 30 TB 30 TB 15 TB 6.5 TB
mirrored capacity High Cap 150 TB 150 TB 75 TB 32 TB

16. Summary of Choices (X3)
X3-8 Full X3-2 Full X3-2 Half X3-2 Quarter X3-2 Eighth
Initial / Licensing Cost $$$ $$$ $$ $ $
Database Servers 2 8 4 2 2
Database Grid Cores 160 128 64 32 16
1024 (max
Database Grid Memory (GB) 4096 512 (max 1024) 256 (max 512) 256 (max 512)
2048)
InfiniBand switches 3 3 3 2 2
Ethernet switch 1 1 1 1 1
KVM Yes Yes Yes Yes Yes
Exadata Storage Servers 14 14 7 3 3
Storage Grid CPU Cores 168 168 84 36 36
Raw Flash Capacity 22.4 TB 22.4 TB 11.2 TB 4.8 TB 2.4TB
Raw Storage High Perf 100 TB 100 TB 50 TB 21.6 TB 10.8TB
Capacity High Cap 504 TB 504 TB 252 TB 108 TB 54 TB
Usable mirrored High Perf 45 TB 45 TB 22.5 TB 9.5 TB 4.5 TB
capacity High Cap 224 TB 224 TB 112 TB 48 TB 23 TB

17. Smart Flash Cache Considerations
Helps with…
•Write-Through cache voids caching data that will not be reused
•Holds hot data, much faster than disk (small, random I/O)
•Data not duplicated from cache in other Storage Servers
•Reduce latency of log write by simultaneous write to flash / disk
(faster writes) with minimal space (512 Mb)
•Write-Back cache 11.2.0.3.9
Don’t touch except for…
•Alter table <table_name> flash_cache keep;
•Create Flash Disks out of the Flash Cache
•Reassign portion for TEMP tablespace on index builds

18. Database Node Considerations
•Database Consolidation
•SGA Settings
•AMM Bad! ASMM Good! (set minimum values)
•Huge Pages (reduce overhead)
•Where’s the shared storage space?
•DBFS is the answer (fix_control=8,ac_timeout=60 and SGA=2Gb)
•Is everything setup correctly?
•Exachk is the answer
•Indexes / Hints / Compression
•Large Segments <- 8 Mb Initial / Next Size with Autoallocate
•TEMP <- BIGFILE, Autoextend 1 Gb, Uniform 1 Mb

19. Exadata Patch Management
Multiple Patches
•Infiniband (once per year)
•DB Nodes / Storage Server (quarterly)
•Bundle Patch (BP) DB Software (quarterly)
•Additional components (Ethernet switch, KVM, PDU)
Bug Fixes included so important to apply
One-off patches / opatch before and after patch
Rolling option time a consideration

20. Security
•DoD 8570 Requirements
•Security Technical Implementation Guide (STIG)
oOracle installation not customizable
oDBFS and idle_time don’t play well together
oAutomatic Service Request (ASR) / Configuration Manager Limitation
oGrid Control / Third Party Certificates (September release)
oBanners / SQLNET.ORA settings impact on tools
Don’t assume security settings will not have impact. Must TEST!!!

21. Migration Strategies
10.2.0.4 to 11.2.0.x Options Considered
•DBFS with external tables (5 to 7 GB/sec file system I/O throughput)
•GoldenGate with datapump (near-zero downtime)
•Datapump
Factors
•Maintenance window
•Risk of data loss
•Familiarity with technology
Whatever the choice, perform multiple trial runs for optimal settings.

22. 2011 – Technical Refresh (Data Center move)
On 9/9/11 at 7pm application
servers at legacy site were Turned
Off:
•Transferred all data to Data Center
•Reloaded data for all installations
•Network outage occurred during
data transfer (2 hours)
•On 9/10/11at 7am New System
Testing was Initiated
• Users were on the system by 3pm
•In Future-> X3 or Storage Expansion

23. Migration Timeline
Text Initial
1/4 Rack Grid DB Cutover
Delivered Migration Oracle Setup / STIG Prod/
Migrate Control Setup to
(Test) Options Setup DBFS Test CAB Load Apply BP Exadata
1/11 2/11 3/11 4/11 5/11 6/11 7/11 8/11 9/11
Chris Initial DB STIG Apply BP Migration 1/4 Rack Migration Grid
First Setup / Test Test Selection Delivered Test Control
Day Load
(Prod) Setup

24. Post Tech Refresh Performance (in hours)
Event Legacy Exadata
BOR1 Process 4 hours 1 hour
BOR2 Process 4 minutes 23 seconds
Datamart IU 3 hours 35 minutes
MV refresh 37 minutes 13 minutes

25. Exadata Lessons Learned
•Ensure hosting center can accommodate Exadata’s dimensions (door height)
•Staff requirements (more communication necessary)
•Smart Scan <- direct path reads, table access full, fast full index scans,
parallel with parallel_degree_policy not auto
•Chained rows / “small” tables void smart scans
•Ideally 2 Exadata Database machines
•EHCC 10x space and performance (DML)
•Expect CPU utilization to decrease
In-Memory
X2-8 Massive Memory X3
2010 All I/Os to Memory 2012

26. Exadata Lessons Learned (cont.)
•Grid Control for monitoring / managing components
•Platinum Support, major assistance
•Exachk and opatch before / after patching
•Standard tuning principles apply (OLTP)
•Bring over OLTP database “AS-IS”
•Time, Experience keys to stability

27. References
http://www.oracle.com/technetwork/database/options/olap/olap-exadata-x2-2-
performance-1429042.pdf
http://www.scribd.com/doc/57026846/Exadata-Smart-Flash-Cache-366203
http://www.oracle.com/technetwork/database/exadata/exadata-technical-whitepaper-
134575.pdf
http://www.oracle.com/technetwork/server-storage/engineered-
systems/exadata/exadata-storage-technical-overview-128045.pdf
http://www.oracle.com/technetwork/database/features/availability/exadata-consolidation-
522500.pdf
http://www.ioug.org/Portals/0/Documents/Anuj%20Mohan.pdf
http://nyoug.org/Presentations/2011/March/Niemiec_Keynote_Exadata.pdf
http://www.oracle.com/technetwork/articles/oem/exadata-commands-part1-402441.html
http://www.centroid.com/webinars/ExadataChangestheGame.pdf
http://www.centroid.com/knowledgebase/blog/using-dbfs-and-external-tables

28. References (cont.)
Database Machine and Exadata Storage Server (888828.1)
Oracle Exadata Database Machine exachk (1070954.1)
Oracle Exadata Best Practices (757552.1)
Best Practices for OLTP on the Sun Oracle Database Machine (1269706.1)
Best Practices for Data Warehousing on Database Machine (1297112.1)
Oracle Sun Database Machine Application Best Practices for Data Warehousing
(1094934.1)
Oracle Sun Database Machine Diagnosability and Troubleshooting Best Practices
(1274324.1)
Expert Oracle Exadata (Osborne, Johnson, Poder)

29. Questions???

30. Backup Slides

31. Exadata Rack Options
2 six Core Processors / 96 Gb RAM per DB node
2 six Core Processors / 24 Gb RAM per Storage Server
Dual ported 40 Gb/sec InfiniBand
Quarter Rack
•2 DB nodes
•2 Infiniband switches
Half Rack
•4 DB nodes
•3 Infiniband switches
Full Rack
•8 DB nodes
•3 Infiniband switches
Infiniband 10x faster than Fiber Channel

32. Cache Hierarchy (Full Rack X2-2)
Database DRAM
768 GB Raw Capacity 100 GB / Second
Flash Cache
5 TB Raw Capacity 50 GB / Second
Disk
100 – 300 TB Raw Capacity 21 GB / Second

33. Quarter Rack Selected
•2 Xeon-based Dual-processor Database Servers (Sun Fire
X4170 M2)
• 24 cores (12 per server)
• 192 GB memory expandable to 288 GB (96 GB per
server expandable to 144 GB)
• 10 GigE connectivity to Data Center
• 4 x 10GbE ports (2 per server)
•1.1 TB High Speed Flash
•3 Exadata Storage Servers X2-2
• All with High Performance 600GB disks
OR
• All with High Capacity 3 TB disks
•2 Sun Datacenter InfiniBand Switch
• 36-port Managed QDR (40Gb/s) switch
•1 “Admin” Cisco Ethernet switch
•Keyboard, Video, Mouse (KVM) hardware Can Upgrade to a Half Rack
•Redundant Power Distributions Units (PDUs) or just add storage

34. Storage Components and Expansion (X2)
-Local Storage (750 GB)
-Storage Servers (12 Disks per server)
Full Half Quarter
InfiniBand switches 3 3 2
Ethernet switch 1 1 1
KVM Yes Yes Yes
Exadata Storage Servers 18 9 4
Storage Grid CPU Cores 216 108 48
Raw Flash Capacity 6.75 TB 3.4 TB 1.5 TB
Raw Storage High Perf 128 TB 64 TB 28 TB
Capacity High Cap 648 TB 324 TB 144 TB
Usable mirrored High Perf 58 TB 29 TB 13 TB
capacity High Cap 288 TB 144 TB 64 TB
Usable Triple High Perf 38 TB 19 TB 8.5TB
mirrored capacity High Cap 192 TB 96 TB 43 TB