The document discusses lessons learned from migrating a Department of Defense application from a legacy Oracle database environment to an Oracle Exadata database platform. Key points include choosing Exadata for its licensing costs, manageability as a total database solution, and Oracle's commitment to helping the DoD succeed. The migration involved a complete hardware replacement and data center move with a narrow window. Testing multiple migration strategies and patch management approaches was important. Performance improved significantly on Exadata. Ensuring the hosting center could support Exadata's size and requiring more staff communication were also lessons.

1. Lessons Learned:
A DoD Exadata Migration
Chris Bradham

2. Vizuri – an Operating Division of AEM
Applied Engineering Management (AEM) Corporation Founded in 1986 as a
100% woman-owned business
More than 25 years of profitable growth
Headquartered in Chantilly, VA with offices located in major metropolitan
areas including; Los Angeles, San Antonio, and Jacksonville
Diversified client base including Fortune 500 and major government
agencies
Industry recognized awards and certifications for performance, capability
and delivery

3. Chris Bradham
• Oracle DBA experience 1997 to present (Oracle 7 to 11.2)
• Replication, Materialized Views, Data Guard, GoldenGate, Exadata, RAC
• Part-time Instructor George Mason University (OCA/OCP)
• Oracle Certified Exadata Implementation Specialist, Oracle Certified
Professional (11g), Oracle Performance Tuning Certified(11g), ITIL
Foundation, Security+
• Oracle Application Server, SAN Administration, Linux Administration,
VMWare, IBM / Sun Blade Administration
cbradham@vizuri.com

4. What’s being covered?
• Technology Refresh
• Legacy Environment / Options
• Operational / Migration Considerations
• Results of Migration
• Lessons Learned
• References
• Q & A

5. Background Information
Global multi-service DoD Web-based application
Over 300 schemas
750 Gb of data
4,300 Active Users
4.2 million annual log ins
4,500 Reports Generated Per Day
AEM Corporation responsible for Hosting / Operations & Maintenance / Technology Refresh

6. Legacy Configuration
Network Topology Diagram Production
Production Environment
Physical Application Servers
Hosting Facility JBoss JVM IDM Server - BOXI R3
JBoss/CF Server FM Server – Tomcat/PPV
§ Physical Application Servers
Oracle Database Servers
Real Application Cluster
Gigabit Switch
§ 4 Gbps Fibre Channel
RAC
Disk
Storage/
RAC-1
Fiber Channel
§ Clarion Disk Array (10k / 15k RPM)
Tape Vault RAC-2 Switches
§ 5 Node RAC Servers
PowerVault
720N
RAC-3
§ Oracle 10gR2 RAC install
RAC-4
RAC-5
Disk Array for:
Transaction Database
Archive data
Binary Media Files
Fiber Cable
Ethernet Cable

7. Pre-Tech Refresh Issues
Legacy hardware over six years old
• Patches (5 nodes, slower machines)
• Deployments, data updates time consuming
• Large or complex reports often hang
• Node evictions
• Oracle 10.2.0.4 Support ended 6/31/11
Data Warehouse delay due to performance requirements
(Oracle Streams attempt)

8. Alternative 1 : Based on Legacy Solution
Proposed Network Topology Diagram Production
Production Environment
Virtual Application Servers
Hosting Facility JBoss JVM IDM Server - BOXI R3
JBoss/CF Server FM Server – Tomcat/PPV
§ Virtualized application servers
Oracle Database Servers
Real Application Cluster
Gigabit Switch § Network bonding
RAC
Disk
Storage/
RAC-1
Fiber Channel
§ 8 Gbps Fibre Channel
Tape Vault RAC-2 Switches
§ EMC Disk Array
PowerVault
720N
RAC-3
RAC-4
RAC-5
§ 5 Node RAC Servers
§ Oracle 11gR2 RAC install
Extended Disk Array
Disk Array w/ partitions for:
PHASE 1 Transaction Database
Fiber Cable
Data Mart (Star Schema)
Ethernet Cable PHASE 2 Archive data
Binary Media Files (Share
Point storage)
*VM Templates

9. Alternative 2 : Based on Exadata Solution
Vmware App
§ Virtualized application servers
Servers
DNS § Network bonding
MGT Server
§ 40 Gbps backbone
CommVault
ORACLE
§ Oracle Storage Servers
READY/ CLEAN MEDIA ERROR
ACTIVITY DRIVE ATTENTION
PowerVault
122T
EXADATA
CANCEL PREVIOUS NEXT ENTER
LTO Autoloader
Tape Library
§ 2 node Quarter Rack
EMC CX4-120
§ Oracle 11gR2 RAC preconfigured
EMC Fibre switch
EMC Fibre switch
100-240V~,1.0A,50/60Hz 1 3 5 7 9 11 13 15
2 4 6 8 10 12 14 16
LAN
T 2A, 250 VAC CAUTION: For continued protection against risk of fire, replace only with same type and rating of fuse.
KVM
PDU1
PDU2
Surprise, we chose Exadata!

10. Exadata X2-2 Selection Points
• Licensing fees made Exadata the low cost solution
• Total database hardware solution
o 2 nodes vs. 5 nodes
o Number of Vendors
• Patch Strategy
• Sizable and expandable
• Oracle vested to help DoD succeed
• Storage Indexes / Smart Scan / Smart Flash Cache

11. Throughput Gb/Second
80.0
75
70.0
60.0
2 Gbps Fibre Channel x2
50.0
4 Gbps Fibre Channel x2
40.0
8 Gbps Fibre Channel x2
37
30.0
Exadata 1/4 - Disk
25
20.0 Exadata 1/2 - Disk
10.0 12.5 16 Exadata Full - Disk
0.4 0.8 5.4
1.6 Exadata 1/4 - Disk & Flash
0.0
Exadata 1/2 - Disk & Flash
Exadata Full - Disk & Flash

12. Exadata Planning/Considerations
• Pre-configuration customer worksheet (DBM Configuration spreadsheet)
• Initial Setup Activities (ACS Services)
• Network Configuration / Software Installation
• Procurement considerations (2 Quarter Racks)
• Space Choices
• High Capacity (3 Tb / 7.2 K RPM)
• High Performance ( 600 Gb / 15k RPM)
• Space Allocation (ASM Disk Group layout)
• 78% DATA, 20% RECO, 2% DBFS_DG
o 25 Tb <- Production (Triple Mirror)
o 8 Tb <- Test (Mirror)

13. Tech Refresh Challenges
• 100% hardware replacement and Data Center move
• Narrow window for outage to avoid negative impact on end users
• Performance of system, database growth, and network bandwidth
• Exadata unproven in DoD space at the time (Security)
• Upgrading Database versions (data/code/reports)
Lots of change, what if issues surface???

14. Smart Flash Cache Considerations
Helps with…
• Write-Through cache persistent
• Holds hot data, much faster than disk (small, random I/O)
• Data not duplicated from cache in other Storage Servers
• Reduce latency of log write by simultaneous write to flash / disk
(faster writes) with minimal space (512 Mb) 11.2.2.4.0
• Write-Back cache 11.2.3.2.0/11.2.0.3.9
Don’t touch except for…
• Alter table <table_name> flash_cache keep;
• Create Flash Disks out of the Flash Cache
• Reassign portion for TEMP tablespace on index builds

15. Database Node Considerations
• Database Consolidation
• SGA Settings
• AMM Bad! ASMM Good! (set minimum values)
• Huge Pages (use_large_pages=only)
• Where’s the shared storage space?
• DBFS is the answer (fix_control=8,ac_timeout=60 and SGA=2Gb)
• Is everything setup correctly?
• Exachk is the answer
• Indexes / Hints / Compression
• Large Segments <- 8 Mb Initial / Next Size with Autoallocate
• TEMP <- BIGFILE, Autoextend 1 Gb, Uniform 1 Mb

16. Exadata Patch Management
Multiple Patches
• Quarterly Full Stack Download (Jan 2013)
• Infiniband (once per year)
• DB Nodes / Storage Server (quarterly)
• Bundle Patch (BP) DB Software (quarterly)
• Additional components (Ethernet switch, KVM, PDU)
Bug Fixes included so important to apply
Proceed with caution: one-off patches
Rolling option time a consideration

17. Security
• DoD 8570 Requirements
• Security Technical Implementation Guide (STIG)
o Oracle installation not customizable
o DBFS and idle_time don’t play well together
o Automatic Service Request (ASR) / Configuration Manager Limitation
o Grid Control / Third Party Certificates (September 2012 release)
o Banners / SQLNET.ORA settings impact on tools
Don’t assume security settings will not have impact. Must TEST!!!

18. Migration Strategies
10.2.0.4 to 11.2.0.x Options Considered
• DBFS with external tables (5 to 7 GB/sec file system I/O throughput)
• GoldenGate with datapump (near-zero downtime)
• Datapump
Factors
• Maintenance window
• Risk of data loss
• Familiarity with technology
Whatever the choice, perform multiple trial runs for optimal settings.

19. 2011 – Technical Refresh (Data Center move)
On 9/9/11 at 7pm application servers
at legacy site were Turned Off: Vmware App
Servers
• Transferred all data to Data Center DNS
• Network outage occurred during data MGT Server
transfer (2 hours) CommVault
ORACLE
• On 9/10/11at 7am New System Testing
READY/ CLEAN MEDIA ERROR
ACTIVITY DRIVE ATTENTION
PowerVault
122T
EXADATA
CANCEL PREVIOUS NEXT ENTER
LTO Autoloader
Tape Library
was Initiated
• Users were on the system by 3pm EMC CX4-120
EMC Fibre switch
EMC Fibre switch
Upgraded to 11.2.0.3 on 7/21/12
100-240V~,1.0A,50/60Hz 1 3 5 7 9 11 13 15
2 4 6 8 10 12 14 16
LAN
T 2A, 250 VAC CAUTION: For continued protection against risk of fire, replace only with same type and rating of fuse.
KVM
Grid Control 12.1.0.2 PDU1
PDU2
In Future-> X3 or Storage Expansion

20. Migration Timeline
Text
Ini@al
1/4
Rack
Grid
DB
Cutover
Delivered
Migra@on
Oracle
Setup
/
STIG
Prod/
Migrate Control
Setup
to
(Test)
Op@ons
Setup
DBFS
Test
CAB
Load
Apply
BP
Exadata
1/11
2/11
3/11
4/11
5/11
6/11
7/11
8/11
9/11
Chris
Ini@al
DB
STIG
Apply
BP
Migra@on
1/4
Rack
Migra@on
Grid
First
Setup
/
Test
Test
Selec@on
Delivered
Test
Control
Day
Load
(Prod)
Setup

21. Post Tech Refresh Performance (in hours)
Event Legacy Exadata
Process 1 4 hours 1 hour
Process 2 4 minutes 23 seconds
Datamart 3 hours 35 minutes
MV refresh 37 minutes 13 minutes

22. Exadata Lessons Learned
• Ensure hosting center can accommodate Exadata’s dimensions
• Staff requirements (more communication necessary)
• Testing required, ideally 2 Exadata Database machines
• Smart Scan <- direct path reads, full table scan, fast full index scans, parallel
with parallel_degree_policy not auto
• Chained rows void smart scans
• EHCC 10x space and performance (DML)
• Data Warehouse (EHCC)
• Reboot after after Write-Back Cache
In-Memory
X2-8 Massive Memory X3
2010 All I/Os to Memory 2012

23. Exadata Lessons Learned (cont.)
• Grid Control for monitoring / managing components
• Expect CPU utilization to decrease
• Expect Disk failures
• Can’t mix drive types and pricy to switch
• Standard tuning principles apply (OLTP)
• DB link opportunities for tuning
• Platinum Support, major assistance
• Exachk and opatch before / after patching
• Time, Experience keys to stability

24. References
http://www.oracle.com/technetwork/database/options/olap/olap-exadata-x2-2-
performance-1429042.pdf
http://www.scribd.com/doc/57026846/Exadata-Smart-Flash-Cache-366203
http://www.oracle.com/technetwork/database/exadata/exadata-technical-
whitepaper-134575.pdf
http://www.oracle.com/technetwork/server-storage/engineered-systems/exadata/
exadata-storage-technical-overview-128045.pdf
http://www.oracle.com/technetwork/database/features/availability/exadata-
consolidation-522500.pdf
http://www.ioug.org/Portals/0/Documents/Anuj%20Mohan.pdf
http://nyoug.org/Presentations/2011/March/Niemiec_Keynote_Exadata.pdf
http://www.oracle.com/technetwork/articles/oem/exadata-commands-part1-402441.html
http://www.centroid.com/webinars/ExadataChangestheGame.pdf
http://www.centroid.com/knowledgebase/blog/using-dbfs-and-external-tables

25. References (cont.)
Database Machine and Exadata Storage Server (888828.1)
Oracle Exadata Database Machine exachk (1070954.1)
Oracle Exadata Best Practices (757552.1)
Best Practices for OLTP on the Sun Oracle Database Machine (1269706.1)
Best Practices for Data Warehousing on Database Machine (1297112.1)
Oracle Sun Database Machine Application Best Practices for Data Warehousing
(1094934.1)
Oracle Sun Database Machine Diagnosability and Troubleshooting Best Practices
(1274324.1)
HugePages on Linux: What It Is… and What It Is Not… (361323.1)
Expert Oracle Exadata (Osborne, Johnson, Poder)

26. Questions???
Thank you!!!
http://vizuri.com/insights/blog
cbradham@vizuri.com

27. Backup Slides

28. Summary of Choices (X2)
X2-8 Full X2-2 Full X2-2 Half X2-2 Quarter
Initial / Licensing Cost $$$ $$$ $$ $
Database Servers 2 8 4 2
Database Grid Cores 160 96 48 24
Database Grid Memory (GB) 4096 768 (max 1152) 384 (max 576) 192 (max 288)
InfiniBand switches 3 3 3 2
Ethernet switch 1 1 1 1
KVM No Yes Yes Yes
Exadata Storage Servers 14 14 7 3
Storage Grid CPU Cores 168 168 84 36
Raw Flash Capacity 5.3 TB 5.3 TB 2.6 TB 1.1 TB
Raw Storage High Perf 100 TB 100 TB 50 TB 21.6 TB
Capacity High Cap 504 TB 504 TB 252 TB 108 TB
Usable mirrored High Perf 45 TB 45 TB 22.5 TB 9.5 TB
capacity High Cap 224 TB 224 TB 112 TB 48 TB
Usable Triple High Perf 30 TB 30 TB 15 TB 6.5 TB
mirrored capacity High Cap 150 TB 150 TB 75 TB 32 TB

29. Summary of Choices (X3)
X3-8 Full X3-2 Full X3-2 Half X3-2 Quarter X3-2 Eighth
Initial / Licensing Cost $$$ $$$ $$ $ $
Database Servers 2 8 4 2 2
Database Grid Cores 160 128 64 32 16
1024 (max
Database Grid Memory (GB) 4096 512 (max 1024) 256 (max 512) 256 (max 512)
2048)
InfiniBand switches 3 3 3 2 2
Ethernet switch 1 1 1 1 1
KVM Yes Yes Yes Yes Yes
Exadata Storage Servers 14 14 7 3 3
Storage Grid CPU Cores 168 168 84 36 36
Raw Flash Capacity 22.4 TB 22.4 TB 11.2 TB 4.8 TB 2.4TB
Raw Storage High Perf 100 TB 100 TB 50 TB 21.6 TB 10.8TB
Capacity High Cap 504 TB 504 TB 252 TB 108 TB 54 TB
Usable mirrored High Perf 45 TB 45 TB 22.5 TB 9.5 TB 4.5 TB
capacity High Cap 224 TB 224 TB 112 TB 48 TB 23 TB

30. X2-2 Quarter Rack Specifications
• 2 Xeon-based Dual-processor Database Servers (Sun Fire
X4170 M2)
• 24 cores (12 per server)
• 192 GB memory expandable to 288 GB (96 GB per
server expandable to 144 GB)
• 10 GigE connectivity to Data Center
• 4 x 10GbE ports (2 per server)
• 1.1 TB High Speed Flash
• 3 Exadata Storage Servers X2-2
• All with High Performance 600GB disks
OR
• All with High Capacity 3 TB disks
• 2 Sun Datacenter InfiniBand Switch 36
• 36-port Managed QDR (40Gb/s) switch
• 1 “Admin” Cisco Ethernet switch
• Keyboard, Video, Mouse (KVM) hardware Can Upgrade to a Half Rack
• Redundant Power Distributions Units (PDUs) or just add storage

31. Exadata Rack Options
2 six Core Processors / 96 Gb RAM per DB node
2 six Core Processors / 24 Gb RAM per Storage Server
Dual ported 40 Gb/sec InfiniBand
Quarter Rack
• 2 DB nodes
• 2 Infiniband switches
Half Rack
• 4 DB nodes
• 3 Infiniband switches
Full Rack
• 8 DB nodes
• 3 Infiniband switches
Infiniband 10x faster than Fiber Channel