NGENSTOR_ODA_P2V_V5

© Copyright 2015 NGENSTOR Alliance
Oracle Database Appliance
Bare Metal to Virtualized Platform Migration
Efstathios Efstathiou
Chairman NGENSTOR Alliance

© Copyright 2015 NGENSTOR Alliance2
Agenda
Introduction
About NGENSTOR Alliance
Initial Situation
Desired Target
Challenges/Constraints
Solution Approach
The odaback-API and it’s benefits
Final Migration Workflow
Benchmarks
Conclusion
Q&A

Introduction

General
• Software Defined Modular Storage
• Made in Switzerland
• Open Source
• Fully customizable (Firmware, ASIC, Parts)
• 2 Base product lines:
• Enigma Series all DRAM scale out tier 1 storage
• Financial Sector
• Military
• HPC
• Ultra High Performance DAS for DB (Exadata Alternative)
• Hurricane Series All Flash Systems
• Oracle Engineered Systems
• General Purpose

Enigma Series
• Hardware
• AMD co-developed MkII Chip rated @ 6.3 Ghz
• Scale out to 32768 nodes and 256 TB memory using vSMP
Advanced Edition
• NVDIMM memory
• PCIe SSD Tier 2 storage (presistency layer 2)
• Software
• EnigmaOS Version 14.3
• Hardware Compression using AMD GPU cores
• TTE Engine host plugin to optimize for databases
• Specs:
• 19’200’000 IOPS @ 4k in PCIe over Cable DAS Setup (8-node)
• 105.6 GB/s throughput in PCIe over Cable DAS Setup (8-node)
• More, if you need it ;-)

Hurricane Series
• Hardware
• AMD co-developed MkII Chip rated @ 4.8 Ghz
• Scale up to 4 nodes
• PCIe SSD Tier 1 storage
• SAS 12GBit/s Tier 2 storage
• Software
• EnigmaOS Version 14.3
• Hardware Compression using AMD GPU cores
• TTE Engine host plugin to optimize for databases
• Specs:
• 2’400’000 IOPS @ 4k in RDMA DAS Setup (4-node)
• 25.6 GB/s throughput in RDMA DAS Setup (4-node)

Initial Situation

Initial Situation
After 2-3 years of operation …
• The customer’s system had been «customized» to fit his own standards:
• SSH-Key management was installed … 
• Non-Standard Packages … 
• Custom bash_profiles with «nice» login prompts … 
• Self written stuff in crontab interferring with oakcli … 
• OWB was installed on the ODA causing upgrade issues … 
• CPU power and memory resources were still ok 
• Disk capacity was exhausted 
• Some disks started to fail 
• I/O performance was becoming a bottleneck (randon read/write)
• No proper housekeeping was done 
«There was a need bring the system back to a proper state 0.»

Desired Target
«Insert our mummy a pacemaker …»

Desired Target
Requirements for future system
• Be as standard as possible
• But still as compliant as possible
• Integration into Oracle Cloud Control
• Corrective Actions e.g. to delete foreign archive logs on logical
standby
• 4x quarterly upgrades should be done easily
• No big surprises like in the past
• Virtualization capability
• Isolate OWB into VM
• Rapid Testing by building a VM on the ODA
• 12c Database support
• Future Proof
• Offloading old data to external storage
• Old Warehouse Partitions

Challenges / Constraints

Key Problems
• Transition from Bare Metal to ODA Virtualized Platform
• System Downtime
• Backup/Restore
• External Storage
• Houskeeping
• Partially restore old standards
• Database Reorgs
• Data Movement

System Downtime
• Transition from Bare Metal to ODA Virtualized Platform requires Re-
imaging
• Transition from Bare Metal to latest ODA Virtualized Platform is a
two step process
• Transition from Bare Metal to ODA Virtualized Platform
• Upgrade from 2.9 to 12.1.0.2
• Downtime on Node Level in-evitable 
• Storage will be «reformatted» 
• The nodes share storage 
• We need a full backup of the entire system for a rollback 
• Backups must be made on external storage
• No problem for Databases but what about the rest?
• If you upgrade one node’s software you have support problems when
using Oracle Data Guard

Backup/Restore
• How do we assure that all our settings are backed up consistently?
• What is the best backup method?
• Which one the simplest to handle?
• To which target do we backup?
• What’s the max speed?
• Which data can be replicated to another ODA-System e.g. using ACFS
Replication?

External Storage
• ODA V1 has only the following physical connectivity options
• 1 Gbit Ethernet (~120 MB/s max)
• 10 Gbit Ethernet (~1200 MB/s max)
• ODA supports the following protocols for external storage
• NFS
• iSCSI
• AOE
• Customer had NetApp NFS Filers 1Gbit and 10gbit connectivity, but:
• Performance was limited to 50 MB/s by QOS => unusuable
• Network Switches also had problems with QOS => scp to other
ODA system with compression did not work well 
«We would need a DAS to plug into the 10Gbe ports at best.»

Partially restore old standards
• Ideally we would need to have a mounted copy of the old filesystems
• Why not simply snaphot the logical volumes?
• Is there enough space on the internal disks for a lvm snapshot?
• How to back it up to external storage?
• How to make it accessible for selective restore?
Database Reorgs
• As part of the migration?
• Datapump Export/Import?
• CTAS?
• Post Migration / separate window?

Data Movement
• How do we move old data?
• Via Backup / Restore?
• RMAN Image Copy
• OS commands post migration?
• When do we move the data?
• During Migration
• Post Migration
• What capacity do we need on the external storage?

Solution Approach

Solution Approach
Key Decisions Made
• First convert from Bare Metal to Virtual Plaform, then upgrade
• Do migration in per ODA-System or ODA-System Group
• Use Backup/Restore method mostly
• Use NGENSTOR Hurricane as Direct Attached Storage
• Move Housekeeping Task «DB-Reorg» to a separate maintenance
window

Solution Approach
First convert from Bare Metal to Virtual Plaform, then upgrade
• Transition from Bare Metal to Virtual Platform is our main goal
• Using this method there are the fewest risk, as we basically just re-install
the current 2.9 image
• No changes on the Oracle Homes
• We can safely restore our RMAN backuped Databases without
any upgrade risks
• Safe checkpoint for a partial migration
• No firmware updates will be done as we are on the same base release
before and after
• Eases system rebuild

Solution Approach
Use Backup/Restore method mostly
• The most error prone method:
• Stop everything
• Backup everything consitent while the system is stopped
• LVM Snaphot for OS disk (block level)
• RMAN Imagecopy plus archivelogs
• ACFS volumes file level & block level backup
• Even if it takes a bit more time => backup everything just in case
• In some cases where downtime is a issue a standby database can be
created on a remote system, giving us the time we need to do proper
backups.

Solution Approach
NGENSTOR Hurricane as Direct Attached Storage
• The Hurricane is directly attached to the ODA’s 2x10Gbit NIC ports
• SPF+ copper cable
• No network switch required
• No latency added
• No QOS, since it is a private network
• 2.4 GB/s wire speed
• We have the full power of the SAN-Controller for use like on the
Exadata Storage Server 
• Hurricane Firmware is set to DAS-Mode
• Cache Partioning for better performance
• Less concurrency => more IOPS possible

Solution Approach
Move Housekeeping Task «DB-Reorg» to a separate maintenance
window
• Time Consuming Task outside of the main scope
• Specific to the database
• Maybe we even requires patches
• Risk of changing something, that used to work before causing a rollback
• Stats
• Execution Plans, etc.
• Way faster using NGENSTOR Hurricane as primary platform (all flash) ;-)

Why?
• We were are lazy
• Bored to do the same tests over and over again
• Less typing is always good
• Automation to avoid complex manual cutover plans
• Reduce Copy/Paste errors
• Have consistent results and errors
• Script as documentation

Odaback infos
• Open Source
• Bash and Python Version
• Shipped as RPM for your ODA-Version (2.8, 2.9, 2.10, etc.)
• Installs to /usr/local/bin/odaback resp. Odaback.py
• Synthax
• odaback {ACTION} {TARGET} {MODE}
• odaback --workflow --file wf_migration_2.9_p2v.oda
• odaback --getconfig cluster exec
• odaback --getconfig cluster script
• odaback --enableaoe host1
• odaback --stopcrs cluster
• odaback --rootbackup host1
• odaback --rootbackup cluster
• odaback --dbbackup cluster

Features
• Configuration Management (baseline rpm, system, settings)
• Root Disk Backup
• DB Backup Code Generator
• ODA Config Backup
• ACFS & external volume backup
• EMCLI Integration
• External Script Support

Configuration Management (baseline rpm, system, settings)
• Can be done:
• Using either EMCLI (config_compare)
• Built-in code
• Or both 
• Option do a LVM Snap and compare with the current system using diff
Sources
http://docs.oracle.com/cd/E24628_01/em.121/e17786/cli_verb_ref.htm#EMCLI397

Root Disk Backup
• Creates an LVM Snaphot of the root disk
• Dump the snapshot device into file on external storage
• Mounts it using losetup mount /mnt/odaback for restore if needed
Sources
ODA: Bare Metal Restore (BMR) Fails With Errors "LVM Operation Failed" and "pvcreate failed for" (Doc ID 1643411.1)
How To Backup And Restore Operating System On Compute Nodes Using LVM Snapshots In Exalogic Physical Environment (Doc ID
1485860.1)
http://www.thomas-krenn.com/de/wiki/LVM_Snapshots
http://www.tutonics.com/2012/12/lvm-guide-part-2-snapshots.html

DB Backup Code Generator
• Generates RMAN Scripts
• File Based
• Global Script in RMAN Catalog
• Generates DBCA Template
• Structure only
• With Data
• Generates Datapump plus DBCA Script
• DBCA Structure
• EXDP/IMP shell script for data load/unload
• Enables 2nd archive dest, if required (alternate)
• 1x internal SAS storage
• 1x external storage
Sources
http://docs.oracle.com/cd/B28359_01/backup.111/b28273/rcmsynta021.htm#RCMRF127
How to Automate Archive Log Overflow Using 'Alternate' (Doc ID 270069.1)

ACFS & external volume backup
• Can be done on block level or file level
• Basically just a file list when using file level
• File Level uses basic tools like:
• rsync
• tar
• No super magic stuff happening here, sorry :-P

EMCLI Integration
• odaback can act as code generator for EMCLI jobs
• odaback can use templates for workflows (multi-task)

External Script Support
• Pre and Post stage
• Workflows (multi-task => emcli => CC required)

The single steps described in words
• Preparation
• Backup onecommand.parms (your oda config) from your ODA
• Modify the config / use as baseline for Appliance Manager
• Create Image Copy Backups of your databases on NGENSTOR
• Create backups of your ACFS volumes or move data to NGENSTOR (NFS-Mode)
• Downtime
• Reset license core keys before re-imaging ODA
• Archive backup of all databases
• Resync ACFS Volumes
• Stop Cluster and Agents
• LVM Snapshot of internal disk
• Start-Reimaging
• Deploy ODA_BASE with configuration file
• Restore Databases and ACFS directories, if not using NGENSTOR as primary storage
• Post-Migration
• Mount LVM Snapshots
• Restore Settings as needed

The single steps described in commands
• Preparation
odaback --workflow --file wf_migration_2.9_p2v_prep.oda
• Downtime
• odaback --workflow --file wf_migration_2.9_p2v_backup.oda
• Re-Image System (Oracle Procedure)
• Install odaback
• odaback --workflow --file wf_migration_2.9_p2v_deploy_odabase.oda
• odaback --workflow --file wf_migration_2.9_p2v_restore_db_settings_volumes.oda
• Post-Migration
• odaback --workflow --file wf_migration_2.9_p2v_mount_lvm_backups.oda
• Restore Settings as needed

The support notes / links that go with it, part1
Sources
Certified Software on Oracle VM (Doc ID 464754.1)
How To Configure xbond,bond1,bond2 on ODA V1 post deploy (Doc ID 1422563.1)
How to setup Oracle Enterprise Manager Solution-in-a-Box on Oracle Database Appliance (Doc ID 1673246.1)
http://blog.oracle-ninja.com/2012/08/upgrading-the-oracle-database-appliance-t-version-2-3-0-0-0/
http://marcel.vandewaters.nl/oracle/database-appliance/oracle-database-appliance-appliance-manager
http://www.oracle.com/technetwork/database/database-appliance/documentation/oda-psft-solninabox-wp-2197457.pdf
http://www.oracle.com/technetwork/database/database-appliance/oracle-database-appliance-faq-1903200.pdf
http://www.oracle.com/technetwork/database/database-appliance/overview/asm-acfs-migration-2379319.pdf
http://www.oracle.com/technetwork/database/database-cloud/privatedbcloudonoda-1522348.pdf
https://blogs.oracle.com/ODA/entry/leverage_the_oracle_database_appliance
ODA (Oracle Database Appliance): acfs_mig.pl a Tool to Migrate Non-CDB Databases To ACFS (Doc ID 1951012.1)
ODA (Oracle Database Appliance): How To make onecommand.params compatible (Doc ID 1662038.1)
ODA (Oracle Database Appliance): How To Reset the license core keys before re-image to the Virtualized Platform option (Doc ID
1559091.1)
ODA 64-bit CPU Paravirtualized Support Guest Operating Systems for ODAVP using Oracle VM 3.2.3 (Doc ID 1927798.1)
ODA FAQ : Understanding the Oracle Database Appliance Core Key Generation usage, common questions and problems ( FAQ ) (Doc ID
1597084.1)

The support notes / links that go with it, part2
Sources
ODAVP: How To Create a Fully-Virtualized Guests (HVM) from an OS ISO image (Doc ID 1524138.1)
ODAVP: How to create a VM template from a Guest VM (Doc ID 1668933.1)
ODAVP: How to move a VM from local to a shared repository (Doc ID 1601860.1)
ODAVP: HowTo backup/restore your VM Guest (Doc ID 1633166.1)
Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)
Oracle Database Appliance - Steps to Generate a Key via MOS to change your CORE Count and apply this Core Key (Doc ID 1447093.1)
Oracle Database Appliance (ODA) End User Deployment (Doc ID 1373617.1)
Oracle Database Appliance Bare Metal Restore Procedure (Doc ID 1373599.1)
Oracle Database Appliance Upgrade Steps Finding Tool (Doc ID 1519650.1)
Step by Step Instructions on Installing Virtualized Image on Oracle Database Appliance (Doc ID 1520579.1)
Upgrade ODA Server where Physical Standby Database running (Doc ID 1528625.1)

Benchmarks

Benchmarks
I/O limits
• ODA V1 SAS disk have a throughput limit of around 1.5 GB/s
• This was measured using RMAN Backups with a 24 channels and 1 GB section size
• The Hurricane’s performance is limited by the 2x10Gbit Ethernet to 2.4 GB/s max connection
used (we reached close to wire speed)
• SSD’s clearly beat HDD’s for random workloads
• AOE layer2 protocol helped to further reduce latency
• An intelligent array helps to save cpu cycles on the database server compared to the SAS-
Expanders used in the ODA
• Waits went down dramatically
• It’s simple spoken the same effect when putting an SSD into 6-year old notebook
«Engineered Systems help you to deploy faster, but they also need regular maintenance to
perform. Still the quest for the holy grail goes on …»

Conclusion

Conclusion
Final Comments
• External Storage for your ODA makes sense
• Can help to reduce I/O problems
• Data can be archived to e.g. (DWH old partitions)
• New capabilities like snaphot or replication can be offloaded (reduces cpu
consumption on db server)
• Customer decided to use NGENSTOR Hurricane as his primary storage 
• The ODA’s internal SAS Storage was re-used as recovery area
• Backup Strategy was adapted to do incrementals to internal SAS Storage
• Tape out to VTL over 1GBit is no longer such a big issue
• Many customizations have been removed 
• Upgrade to 12.1.0.2.0 worked flawlessly 
• Reorg brought another boost in performance

Questions
and
Answers

Contact Information
• efstathios.efstathiou@ngenstor.com

Thank You!

NGENSTOR_ODA_P2V_V5

More Related Content

What's hot

Similar to NGENSTOR_ODA_P2V_V5

More from UniFabric

NGENSTOR_ODA_P2V_V5