Oda as an enterprise solution at walgreens oow 2012 v7

Oracle Database Appliance
as an Enterprise Solution at Walgreens
Oracle Open World 2012
Session CON3776

We will optimize our technology so you can maximize your business.
*Accountability*Communication*Quality*Teamwork* 1

Agenda

• Background – Review Walgreens’ Experience with the Oracle Database
Appliance
– ODA Business Case
– Why Walgreens picked the ODA for a critical business application

• Key understandings needed to manage ODA's as an enterprise solution
• Blueprint for deploying the ODA as an enterprise solution
– Pre-deployment Steps
– Post-deployment Steps

• ODA Best practices
• ODA Reference Guide & Resources
• Question & Answer
• Slide Notes

*Accountability*Communication*Quality*Teamwork*

Walgreens
• Nation's largest drugstore chain with fiscal 2011 sales of $72 billion.
• 7,929 drugstores in all 50 states, the District of Columbia and Puerto Rico
• Fortune magazine World's Most Admired Companies (published March 4, 2011)
– Walgreens ranked No. 4 among food and drugstores and has been listed for the past
18 consecutive years.
– Walgreens ranked 32nd overall in revenue, 3rd among food and drugstores, on the
Fortune 500 list

• Walgreens scope of pharmacy services includes retail, specialty, infusion,
medical facility and mail service, along with respiratory services.
– These services improve health outcomes and lower costs for payers including
employers, managed care organizations, health systems, pharmacy benefit managers
and the public sector.

• In June, 2012, Walgreens acquired a 45 percent stake in Alliance Boots, a
European Pharmacy retailer, with an option to acquire the remaining 55% stake
in 3 years.
– Combined: 11,000 stores in 12 countries


Who We Are
• Matt Gallagher
– Database Architect Lead
– 14 years at Walgreens
– 16 years of Oracle experience
– Linked In

• Fuad Arshad
– Senior Database Architect
– 15+ years of Oracle experience
– 11 years at Walgreens
– Twitter @fuadar G+ +Fuad Arshad


• What Is it to the World

Copyright 2010 Walgreen Co. *Accountability*Communication*Quality*Teamwork*

• What we feel it is .


ODA Business Case – Top Business Benefits

1. Engineered System - Hardware and software designed to fit together
2. Cost – Engineered system at commodity hardware prices
3. Deployment in Days, not Months
4. Expertise levels required to deploy HA solutions reduced
5. Standard known configuration
6. ODA’s come with an extensive administration and monitoring toolkit
7. Consolidation platform for instances requiring 12 cores or less
8. Patching & upgrade automation of all tiers as a Unit (vs. fall behind)


9. Administration benefits / savings
10. Stability & availability from standard tested configurations
11. ODA has dedicated development & support teams
12. The Next Versions


Why Walgreens picked the ODA for supporting
a critical Application
• Main Reason for choosing the ODA - Time to Deployment requirement of 2 weeks,
which included ordering and shipping the hardware
• We needed a High Availability architecture deployed over 2 data centers
• Typical Server setup steps & Teams not needed
– AIX (107); Solaris (103); Linux (103)
– ODA Setup steps – in the 20’s
– Teams not needed (storage, server, PM’s, less networking)

• Major Tasks Avoided
- Equipment specing
- Building private interconnect networks
- IO virtualization setup
- Server and SAN setup, OS installs
- Clusterware and DB installs


Why Walgreens picked the ODA for supporting
a critical Application
• IVR Metrics
– Mission critical application – Centralizing phone call routing for 8,000 stores, including
Rx refills
– DR Class A (RTO – near zero; RPO – near zero)
– Designed for 800 million phone calls per year – Peak (548K / hour; 9,100 / min.,
152/sec)
– Number of ODA’s deployed for IVR DB's - 14

• ODA Deployment Example
– From data center floor to production in 3 days (Monday - DC install, Tuesday – network
cabling, Wednesday – install 4 ODA’s in parallel)
– Thursday - Lunchtime rebuild of a production server as a confidence test


After the picked ODA’s – More reasons

• DBaaS (Database as a Platform) Strategic choice along with Exadata
– Pre-deploy DB infrastructure vs. wait for builds

• Exadata non-production environments
• The business, project managers and application teams are asking for ODA’s
– Frequently heard question – “Can we do this on an ODA”?

• Current number of appliances – 30 + and growing (Sept., 2012)
– Plus Exadata


ODA Time to Deploy Comparison (in days)


Key Understandings needed to manage ODA’s
as an Enterprise Solution


Key Understandings – ODA’s as an Enterprise Solution

1. Develop a formal solution architecture
2. Understand the ODA limitations
3. Security Lockdown
4. Set a goal
5. Technical project tracking
6. ODA monitoring
7. ODA Administration



8. Managing resources to control licensing & resources
9. Prepare to handle potential failures in advance
10. Break any bad habits
11. Understanding patching is a key challenge
12. An ODA is not a black box


Key Understandings – Evolution of the Oracle Appliance
Kit (OAK)

1. OAK 2.1.0.1-2.1.0.2
– Base Release
– Oakcli

2. OAK 2.1.0.31
– Performance enhancements – BIOS fix that improved CPU performance
– ASR integration
– Allowed oakcli deploy –conf to pass deployment files.

3. Oak 2.2 –
– Upgraded the Kernel to the UEK and fixed the Cluster Health Monitor and ASR functionality
– 11.2.0.3 support introduced
– Odachk for healthchecks
– Cluster Health Monitor(CHM)is fixed

4. OAK 2.3 - A major leap forward
– Multiple DB homes now supported (11.2.0.2.7 & 11.2.0.3.2 onwards)
– Expansion of the oakcli command set
– Odachk enhancements
– Database patching is rolling upgradable

Blueprint for deploying the ODA
as an Enterprise Solution


Pre-deployment steps

1. Develop the solution architecture
2. Data Center Install request
3. Assign rack location (4u) with Power
4. Security zoning / Firewall architecture
5. Create the Deployment template
– Cabling & IP specs

6. IP assignment request
7. Cabling request
8. Firewall request
9. DNS request
10. VPN request for remote access


Pre-deployment steps (Cont.)

11. Order the ODA
12. Rack the appliances
13. Cable the servers
14. Create the Install configurator files using the offline configurator tool
15. Decide What type of template you want to use for Database creation


The Deploy Poster


Deployment


Post-deployment steps

1. Instances are setup using templates – they may need some tweaking
2. Build additional instances (dbca or oakcli)
3. Change the default passwords (root, oracle)
4. Configure mail
5. Configure and test ASR (Automatic Service Requests) – Phone Home
6. Install YaST rpm’s (needed for server administration via OEM)
7. Install OEM (single monitoring stack) 12c agent and configure OEM
8. Setup backups
9. Modify adrci settings (base & homes)



10. Setup proc watcher (as a RAC resource but don't run it unless needed)
11. Label appliances and servers & record the data center rack locations
12. Inventory CSI’s / serial numbers
13. Register CSI’s in MOS and grant access to team members
14. Install 11.2 clients & jdbc versions required for scan listener support
15. Setup instance caging & other resource control mechanisms
16. File system backups vs. images
17. Save your images for faster restores


18. Complete the RAC setup
19. Security lockdown
20. Implement and test the remaining elements of the Solution Architecture
– This includes the data recovery and DR facilities

21. Utility deployment
22. Database migration
23. Post-upgrade


ODA Best Practices


ODA Best Practices

1. Submit Requests for packages / bug fixes to the ODA support team (SR)
2. Technical project management matters
3. Create a versions tracker
4. Retain system images (ISO’s)
5. Order a sandbox ODA if you can afford it
– Order spare disk drives if you can afford it

6. Develop a method for remote installs
8. Label the servers
9. Test everything (failover, recovery, …)
10. Define and implement non-technical operations best practices


ODA Best Practices (cont.)
11. Take an architecture driven vs. an incident driven approach to HA
12. Develop and implement a formal Data Recovery plan
13. Cache as much data in memory as possible (to drive Logical IO from Physical)
14. Specialize skill sets, at least to an extent
– Similar to the Database Machine Administrator concept

15. Think twice (or more) before making customizations
– Including one off patching

16. Consider using a new image vs. the image the ODA comes with if the version is
two or more generations back
17. Manage resources on the appliance
18. Training and process documentation
19. Pay attention to capacity monitoring and planning


ODA Reference Guide


MOS Documents

Category MOS Doc Document
Deploy 1373617.1 ODA end user deployment including
configurators
1409835.1 ODA deployment including restarting steps &
cleanup (start over) process
1448278.1 ODA deploy step-by-step deploy guide
(Version 2.1)
1469093.1 ODA deploy step-by-step deploy guide
(Version 2.2)
Post-Deployment 1353507.1 Enable outbound sendmail
1415573.1 ODA support for new features (multiple
Homes & 3rd party agent support)
1422563.1 Configure additional networks post
deployment
1435019.1 ODA: How to setup ACFS post deployment


MOS Documents
Category MOS Doc Document
Post- 1436335.1 ODA post deployment configurations index
Deployment including additional networks & ACFS
1461798.1 Download Linux rpms directly to an ODA
1461818.1 Using a separate Linux machine as an RPM
repository
Start 888888.1 2.x supported versions and Known issues
1392174.1 Engineered Systems welcome center including
opening SR's for engineered systems
1417713.2 ODA Information Center
1463638.1 ODA FAQ
Recovery 1373599.1 ODA bare metal restore procedure
Security 1450387.1 Responses to common ODA security scan
findings (not released yet)
1461102.1 & Security Technical Implementation Guide
1456609.1 (STIG) script

Resources
• MOS Community (Engineered Systems)
• Blogs
– https://blogs.oracle.com/eSTEP/

• http://www.oracle.com/technetwork/server-storage/engineered-
systems/database-appliance/overview/index.html
• Tips for Hardening an Oracle Linux Server
– http://www.oracle.com/technetwork/articles/servers-storage-admin/tips-harden-oracle-
linux-1695888.html
– http://www.oracle.com/technetwork/articles/servers-storage-admin/secure-linux-env-
1841089.html

• UEK benefits podcast
– https://blogs.oracle.com/OTNGarage/

• Dan Morgan’s Library
– http://www.morganslibrary.org/reference/oda.html

• Arup Nanda Linux & Exadata series
– http://bit.ly/k4mKQS & http://bit.ly/lljFl0

Resources
• Oracle Linux articles
– http://www.oracle.com/technetwork/articles/linux/index.htm

• Oracle Technology Network (backup, migration, Data Guard, expanding storage,
setup resources, demos, documentation …)
– Documentation - Getting Started Guide, Owners manual, Service manual, ASR
manual, Setup Poster
– http://www.oracle.com/technetwork/server-storage/engineered-systems/database-
appliance/overview/index.html

• www.youtube.com
– Too many ODA videos to list

• Database as a Service (DBaaS)
– http://www.oracle.com/technetwork/topics/entarch/oes-refarch-dbaas-508111.pdf

• Coming soon
– Official release of the one button Data Guard deployment script
– More “One Button” deployments to follow


Resources
• Misc. Resources
– http://blog.oracle-ninja.com/2011/09/inside-the-oracle-database-appliance-part-1/
– http://blog.oracle-ninja.com/2011/12/inside-the-oracle-database-appliance-
%E2%80%93-part-2/
– https://blogs.oracle.com/eSTEP/entry/oda_announcing_oda_external_storage
– http://docs.oracle.com/cd/E22693_01/index.html
– https://blogs.oracle.com/ODA/
– http://www.oracle.com/us/products/database/database-appliance/overview/index.html
– www.oracle.com/databaseappliance
– http://www.pythian.com/news/26701/oracle-database-appliance-faq/
– http://www.pythian.com/news/34715/migrating-your-10g-database-to-oda-with-minimal-
downtime/
– http://www.pythian.com/news/33245/insiders-guide-to-oda-performance/
– http://www.slideshare.net/gwenshap/shapira-oda-perfwebinarv2
– http://www.pythian.com/news/27201/oracle-database-appliance-storage-performance-
part-1/
– https://blogs.oracle.com/dragonfly/entry/making_easy_easier (RAC in 2 hours)
– https://blogs.oracle.com/dragonfly/ (Overall ODA ease of deployment


Resources
• Misc. Resources (Cont.)
– http://www.oracle.com/us/products/database/oracle-database-appliance-faq-
495423.pdf
– http://www.oracle.com/us/technologies/linux/uek-r2-features-and-benefits-1555063.pdf
– http://www.oracle.com/partners/en/knowledge-zone/server-storage/database-
appliance-494792.html
– http://www.oracle.com/partners/en/most-popular-resources/na-databaseappliance-
fridaysessions-1522340.html


Additional Thanks to our ODA Strategic
Customer Program support team
• Ian Cookson – ODA Technical Manager – Keeping the SCP on track
• Duane Smith – ODA Technical Guru
• Raji Sabbagh – Our enthusiastic “get started” installation master
• Charlotte Momich – ODA marketing
• Jay Maddox – Sales & “Make it happen”
• Ravi Sharma - Security


Questions?


Slide Notes


1. Engineered System - Hardware and software designed to fit together
– Resources that fit within 4u (2 servers, 12 cores per server, 24 core total, 96G memory
per server)
– Deploys the standard Oracle setup utilities (dbca, asmca)
– Automated deployment & patching all components
– While you can build a commodity hardware solution, you can’t build your own
engineered system
– Balanced components to eliminate bottlenecks
– LIO performance is equivalent to Exadata LIO performance
(http://fritshoogland.wordpress.com benchmarks)
– Single supplier / escalation point
– Singles point of management (oakcli, ILOM and OEM)

2. Cost – Engineered system at commodity hardware prices
– Engineered Systems at commodity hardware costs with embedded management
facilities

3. Deployment in Days, not Months
– Eliminates unpredictable, time wasting setup issues
– Meet the needs of the business #1 benefit
– Drives project flexibility for inevitable changes



4. Expertise levels required to deploy HA solutions reduced
– Interconnect (dedicated switches, MTU, jumbo frame, portfast spanning)
– RAC assessment
– RAC check findings
– RAC starter kit requirements
– Server, storage and network setup
– Starter Kits, FAQ’s, manuals, assessments, ……

5. Standard known configuration
– Foundation for your own standard setup
– Platform for Oracle automation (ex: One button Data Guard)
– Instant RAC clusters or RAC One deployments
– MAA facilities including 1 button Data Guard with many MAA resources

6. ODA’s come with an extensive administration and monitoring toolkit
– OS Watcher, Cluster Health Monitor, oakcli (Oracle Appliance Manager), ilom, asr (with
automatically updated rules), odachk (version of RAC check), cluster health monitor,
logwatch (log alerting script), secure backup, vncserver



7. Consolidation platform for instances requiring 12 cores or less
– Largest instance of 12 cores or less
– RAC One enables HA

8. Patching & upgrade automation of all tiers as a Unit (vs. fall behind)
– Patch components are tested together
– Server, OS, ilom and BIOS
– Clusterware
– DB

9. Administration benefits / savings
– San, server, network (RAC interconnect), database setup time and on-going support
time and cost savings


10. Stability & availability from standard tested configurations
– ODA’s include the Oracle unbreakable Linux kernel (the resources section includes a
video link on the benefits of the UEK)
– Disk is triple mirrored, highly resilient (data disk failures don’t stop the system)

11. ODA has dedicated development & support teams
12. The Next Versions
– Bigger and faster is expected
– App server support in 2013
– ODA’s will keep their place within the Engineered Systems lineup



1. Develop a formal solution architecture
– Formalize requirements:
– RTO (recovery time objective), RPO (recovery point objective)
– HA requirements based on business impact
– Backup / Recovery plan
– Data recovery plan
– Application specific requirements
– Capacity estimates (CPU, memory, storage, IOPS) and growth
– Instance architecture (single instance, RAC One, RAC)

2. Understand the ODA limitations
– IOPS: - 4000=200/disk at 5ms.; 6000=300/disk at 10ms.; 8000=400/disk at 20ms.
– Storage volumes (3.2T usable)
– N, N+1 capacity
– Server Patches are not rolling (Grid & Database patches are)
– Account uid / guid can't be changed at this time
– Resource levels - Largest instance is 12 cores
– Number of network interfaces available (Only 1 – 10g network, but should be enough
for most deployments)


3. Security Lockdown
– Use your standard company server security procedures
– Security Scans (Qualys)
– Many of the security enhancements are implemented in the STIG (DOD – Security
Technical Implementation Guide) script. Keep it updated, but remember it is still a work
in process.
– Keep un-necessary users off the system

4. Set a goal
– Our goal was “make the ODA’s the best system platform within Walgreens”
– Standard Configuration that is consistently and completely implemented - Automate as
necessary
– Continual improvement is like compound interest (Einstein: “The most powerful force in
the Universe is Compound Interest”)

5. Technical project tracking
– Issue tracking
– Setup Process tracking (Many systems failures occur because what is known to be
needed to done is lost sight of)
– Deployment tracking



6. ODA monitoring
– ASR phone home setup
– ODA administration integration with OEM 12c
– Ops Center support for ODA's
– ODA native monitoring & Diagnostics (allows DBAs to monitor the complete stack)

7. ODA Administration
– Integrated Lights Out Management (ILOM) – new skill to become familiar with
– Learn Oakcli – The Appliance Manager – This is one of the key ‘secret sauces’ of the
ODA

8. Managing resources to control licensing & resources
– Avoid underutilized systems
– Instance caging
– Capacity planning

9. Prepare to handle potential failures in advance
– Go through the key failure scenarios in advance



10. Break any bad habits
– Patch! – It’s an appliance, not a server
– Break Bad habits part 2 - Don't break the setup (ex: RAC best practices - don't set
TNS_ADMIN variable)

11. Understanding patching is a key challenge
– Request new patches -> SR -> ODA support group -> ODA development group
– Always refer to the latest reference architecture information & read the documentation,
especially the Patch documents

12. An ODA is not a black box. You need to understand the inner workings of:
– Patching, ILOM, ASR
– Basic server administration
– Server security
– How RAC works


Pre-deployment steps

1. Develop the solution architecture
2. Data Center Install request
3. Assign rack location (4u) with Power
4. Security zoning / Firewall architecture
5. Create the Deployment template
– Cabling & IP specs - 1g vs. 10g (10g requires SFA adapters)
– 2 basic Models (10g public / 1G Backup; 1 g public / 10g backup) – Not Both

6. IP assignment request
7. Cabling request
8. Firewall request
9. DNS request
10. VPN request for remote access


Pre-deployment steps (Cont.)

11. Order the ODA
– ODA quote

12. Rack the appliances
13. Cable the servers
14. Create the Install configurator files using the offline configurator tool
– MOS 1373617.1

15. Install the ODA (Setup poster tasks)
– A conserve (remote KVM connection) appliance can eliminate the need to go on-site
for the installs
– Data Center connection terminal (“KVM on a cart”)
– Connect directly from your laptop (USB to RS232 to Serial Port connectors)
– Connect to the ILOM on each node and configure the ILOM IP’s
– Connect to the ILOM through your browser, run the remote terminal and run the first
net script to get the ODA on the network
– Display the version to determine if patching will be needed
– Run the deployment scripts



1. Instances are setup using templates – they may need some tweaking
– Init.ora settings
– Only 1 control file is created
– Redo logs are not mirrored

2. Build additional instances (dbca or oakcli)
3. Change the default passwords (root, oracle)
– Both ODA servers and both ILOM’s
– Create additional Unix accounts as needed

4. Configure mail
– Install “sendmail –cf” rpm

5. Configure and test ASR (Automatic Service Requests) – Phone Home
6. Install YaST rpm’s (needed for server administration via OEM)
– Being added by Oracle into the base package. Beware that they need to be uninstalled
b/4 patching until then.



7. Install OEM (single monitoring stack) 12c agent and configure OEM
8. Setup backups
– Our solution: Data Guard master server for backups & backup appliances
– ASM disk header backups may be needed
– Set the Rman configuration settings

9. Modify adrci settings (base & homes)
10. Setup proc watcher (as a RAC resource but don't run it unless needed)
11. Label appliances and servers & record the data center rack locations



13. Register CSI’s in MOS and grant access to team members
– CSI administration

14. Install 11.2 clients & jdbc versions required for scan listener support
15. Setup instance caging & other resource control mechanisms
16. File system backups vs. images
– Bare Metal Restore will wipe out any additional software or directories, including OCR
backups. Tnsnames files, third party agents or software, OEM agents, controlfile trace
backups, exports, additional installed packages

17. Save your images for faster restores
18. Complete the RAC setup
– RAC services
– RAC tuning – right handed indexes, etc.
– Failover testing
– Run the ODA check (odachk)
– Cluster health monitor checks


19. Security lockdown
– Follow Oracle’s best practices:
– Tips for Hardening an Oracle Linux Server
– http://www.oracle.com/technetwork/articles/servers-storage-admin/tips-harden-oracle-
linux-1695888.html
– http://www.oracle.com/technetwork/articles/servers-storage-admin/secure-linux-env-
1841089.html
– CISP requirements
– PCI requirements
– Dept. of Defense security reviews
– Script to make "su to oracle and root only" - STIG script
– STIG script ML 1461102.1 - prevent direct access to oracle & root
– Disable for patching which requires direct access to root
– Sudo for access to specific functions only (Oracle publishes an Exadata paper on the
subject)
– Keep people off the server if they don’t have solid requirements for access
– Run a security scan (Qualys)
– Enable auditing


19. Security lockdown (cont.)
– User management (ex: deploy standard password files) or OEM 12c
– Find open ports
– Firewall design - security zoning
– July 2012 2.3 release had additional lockdowns
– Oracle’s approach to locking down an ODA
– Looked at common vulnerabilities and classified them
– Only installed 700 of the possible 3000 Linux rpm’s
– Used government standards
– Oracle is now looking at additional financial services lockdowns (They’re at the 1st
pass of this now)
– Oracle ran their own security Qualys scans as part of their lockdown
– ODA development team is working closely with the Oracle Linux team to build
security into the ODA
– Some changes are being installed into the ODA images. A script will handle the
rest (STIG script has a version number)


20. Implement and test the remaining elements of the Solution Architecture
– This includes the data recovery and DR facilities

21. Utility deployment
– Cleanup logfiles
– Mine the system logs
– Solution for backing up files that would be erased during a bare metal restore

22. Database migration
– Oracle has published an ODA migration whitepaper

23. Post-upgrade
– Relink the Oracle OEM agent


ODA Best Practices

1. Submit Requests for packages / bug fixes to the ODA support team (SR)
2. Technical project management matters
– Create an issues tracker
– Create a deployment checklist covering all servers and deployment steps
– Create standard processes and documentation (deployment)
– Automate where it makes sense

3. Create a versions tracker
– Once you start deploying more than a small number of ODA’s

4. Retain system images (ISO’s)
5. Order a sandbox ODA if you can afford it
– Order spare disk drives if you can afford it

6. Develop a method for remote installs
– ODA’s come with VNC server installed
– Conserves (KVM consoles) are a good option


ODA Best Practices

8. Label the servers
9. Test everything (failover, recovery, …)
10. Define and implement non-technical operations best practices
11. Take an architecture driven vs. an incident driven approach to HA
– Be prepared for what could go wrong vs. reacting to what did go wrong

12. Develop and implement a formal Data Recovery plan
13. Cache as much data in memory as possible (to drive Logical IO from Physical)
– Also remember the result cache


ODA Best Practices (cont.)
14. Specialize skill sets, at least to an extent
– Similar to the Database Machine Administrator concept

15. Think twice (or more) before making customizations
– Including one off patching
– You have to replicate customizations
– Customizations may cause patching issues or may break current or future automation

16. Consider using a new image vs. the image the ODA comes with if the version is
two or more generations back
– But be careful. There are issues with BMR’s, for example not patching the ILOM’s.
This can be resolved by applying the latest patch with the “infrastructure” option.

17. Manage resources on the appliance
– Memory (96G per server, or 8G per cores)
– Implement instance caging if needed

18. Training and process documentation
19. Pay attention to capacity monitoring and planning


Oda as an enterprise solution at walgreens oow 2012 v7

More Related Content

What's hot

Viewers also liked

Similar to Oda as an enterprise solution at walgreens oow 2012 v7

Oda as an enterprise solution at walgreens oow 2012 v7