This document discusses challenges with online patching in Oracle E-Business Suite release 12.2.5. It begins with an overview of the 12.2 architecture and how it enables features like file system editioning and database edition-based redefinition to allow patching while the application is online. It then covers the online patching cycle in detail and discusses options for developing custom code to be either fully or runtime compliant. The document concludes with lessons learned around areas like database object grants, the DB_Domain parameter, executing autoconfig, and administering application nodes. It also discusses some common challenges seen with online patching and useful utilities for monitoring and diagnosing issues.
Database failover from client perspectivePriit Piipuu
In this presentation we will look deep into high availability technologies Oracle RAC provides for database clients, what actually happens during database instance failover or planned maintenance and how to configure database services so that Java applications experience no or minimal disruption during planned maintenance or unplanned downtime. This presentation will mainly focus on JDBC and UCP clients.
Database failover from client perspectivePriit Piipuu
In this presentation we will look deep into high availability technologies Oracle RAC provides for database clients, what actually happens during database instance failover or planned maintenance and how to configure database services so that Java applications experience no or minimal disruption during planned maintenance or unplanned downtime. This presentation will mainly focus on JDBC and UCP clients.
Policy based cluster management in oracle 12c Anju Garg
Oracle Grid Infrastructure 12c enhances the use of server pools by introducing server attributes e.g. memory, CPU_count etc. which can be associated with each server. Server pools can be configured so that their members belong to a category of servers, which share a particular set of attributes. Moreover, administrators can maintain a library of policies and switch between them as required rather than manually reallocating servers to various server pools based on workload. This paper discusses in detail the new features of policy based cluster management in 12c.
OOW15 - technical upgrade best practices for oracle e-business suite 12.2vasuballa
This session is ideal for organizations thinking about upgrading to Oracle E-Business Suite 12.2. It covers the fundamentals, including the technology stack components and supported upgrade paths. Hear from Oracle development experts about the recommended practices for executing the Release 12.2 technical upgrade, with special considerations for minimizing your downtime. Also get to know about relatively recent upgrade resources.
A presentation about new features and enhancements related to indexes and indexing in Oracle 12c.
See also the related post: http://db-oriented.com/2015/07/03/indexes-and-indexing-in-oracle-12c
Oracle e-business suite R12 step by step InstallationOraERP
http://www.oraerp.com - Installing e-Business Suite R12 on Linux, Step by Step Guide for Oracle E-Business Suite R12 installation. Redhat Linux Installation, Pre-requisite/Preparation for Oracle e-Business suite R12, Readiness with Oracle EBS R12 software from Media or e-Delivery
In-depth overview of Oracle Real Application Clusters (RAC) 12c Release 2, which was first presented during UKOUG Tech16 under the title "Under the Hood of Oracle Real Application Clusters (RAC) 12c Release 2" and before Oracle Database 12c Release 2 became generally available (GA) in March 2017.
The Autobahn Has No Speed Limit - Your XPages Shouldn't Either!Teamstudio
Using XPages out of the box lets you build good-looking and well-performing applications. As XPage applications become bigger and more complex, performance can become an issue. There are several ways to improve scalability and performance that you should take into consideration. In this webinar, learn how to use partial refresh and partial execution mode and how to monitor its execution using a JSFLifeCycle monitor to avoid multiple re-calculations. See how readily available tools from OpenNTF will allow you to profile and analyze your code to improve the speed of your applications. Using Server Side Java Script and encountering a significant slow down when using Script Libraries? Learn how you can improve the speed of your application using JAVA instead of JavaScript, JSON and even @formulas.
Mixing d ps building architecture on the cross cutting examplecorehard_by
В рамках доклада мы поговорим о важности архитектурных решений, в том числе, для обеспечения высокого качества ПО при минимальных трудозатратах. Сквозной пример из области резервного копирования данных позволит лучше понять техническую, QA и общепроцессную составляющие подхода. Прошло достаточно времени, чтобы раскрыть технические детали без нарушения NDA, предложенный вариант на базе метрик, которые мы обязательно упомянем, был признан лучшим архитектурным решением в рамках компании – одного из лидеров отрасли, получил награду Microsoft, был «размножен» на смежные области. Приступаем: Builder, Decorator, Composite, Iterator и Visitor - как эти паттерны помогли решить нетривиальную С++ задачу.
LinkedIn leverages the Apache Hadoop ecosystem for its big data analytics. Steady growth of the member base at LinkedIn along with their social activities results in exponential growth of the analytics infrastructure. Innovations in analytics tooling lead to heavier workloads on the clusters, which generate more data, which in turn encourage innovations in tooling and more workloads. Thus, the infrastructure remains under constant growth pressure. Heterogeneous environments embodied via a variety of hardware and diverse workloads make the task even more challenging.
This talk will tell the story of how we doubled our Hadoop infrastructure twice in the past two years.
• We will outline our main use cases and historical rates of cluster growth in multiple dimensions.
• We will focus on optimizations, configuration improvements, performance monitoring and architectural decisions we undertook to allow the infrastructure to keep pace with business needs.
• The topics include improvements in HDFS NameNode performance, and fine tuning of block report processing, the block balancer, and the namespace checkpointer.
• We will reveal a study on the optimal storage device for HDFS persistent journals (SATA vs. SAS vs. SSD vs. RAID).
• We will also describe Satellite Cluster project which allowed us to double the objects stored on one logical cluster by splitting an HDFS cluster into two partitions without the use of federation and practically no code changes.
• Finally, we will take a peek at our future goals, requirements, and growth perspectives.
SPEAKERS
Konstantin Shvachko, Sr Staff Software Engineer, LinkedIn
Erik Krogen, Senior Software Engineer, LinkedIn
Docker & aPaaS: Enterprise Innovation and Trends for 2015WaveMaker, Inc.
WaveMaker Webinar: Cloud-based App Development and Docker: Trends to watch out for in 2015 - http://www.wavemaker.com/news/webinar-cloud-app-development-and-docker-trends/
CIOs, IT planners and developers at a growing number of organizations are taking advantage of the simplicity and productivity benefits of cloud application development. With Docker technology, cloud-based app development or aPaaS (Application Platform as a Service) is only becoming more disruptive − forcing organizations to rethink how they handle innovation, time-to-market pressures, and IT workloads.
This is episode 4 of the building the perfect PHP app for the enterprise webinar series. Nothing is faster than a frustrated user clicking away from your site or abandoning your service. Avoid attrition by learning how to tune your applications towards lightning-fast page loads and response times. Learn: the basic principles behind enterprise PHP performance management; How to optimize workloads through background jobs and caching; How to measure performance and make data-driven decisions.
Abstract
The idea of this talk is to help development teams to make correct architectural decisions.
Andrei will highlight the basic architectural principles and show ways to achieve architecture that is good enough to cover the project requirements and evolve in the future.
He will also present several cases from real projects, where wrong, missing, or over-sophisticated architecture decisions really hurt the development teams:
- Painful sharing: do shared modules increase reusability or will be the source of problems?
- Microservices are the solution to every problem!
- Non-extensible extensibility: too sophisticated configuration hurts
- Over fine-grained: incorrect splitting to Microservices can make life even harder as with monolith
- Convey horizontal split: how organizational driven split can jeopardise the architecture
- Model-driven: central responsibility blocks and limits the team
- Cargo cult: blindly following patterns and rule can produce an unmaintainable system
- Freestyle architecture: what happens if teams completely ignore architecture
- Improve with less intelligence: smart endpoint and dumb pipes
Abstract
The idea of this talk is to help development teams to make correct architectural decisions.
Andrei will highlight the basic architectural principles and show ways to achieve architecture that is good enough to cover the project requirements and evolve in the future.
He will also present several cases from real projects, where wrong, missing, or over-sophisticated architecture decisions really hurt the development teams:
- Painful sharing: do shared modules increase reusability or will be the source of problems?
- Microservices are the solution to every problem!
- Non-extensible extensibility: too sophisticated configuration hurts
- Over fine-grained: incorrect splitting to Microservices can make life even harder as with monolith
- Convey horizontal split: how organizational driven split can jeopardise the architecture
- Model-driven: central responsibility blocks and limits the team
- Cargo cult: blindly following patterns and rule can produce an unmaintainable system
- Freestyle architecture: what happens if teams completely ignore architecture
- Improve with less intelligence: smart endpoint and dumb pipes
Software Architecture and Architectors: useless VS valuableComsysto Reply GmbH
Abstract:
This talk introduces definitions of system architecture and proposes a way to achieve "good enough" architecture covers project requirements
Andrei will show several cases from real projects, where wrong, missing or over-sophisticated architecture decisions really hurt the development teams:
Painful sharing: do shared modules increase reusability or will be the source of problems?
Non-extensible extensibility: too sophisticated configuration hurts
Over fine-grained: incorrect splitting to microservices can make life even harder as with monolith
Cargo cult: blindly following patterns and rules can produce an unmaintainable system
Freestyle architecture: what happens if teams completely ignore architecture
Improve with less intelligence: smart endpoint and dumb pipes
We are looking forward to meet many of you in person and have great discussions around this topic!
https://www.meetup.com/de-DE/meetup-group-tfyvuydp/
On-the-Fly Containerization of Enterprise Java & .NET Apps by Amjad AfanahDocker, Inc.
Dockerizing brownfield enterprise applications can often be a daunting task - involving changes to the application code/configuration and existing build processes. The DCHQ platform provides “on-the-fly” containerization of both Linux & Windows enterprise applications – including Java, Oracle, .NET and others. By doing so, DCHQ transforms non-cloud-native applications into completely portable applications that can take advantage of cloud scaling, storage redundancy and most importantly, deployment agility without introducing a single change to the application source control repository.
In this session, we will cover the deployment automation of an Enterprise Java application with PostgreSQL multi-host cluster set up for Master-Slave replication and automated storage management with redundant EBS volumes on AWS using DCHQ + EMC REX-Ray. We will also cover the deployment automation of an Enterprise .NET application demonstrating the application life-cycle management capabilities post-provision -- including monitoring, alerts, continuous delivery, application backups, scale in/out, in-browser terminal to access the containers, log streaming, and application updates.
R12.2 is no more a new kid on the block. With its latest release of 12.2.4, it is much more stable and user adoption is increasing day-by-day. Upgrading to R12.2 is on the road map of nearly all Oracle E-business Suite customers and many organizations have already started planning their upgrades. In this session we provide 10 quick tips to consider while you plan this R12.2.4 upgrade.
The Fastest Way to Redis on Pivotal Cloud FoundryVMware Tanzu
What do developers choose when they need a fast performing datastore with a flexible data model? Hands-down, they choose Redis.
But, waiting for a Redis instance to be set up is not a favorite activity for many developers. This is why on-demand services for Redis have become popular. Developers can start building their applications with Redis right away. There is no fiddling around with installing, configuring, and operating the service.
Redis for Pivotal Cloud Foundry offers dedicated and pre-provisioned service plans for Cloud Foundry developers that work in any cloud. These plans are tailored for typical patterns such as application caching and providing an in-memory datastore. These cover the most common requirements for developers creating net new applications or who are replatforming existing Redis applications.
We'd like to invite you to a webinar discussing different ways to use Redis in cloud-native applications. We'll cover:
- Use cases and requirements for developers
- Alternative ways to access and manage Redis in the cloud
- Features and roadmap of Redis for Pivotal Cloud Foundry
- Quick demo
Presenters: Greg Chase, Director of Products, Pivotal and Craig Olrich, Platform Architect, Pivotal
Similar to AOUG_11Nov2016_Challenges_with_EBS12_2 (20)
2. 2ni.com
About National Instruments
Founded: 1976
Corporate Headquarters: Austin, Texas
Industry: Test and Measurement
Our Mission:
We equip engineers and scientists with systems that
accelerate productivity, innovation, and discovery.
Revenue: $1.23 billion in 2015
Global Operations: Approximately 7,100 employees;
operations in almost 50 countries
IT Development Centers:
• Austin, Texas
• Debrecen, Hungary
• Penang, Malaysia
• San José, Costa Rica
• Shanghai, China
3. 3ni.com
Agenda
• Review of 12.2 Architecture and ADOP Procedures
• 12.2 Architectural Changes and Philosophy
• Online Patching
o File System Editioning
o Edition Based Redefinition
o ADOP Patching Cycle
• Custom Development Standards
• Full vs Runtime Compliance
• Custom Development Options
• Lessons Learned
• Database Object Grants
• DB_Domain
• Executing Autoconfig
• Administering Application Nodes
• Challenges with Online Patching
• ADOP - Useful Utilities
5. 5ni.com
Copy the Code, NOT the Data
• EBS uses both the file system and the database to store the code and data that
make up the application
• Code
o Stored both on the File system & In the Database
o Any code object changed in a patch is copied
• Data
o Stored both on the File system & In the Database
o Application data is NOT copied by a patch
6. 6ni.com
Downtime Redefined as “Cutover”
• Cutover is the time taken to switch users from the production system to the
newly patched copy
• Cutover changes the unit of measure for downtime
• Measured in minutes NOT hours or days
• Cutover time is very predictable
• The time taken to bounce the Middle Tiers
Changing the Unit of Measure for Downtime
7. 7ni.com
Downtime Limited to Short Cutover
• Users are logged off the production Instance
• When users reconnect they are directed to the newly patched system
E-Business Suite
12.2.3
Patches
•12.2.4
•12.2.5
•….
E-Business Suite
12.2.3
Production Patched Production
8. 8ni.com
• Downtime Patches
• No online users
• Wall clock time very important
• Consumes all resources
• Upgrade designed to run as fast
as possible
• Online Patches
• User remain online
• Wall clock time is no longer an
overriding concern
• Online users share resources
• Data upgrades designed to not
affect the running application
Changing the Patching Paradigm
12. 12ni.com
• Run file system
• Used by online users
• Stores a complete
copy of all Applications
and Middle Tier code
• Logically mapped to
either fs1 or fs2
• Patch file system
• Used by patching tools
• Stores a complete
copy of all Applications
and Middle Tier code
• Logically mapped to
either fs1 or fs2
• Non-Editioned file
system
• Used for data files
e.g.: data import/export
files, log files, report
output files
• Only stores data files
Online Patching uses a Dual File System
fs1 and fs2 switch Run and Patch designation during the cutover phase of an Online
Patching cycle
fs1
Run
fs1
Cutover
fs1fs2
PatchPatch
fs2
Run
14. 14ni.com
11gR2 Edition-Based Redefinition (EBR)
• Key enabling technology for Online Patching
• Enables the online upgrade of the application definition
• Allows an application to efficiently store multiple copies of its code and schema in the same
database
• Provides an isolation mechanism that allows pre-upgrade and post-upgrade
schemas to co-exist
• Changes to database objects are made in the isolation of an “Edition”
• Changes to database objects do not affect the running application
15. 15ni.com
Online Patching is Enabled by 11gR2 Edition-Based Redefinition
Multiple Copies of Database Code Objects Can Coexist
• Client code connects to an “edition”
of the database
• Run Edition
o Used by online users
o Never changed by a patch
• Patch Edition
o Used by patching tools
o Changes do not affect the running application
• Patch Edition becomes the
Run Edition at Cutover
15
DBA
Business Users
17. 17ni.com
Online Patching Cycle - Overview
Understanding the Online Patching Cycle
• The Basics
• Remove
obsolete
objects
Cleanup
• Restart
application on
Patch Edition
Cutover
• Compile invalid
Objects
• Wait for a good
downtime
window
Finalize
• Apply one or
more patches to
the Patch Edition
Apply
• Copy the
production
application
code
• Create a new
Patch Edition
in the database
Prepare
Users Online Users OnlineUsers Offline
• Online Patching is used to apply all patches in 12.2
• Online Patching cycle includes 5 major phases
• Application is only offline during the Cutover phase
19. 19ni.com
Full Compliance vs Runtime Compliance
• Full Compliance
• Development meets all standards to
support runtime and online patching
operations
• Specialized deploy methodology to
implement changes in the patch
edition/file system
o See Doc ID 1577661.1 for all online
patching compliance standards
• All Oracle eBusiness Suite products
meet full compliance
• Runtime Compliance
• A subset of the full compliance
standards that allow a product to
operate correctly in R12.2
• Customizations and third-party tools
may only meet runtime compliance
20. 20ni.com
Pro/Cons of Online Patching Full Compliance
• Oracle’s Deployment Methodology
• Doc ID 1577661.1 – Section 1.4
• Standard application development is executed directly against the run edition
• Option #1: Apply Customizations as Online Patch. When application development is
completed, a custom patch is created and deployed in an ADOP cycle
• Option #2: Apply customizations as a downtime patch. Custom code is added to run edition in
a downtime window
• Advantages
• Higher guarantee of system stability
• Disadvantages:
• Changes to the Applications are less agile – Both paths require downtime
• Option #1 requires the creation of patches. Steps vary based on AD/TXK patchset level and
database objects being modified. See Part 2 of Doc ID 1577661.1
21. 21ni.com
Disadvantage of Runtime Compliance
• Deployment Methodology
• Deploy code directly to the run edition
• Advantages
• Higher degree of flexibility. Code changes can be done without system wide downtime
• Removes the learning curve for developers
• Disadvantages
• Extra maintenance required before ADOP cycle. Must execute a process to synchronize
run and patch file systems
o Delta 7 and below: fs_clone or custom process
o Delta 8: Possibly taken care of in the prepare phase
• Higher risk of system stability issues during code deploys
23. 23ni.com
Database Object Grants
• Doc ID 1987947.1
• Symptoms
• Grants cannot be performed in the run edition when the application is being used
• Granting privileges on an object may cause object invalidations in the current edition
• Problem Description
• When grants are given to "Stub objects" it invalidates all dependent stub objects.
• “Stub objects" are simply pointers to an actual object definition from an ancestor edition
that is still being inherited by the given edition.
24. 24ni.com
Database Object Grants
Solutions
• Option #1: Execute an ADOP cycle when granting privileges on the APPS
schema to other users
• Option #2: Leverage new API for granting privileges with minimum
invalidations
• API: AD_ZD.GRANT_PRIVS
25. 25ni.com
Database Object Grants
• Syntax:
ad_zd.grant_privs
(X_PERMISSIONS in varchar2,
X_OBJECT_NAME in varchar2,
X_GRANTEE in varchar2,
X_OPTIONS in varchar2 default NULL);
• X_PERMISSIONS: comma-seperated permission list
• X_OBJECT_NAME: name of APPS object or APPS synonym
• X_GRANTEE: grantee schema or role
• X_OPTIONS: grant options, example: 'WITH GRANT OPTION'
26. 26ni.com
DB_Domain
• Specifies the logical location of the database within the network structure
• Rapid Clone and ADOP are making assumptions about the DB_Domain
• Assumption #1: DB_Name + DB_Domain is less than 8 characters
o If this condition is not met, Rapid clone on the Application tier will not execute
o Workaround: Unset DB_Domain, run rapid clone, reset DB_Domain, run Autoconfig
• Assumption #2: The DB_Domain has the same value as the host domain
o In our case:
– DB_Domain = WORLD
– Host Domain = natinst.com
o ADOP builds a JDBC connect string during the ADOP patching cycle
o Workaround: Create a service for db_name + host_domain
– d1ebiz.natinst.com
27. 27ni.com
ExecutingAutoconfig
• Application services should be down while running autoconfig
• Autoconfig has to be executed on ALL application tier nodes
• Autoconfig cannot be executed in parallel on multiple application tier nodes
• While the autoconfig script will show a successful completion, data is not properly
inserted into FND_OAM_CONTEXT_FILES
• This will lead to potential issues during the next ADOP cycle in which nodes may need to
be abandoned
28. 28ni.com
AdministeringApplication Nodes
• Adding Application Nodes
• Doc ID 1375769.1
• Application nodes cannot be added or removed when application services are running
• Major steps
• Setup ssh between Primary APPS Tier and New Node
• Run adpreclone.pl appsTier for both patch and run file system
• Create Oracle Inventory
• Prepare Pair File for adclonectx
31. 31ni.com
AdministeringApplication Nodes
Execute the Add Node Utility
/opt/apps/r12/<db_name>/apps/<run_fs>/FMW_Home/webtier/p
erl/bin/perl ./adclonectx.pl
addnode contextfile=<full path of the cm1 node's Run
File System Context file>
pairsfile=<full path of custom pairsfile>
dualfs=yes
32. 32ni.com
AdministeringApplication Nodes
• Changing apps.conf and mod_wl_ohs.conf
perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl
-contextfile=<full path of the <host> Run File System Context file>
-configoption=addMS
-oacore=<new host>.natinst.com:<oacore port number>
-oafm=<new host>.natinst.com:<oafm port number>
-forms=<new host>.natinst.com:<forms port number>
-formsc4ws=<new host>.natinst.com:<formsc4ws port number>
33. 33ni.com
AdministeringApplication Nodes
• Change context file entries (if needed), eg:
Change following entries from enabled to disabled on CM nodes
s_oacorestatus
s_formsstatus
s_oafmstatus
• Run autoconfig
34. 34ni.com
AdministeringApplication Nodes
• Adding Managed Servers
• Doc ID 1905593.1
• New managed servers can be added when the applications are running
• However, they will not be used until the applications are stopped and autoconfig is
executed
perl $AD_TOP/patch/115/bin/adProvisionEBS.pl
ebs-create-managedserver -contextfile=$CONTEXT_FILE
-managedsrvname=forms_server17 -servicetype=forms
-managedsrvport=7412 -logfile=/tmp/addforms_server.log
35. 35ni.com
AdministeringApplication
• E-Business Suite Technology Codelevel Checker (ETCC)
• Rapid Install StartCD 50 for EBS 12.2 comes with a 11.2.0.3 database
• Since Feb. 22, 2016, Rapid Install StartCD 51 is available. It comes with 12.1.0.2 DB.
You can download as Patch
22066363
• Since Sept 15, 2016, 12.2.6 is available. New Installation and Upgrade (upgrading to
Oracle E-Business Suite Release 12.2.6 from Release 11i, 12.0, or 12.1) customers
should use StartCD 12.2.0.51. (Doc ID 2114016.1)
• Weblogic 11.1.1.7 upgrade to 11.1.1.9 was not possible till we applied all ETCC
recommended patches
36. 36ni.com
Challenges with Online Patching
• Serial number in context file should match the value of database copy. If not,
adop can fail (Doc ID 1916658.1)
• adop phase=fs_clone
• In our environment, performance is at least 3 hours
• The addition of more application nodes increases the time for the procedure to complete
• Recommended to be executed after any configuration changes
• Not all files get transferred on slave nodes
o oaea_wls.properties
• adop fs_clone allnode=yes|no
• adop fs_clone force=yes
37. 37ni.com
Challenges with Online Patching
• How it supposed to look like (eg.:):
SSO_SERVER_RELEASE=11
SSO_SERVER_URL=https://winnie-app.natinst.com:443
SSO_SERVER_TYPE=OAM
LOG_CONFIG_FILE=DEFAULT
APPL_SERVER_ID=3DA0E4FD00B9625FE05400144FFB48B032374020223902728214531018323218
OAM_LOGOUT_URL=/accessgate/logout
CONNECTION_REF=jdbc/OAEADatasourceDS
WEBGATE_LOGOUT=
oracle.apps.fnd.sso.WebEntries=DEFAULT
• How it was looking like (everywhere, except master node):
SSO_SERVER_RELEASE=
SSO_SERVER_URL=
SSO_SERVER_TYPE=
LOG_CONFIG_FILE=DEFAULT
APPL_SERVER_ID=
OAM_LOGOUT_URL=
CONNECTION_REF=
WEBGATE_LOGOUT=
oracle.apps.fnd.sso.WebEntries=DEFAULT
38. 38ni.com
Online Patching – Useful Utilities
• adopmon - utility to monitor the progress of adop session.
• adopreports - Online Patching Diagnostic Reports utility,
$AD_TOP/bin/adopreports, can be used to help diagnose issues or simply
gather information about the status of your system.
• txkADOPValidations.pl - before prepare, cutover or fs_clone, this tool can
validate the instance if it is ready for those phases
perl $AD_TOP/patch/115/bin/txkADOPValidations.pl -contextfile=$RUN_CONTEXT_FILE -
patchctxfile=$PATCH_CONTEXT_FILE -phase=fs_clone -logloc=/tmp
• adopscanlog - utility makes the work easier and analyzes adop log
directories for errors and warnings
Online patching has 5 cycles:
First, the system is prepared by creating a copy of production code and creating a new edition in the database
Then, you apply one or more patches to the patch edition with adop
Next, the finalize phase is executed which includes compiling invalids
Then you wait for a time to cutover from the run filesystem and edition to the updated patch filesystem and edition
Cutover – is in red as this is the only time the system is down
You execute cutover, when it is convenient for your environment
Finally you can execute the cleanup phase any time after cutover