How using Puppet and enterprise modules Puppet modules for Oracle and WebLogic provisioning, enables devops in large enterprises.

1. EASY ORACLE
AND WEBLOGIC
PROVISIONING
& DEPLOYMENT

2. TODAY
ENTERPRISE DEVOPS. CAN IT
BE DONE?
AUTOMATION IS REQUIRED THE VALUE OF
COMMUNICATION
ORACLE AND WEBLOGIC
CONFIGURATION
HOW DOES IT WORK?
WAT IS THE ISSUE? HOW TE GET STARTED?

3. INTRODUCTION
• More then 20 years experience in IT
• As project manager & development
manager at large IT companies and
government
• Founder of Enterprise Modules (twitter:
@enterprisemodul)
• (Co)developer of Puppet modules for
Oracle RDBMS en WebLogic
• bert.hajee@enterprisemodules.com
• twitter: @bhajee

4. MISSION STATEMENT
Provide our customers with
high quality Puppet modules
for all the Enterprise products.
Thereby allowing them to
exploit all advantages of
Puppet not only for their base
OS and open source
products, but for their entire
Enterprise Infrastructure
https://www.enterprisemodules.com
twitter: @enterprisemodul

5. FINALLY...
John looked at the ceiling. Yesterday, after more than one and a half years of preparation, the latest version of the application went live. But it took a lot
of effort and sparked a lot of negative energy.
The application was a registration application that used Oracle WebLogic, and stored all data in an Oracle database.
Before getting the application into production, the development team and the operations team where coastally at each other’s throat. Every time an issue
was reported, during the tests, they were pointed fingers at each other and calling names.
The operations guys blamed the developers. Saying the only delivered bad code. And that in the end the operations guys would be called in the middle
of the night to clean up the mess. The developers, on the other hand, said that the operations team was just splitting hairs and fretting about non
important issues. They were just stalling the project. If the operations team hadn’t been so fearfull of changes, the would already be running in production
for a long time.
When there was a performance problem, the DBA-er told the developer that he was not using the indexes correct. But the developers blamed the
operations team for not providing a correctly configured operating system.
Does anybody recognize these stories? These are stories we often hear from customers. Although the word DevOps has been buzzing in the world of
startups, it has been tough to get it to work in larger companies.

6. ENTERPRISE DEVOPS??
Wouldn’t it be great if we are able to do things in the devops-way?
Wouldn’t it be great if John did not have to wait for one and a half years to get this new release into production? But actually do a new release every
week. Or even do a release every day.
Wouldn’t it be great if it didn’t matter if the change was in the application or in the underlying middleware database or operating system? The change
would be done quick, consistent and painless.
Wouldn’t it be great if the development team and the operations teams worked together in close cooperation towards attaining these goals.

7. Or is it just impossible to get devops working in large companies……
Well…. I can tell you it is possible. In this presentation I will take you down the road that John and his team traveled. The steps they took to transform the
current status que into a smooth operation where they were able to do multiple deployments per week and make controlled changes to the applications,
middleware and operating systems.
This story is based on actual experiences we had with multiple customers, both in the Netherlands and in other parts of Europe.

8. FAST RELIABLE &
CONSISTENT
As I mentioned before, John and his team needed one and a half years to put the new version of the application into production. This time was divided
over
and his team had needed a half year to put the new version of the application into production. This time of course was divided on the various aspects
and stages of such development process. What I did notice was that it proved difficult to put down a fast reliable and consistent environment.
....
a lot of hand work ...
Inconsistent …
Sensitive to errors
Small annoying errors
labor-intensive
Takes a long time

9. AUTOMATION REQUIRED
For the team things where clear. Automation of the work flow was required.

10. Application
Problem??
WHAT IS THE PROBLEM???
But before we continue. Let’s see what the problem is. How can installing an application be a problem?

11. Application
Middleware
Problem?
WHAT IS THE PROBLEM??
Well, one of the reasons is, you normally need a little bit more then just an application. Most enterprise applications need some sort of middleware.

12. Application
Middleware
Database
Problem
WHAT IS THE PROBLEM?
And since data is important for large companies. Most of them need a database too.

13. Application
Middleware
Database
OS
Problem!
WHAT IS THE PROBLEM
And you would almost always need an operating system.

14. Problem!!
Application
Middleware
Database
OS
Network
WHAT IS THE PROBLEM!
And stuff needs to be connected to a network. And as you can see slowly but surely the number of components that need to be installed and
configured, increases.

15. Problem!!!
Application
Middleware
Database
OS
Network
Application
Middleware
Database
OS
Network
WHAT IS THE PROBLEM!!
Large companies most of the time, cannot afford downtime. So one of the risk mitigation counter measures used a lot it make the infrastructure high
available by duplicating everything. And as you can see, the tasks of installing and configuring you IT infrastructure increasingly becomes a challenge.

16. Problem!!!!
Application
Middleware
Database
OS
Network
Application
Middleware
Database
OS
Network
WHAT IS THE PROBLEM!!!
But even this is manageable. But when we start to look at the the number of individual settings that need to be controlled, we start to see the real
problem. The amount of stuff to manage is huge.

17. Problem!!!!!
Application
Middleware
Database
OS
Network
Application
Middleware
Database
OS
Network
WHAT IS THE PROBLEM!!!!
It is not only huge, it is also complicated. Some of the configuration points are interdepended. So when you change the size of the SGA, you also need
to change the amount of memory reserved. When you change the port number of a weblogic server, you also need to change the firewall rules to allow
communication to the specified IP port.
And we start to see why it has been incredible difficult to get all of the systems in our chain of systems configured correctly.

18. A SOLUTION
Like the figured out before, they needed a tool. A tool that would help them manage these infrastructure layers and the huge amount of configuration points.
Like the figured out before, they needed a tool. A tool that would help them manage these infrastructure layers and the huge amount of configuration
points.

19. In the team the chose Puppet. Reasons for that where:
• Communication
• Large and stable player in the market
• Big partners including Cisco, VMWare, Microsoft and EMC
• Multi-platform including windows
• Also for configuration of network and storage components
• Support for configuration Oracle databases and WebLogic

20. Your infrastructure on a blueprint
YOUR INFRASTRUCTUE IN A BLUEPRINT
For the people who don’t know, let me explain how puppet works. When you are using using Puppet you describe your infrastructure in sort of a blue
print. This blueprint contains a description of all the required components in your infrastructure, all the components on your servers. All the settings you
need, are described in your blueprint. For example…

21. What’s reality?
REALITY?
After Puppet has read your blueprint, it goes to the system. It starts to make pictures of how the components are really configured and if they are
installed at all. It makes an inventory of reality of the current system.

22. Spot the differences
SPOT THE DIFFERENCES
By comparing the blueprint and the pictures, it can make a list of differences. And based on the list of differences, it can make a list of
actions that need to be executed. For example: when your blueprint says it needs a datasource in domain ‘production’ in a WebLogic
12.2.1 environment, Puppet will look at its pictures and see no weblogic software is installed and obviously no datasource is configured.
The puppet modules will have the knowledge of what the required actions are to install the software, create the domain and create the
dataasource.
If however an existing datasource is available, but just the url of the database has changed, puppet will see everything is ok and just
change the url property of the datasource.
What is really important, is tha puppet describes the end result and not the procedural steps needed to get there. Because depending on
the begin situation, they might be different.

23. For new install’s and updates
NEW INSTALLS & UPDATES
This is a really handy mechanism. You can use this mechanism to provision a totally new system. Puppet will see a huge difference between the blue
print and the pictures and will make a long list of actions to execute.
But if there is just a small change, this mechanism also works. Just the list will be much shorter.

24. Blueprint for Oracle Table space..
ooora_tablespace {'my_app_ts@sid':
ensure => present,
datafile => 'my_app_ts.dbf',
size => 5G,
logging => yes,
autoextend => on,
next => 100M,
max_size => 20G,
extent_management => local,
segment_space_management => auto
},
}
NEW INSTALLS & UPDATES
Here is an example of a part of a blueprint. This is a definition of an oracle tablespace. Key here is that you don’t have to be a database
specialist to understand what is defined here. Later in the presentation I will come back to this.

25. ABSTRACTION
But how does this solve the issue of the many many connected configuration points in our infrastructure? and the answer is: Abstraction

26. Abstraction Application
Middleware
Database
OS
Network
Application
Middleware
Database
OS
Network
In essence Puppet provides a consistent abstraction layer over the real infrastructure components. It allows you to make your infrastructure
components into building blocks with a defined interface. Some of the building blocks are small and provide just a small configuration element. An
example of this, is the ora_tablespace definition we saw before.
But you can also put those small building blocks into a little bit bigger building blocks and define an interface for your standard configured weblogic
domain, or your standard application setup on your database.

27. Separation of ConcernsApplication
Middleware
Database
OS
Network
Application
Middleware
Database
OS
Network
DEV
MYAPP
10.10.10.100
Queue1
APP
DEV
ns1.dev.com
username
password
mw_home
db_home
wls_server
db_size
em_size
webserver
db_server
db_server
Another part of the way Puppet helps you is by separating concerns. Separating the what from the how.
Most of the times we need multiple kind of setups. We need a setup for development, we need a setup of production and in between we
need multiple setups for regular testing and acceptance testing. For the most parts, these setups are the same. But porbaby you need
different IP setting in testing then in production. And maybe we can do with a little bit less space in the database.
Puppet has a mechanism for allowing you to separate those settings from the rest. It is called hiera. Using hiera you can connect certain
settings in your infrastructure to values in your hiera key-> value lookup.
Here you see a setup where we have “wired" our setup for a development environment.

28. For different EnvironmentsApplication
Middleware
Database
OS
Network
Application
Middleware
Database
OS
Network
TEST
MYAPP
10.10.10.100
Queue1
APP
TEST
ns1.dev.com
username
password
mw_home
db_home
wls_server
db_size
em_size
webserver
db_server
db_server
Because we have extracted the information, it becomes really easy to transform this blueprint into a blueprint for a testing environment.
We just have to adjust the specified settings and we can use the same blueprint with a different layover to manage our testing
environment.

29. Through AcceptanceApplication
Middleware
Database
OS
Network
Application
Middleware
Database
OS
Network
ACC
MYAPP
10.10.10.100
Queue1
APP
ACC
ns1.dev.com
username
password
mw_home
db_home
wls_server
db_size
em_size
webserver
db_server
db_server
And acceptance environment….

30. Finally in Production Application
Middleware
Database
OS
Network
Application
Middleware
Database
OS
Network
PROD
MYAPP
10.10.10.100
Queue1
APP
PROD
ns1.dev.com
username
secret
mw_home
db_home
wls_server
db_size
em_size
webserver
db_server
db_server
And of course also our production environment.

31. DIFFERENT KIND OF
ENVIRONMENTS
When John and his team started with puppet and the environment, things worked well. Finally they where able to consistently provision and deploy
systems and applications. The started with a Puppet blueprint for production, then the extracted the properties that needed to be different for acceptance
and put those into hiera. They gained a lot. But something was missing….
They noticed that a lot of issue still where only discovered when the application was deployed into the acceptance setup. There was still to big of a gap
between what the developers used on there development workstations and what was running in production.
What they wanted was to provide all developers with a minified version of the production environment on there own workstation. A setup that didn’t need
to be high available and high performance, but when looked at from the application, provide the same interface as a production environment. The same
queue’s, the same dataseource, the same tablespaces and the same tables. Just the clustering (for HA) and the sizing would be different.

32. PORTABLE PLATFORM
An they managed to get that working. It is called the portable setup. Now they can use the same puppet blueprint to:
- install a minified version of the setup on a PC. This contains both the database, the weblogic domain and loadbalancers.
- Install a test setup where the database is on one machine and the weblogic domain is on an other machine
- And a HA setup where they use a RAC oracle database and a 10 node WebLogic cluster to run the application on.
John and his team noticed that a lot of the late issues disappeared. Because the developers had access to the same setup as used in production, a lot of
setup issues like missing tablespaces, differently named queue’s and datasources, would be found much earlier in the development process.

33. PLATFORM CI
Platform Code
Application Code
Because of this easy portable setup, they were able to get two interconnected CI environments.

34. PROCES CHANGES
Because of this interconnected CI environments, they had a consistent view of the quality of the platform and application. So they became pretty
confident. At any moment they were sure they could provision and deploy the latest combination of working platform with a working application.
Based on this confidence, they started to change their processes. Every week they would release a version of their platform puppet code and there
application. Depending on the customer’s needs, they would just use this in their development chain, or deploy this to production.
With this process they achieved agility while retaining stability. With this mechanism and process they were able to deploy new versions of the
application, provision changes in the WebLogic and Oracle setup and provision patches, both on the OS as well as in the middleware layer.

35. COMMUNICATION
AND COOPERATION
Automating their setups sure helped. But John and his team where certain tooling was not enough to overcome the conflicts and heated discussions
between the developers and the operations guys. Something had to be done to improve the collaboration.
When you look at the literature, one of the key enablers of collaboration, is communication. Many development teams use a Kanban board to enable
communication about the status of the project. The scrum methodology uses standups for communication and mandates that a team has all required
skills and knowledge to do everything needed from idea to delivery.

36. A BIG TEAM
Based on the scrum methodology, they thought about having one big team. A team that would contain all required skills and be responsible and
accountable for the whole process.
But they noticed it didn’t work. The amount of different IT skills is large in large companies. If you would put all those skills in one team, the team would
become huge.
Also because of corporate policies it is hard to get all authorizations into that one team. Large companies often require separation of duties and
authorities.
So… Although in theory very nice. Not practical for large companies.

37. COMMON LANGUAGE
But while John and his team where looking at solutions, they noticed the amount of conflicts was decreasing. There were still heated discussion, but they
seemed to be targeted at some specific setting.
The use of Puppet seemed to have a huge positive impact on communication as well.

38. FACILITATES TARGETED DISCUSSIONS
The simplicity of the Puppet language provided a common language between the development team and the operations team. No longer were they
fighting over some unseen settngs somewhere, but they were discussing the pro’s and con’s of certain settings.

39. SMALL DEMO

40. OUR PUPPET
MODULES

41. OUR MODULES

42. ORA_CONFIG module
• ora_asm_diskgroup
• ora_asm_volume
• ora_database
• ora_exec
• ora_init_param
• ora_listener
• ora_object_grant
• ora_record
• ora_role
• ora_schema_definition
• ora_service
• ora_tablespace
• ora_user
• AND MORE
ORA_INSTALL module
• db_control
• db_listener
• db_rcu.rb
• db_opatch
• installdb
• installem
• install_emagent
• installs
• opatchupgrade
• tnsnames
• net
• goldengate
• client
• autostartdatabase
FOR ORACLE WE HAVE GOT

43. WLS_CONFIG module
• wls_authentication_provider
• wls_cluster
• wls_datasource
• wls_deployment
• wls_domain
• wls_jms_queue
• wls_jms_topic
• wls_messaging_bridge
• wls_role
• wls_saf_imported_destination_object
• wls_saf_remote_context
• wls_server
• wls_workmanager_constraint
• AND MUCH MUCH MORE
WLS_INSTALL module
• wls_install::bsu
• wls_install::cluster_node
• wls_install::domain
• wls_install::fmw
• wls_install::managed_server
• wls_install::nodemanager
• wls_install::opatch
• wls_install::packdomain
• wls_install::software
• wls_install::storeuserconfig
• wls_install::utils::fmwcluster
• wls_install::utils::fmwclusterjrf
• wls_install::utils::oimconfig
• wls_install::utils::webtier
• … AND MORE
FOR WEBLOGIC/FUSION WE HAVE GOT

44. HOW TO GET STARTED?

45. IN RETROSPECTIVE

46. BUSINESS AS USUAL
John was staring at the ceiling. He was just thinking back about a couple of months ago. Then they needed months, or even more than a year for a new
version in production. And not only that, but these times would be filled with conflicts and over heated arguments,
But not anymore. They were able to put a release into production any time. It doesn’t matter if it is a functional change to the application, of changes in
middleware, database or OS.
Although in case of an emergency, they were able to a release any time, the customers most of the time didn’t need this. So they adjusted their process
to a release cycle of a small changes every week and once a month they did a larger release. For the larger release they had more personnel standby
for problems, but until now the larger release had been just as stable as the smaller releases.