DevOps has become the de facto approach for custom software delivery. Yet, if automation is claimed to be the answer to all ills, why do many organisations struggle to implement it well? This session reflects on experiences from the last decade or so of provisioning projects, highlighting lessons (and one or two regrets!) and considers how organisations building custom software should focus their Oracle platform engineering efforts to deliver better software to users, faster.

1. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Platform Engineering for
the Modern Oracle World
Simon Haslam
Director of Infrastructure & Platforms
eProseed
1 & 2 December 2022 | Birmingham

2. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
DevOps has become the de facto approach
for custom software delivery.
Yet, if automation is claimed to be the
answer to all ills, why do many organisations
struggle to implement it well?
2

3. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
DevOps is more than just Automation
Lots of definitions, but I like AWS’ :
DevOps is the combination of cultural philosophies, practices, and tools that increases
an organization's ability to deliver applications and services at high velocity: evolving
and improving products at a faster pace than organizations using traditional software
development and infrastructure management processes.
3
https://aws.amazon.com/devops/what-is-devops/

4. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Change is Continuous
• Business change / enhancements
• Bug/security fixes
• New products
• New technology
etc
4
This change affects from top to bottom of the application stack

5. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Landscape
5
Infra
Platforms
Apps
Operations
Development
‘Monster’ apps Not discussed in this presentation
Automation
Cloud
services

6. 6
Lessons from a decade (or so)
of Oracle Platform Provisioning

7. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Background
• Design & deployment of Oracle product suites
– mostly Fusion Middleware but some Database
– SOA/BPM, OSB, ODI, Forms, WCP, OIM/OAM, ODI, OID, OVD, WebLogic, etc
– production Oracle Cloud since 2016 (SOAC, DBCS, ATP, OAC, OIC, DIPC, ODI MP, SOA MP)
– always HA, normally +DR
• Enterprise Deployment Guides & Maximum Availability Architecture
• Solely or with a team; ½ with peers at client (usually Ops/Support)
• Drawing on experience of ~25 projects
7
Manual
Various
scripted
State-
driven
Cloud
native

8. 8
The “80-page” Word doc

9. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Traditional Approach
• Long Word doc
– Easier to use than Oracle docs (e.g. contains screenshots)
– Can get out of date
• on-prem: slowly
• cloud: quickly
– Suits very static environments that don’t get rebuilt often
• There is always a set of post-provisioning steps which
may/may not be included
• Often if environment build was outsourced
9

10. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
• Documents almost always incomplete; often it’s
just a snapshot
• Not reviewed by anyone else
• Focus can be on wrong things
• Choices not justified especially if no reference
architecture to back it up
• Probably (hopefully!) includes applying the latest
patches, but almost never considers Day 2
operations
• Inaccuracies develop over time
– Changes to vendor software
– Manual processes – e.g. how to get SSL certs
• Version doc carefully & reference
envs built with it
• The bulk of the “code” belongs to
the vendor. Anything written
down is better than nothing
• Still viable for some kinds of
environments
10
~Word Doc: Weaknesses & Recommendations
Recommendations

11. 11
VM Golden Images

12. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
VM / OS Images
• Create image from raw operating system baseline, or use tools providing
introspection of existing envs
• EMCC supported it for a while (still does?)
• “Golden Image” concept is It is the basis of containerisation… immutable images etc.
Big difference is that container imaging (Docker etc) was always designed by layers
so that the image can be easily re-created or a layer updated
12

13. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
• Tooling is very OS specific
• Drags in whole set of additional
concerns, e.g. storage
• Hard to have idempotency & slow
to re-provision
• As soon as the creators leave it
will never get used again!
• Almost no use cases now - avoid
13
VM Image: Weaknesses & Recommendations
Recommendations

14. 14
Shell Scripts

15. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Shell Scripts
• Bash/Korn are very widely understood
• Variables (at a basic level)
• Can then call DBCA, WLST, sqlldr, etc
• DBCA templates, domain templates provide building blocks
• Versionable
• Solutions will be unique so there’s minimal help in the
community
15

16. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
• Variable system but weak
• Minimal/no error handling
• Difficult to achieve reliable
idempotency
• As soon as the creators leave it
will never get used again
– At which point you have no
provisioning and very little doc
• Best avoided, except for very
tactical
• Ensure git is on all hosts running
scripts & you can easily push
• Do at least basic error handling
• Use a rudimentary file service
(cloud files, simple webserver)
rather than copying files
16
Shell Script: Weaknesses & Recommendations
Recommendations

17. 17
2nd Generation Automation Tools
Puppet & Chef

18. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Puppet & Chef
• Tooling was at forefront of DevOps
– See also Puppet Labs’ State of DevOps Reports & research
• Infra As Code – can build very complex multi-node envs
(e.g. 8+ nodes)
• Likely still used in some Oracle PaaS products
• It will probably continue to work of on-prem; if cloud it’s
almost certainly broken if not recently tested!
• These were first builds we started running on CI/CD (on OCI)
• Edwin Biemond’s orawls modules were very mature & well
maintained for some time
18

19. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
• Both Puppet & Chef have been
acquired
• orawls not maintained since12.2.1
• DSL & Ruby is not easy
• Client installations needed
• Open Source licence approvals,
e.g. outside Red Hat
• If using Cloud APIs then needs
regular testing, otherwise assume
broken
• This tooling is the most at risk
• Avoid for new environments
unless you have existing
infra/investment
19
Puppet & Chef: Weaknesses & Recommendations
Recommendations

20. 20
3rd Generation Automation Tools
Ansible… and on to Terraform

21. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Ansible
• Puppet & Chef more focussed on state convergence; Ansible allows
script-like playbooks
– Easier for shell programmers to pick up
• Ansible is more human readable than most
• Script/sequential layout used mostly for fire & forget
• Depends on modules – if cloud you need to keep testing
– E.g. migration from oci.modules
• Meets the “it’s more than shell” criteria but also not 10x better
• Acquired… but by Red Hat (…IBM)
• Where vendors (e.g. firewall) provide 3rd party automation most
likely to be Ansible modules
21

22. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
• Standalone Playbooks allow
(encourage) fire & forget
bad practice
• Some of the constructs don’t
model reality well
• Galaxy support still variable
• Ansible is probably currently the
best way to do configuration
management
• If vendor has Ansible modules, but
no Terraform providers, then
consider it
22
Ansible: Weaknesses & Recommendations
Recommendations

23. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Terraform
• Has state management at its centre
– (it didn’t invent this!)
• De facto standard, especially for public cloud
• Huge ecosystem of providers or write your own (Golang)
• Very little competition
23

24. 24
State Management & GitOps

25. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
GitOps
• If state definition purely defines the application
operating environment it has to be careful managed
• Git is natural choice for version control system today
• Logical step for controlling state changes by changing it
in git and then the state updates accordingly
 GitOps
• Every environment has a definition in code
• Environments are created from templates
• Disposable since created automatically
• Fosters continuous improvement
25
Containers running application images are the
most common manifestation of GitOps but
actually the concepts are not linked

26. 26
Lessons from re-visiting Old Projects

27. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
A few thoughts
• Automation is not easy
– Commonly people quote automation breakeven from 2 or 3 manual
times…
• Compared to a console, automation will take (subjective
opinion)
– 3x as long if you know the tooling
– 10x as long if you don’t know the tooling, or even are rusty
• Automation code, especially cloud, deteriorates quickly
– Cloud APIs evolve for new features (e.g. OCI flexible load balancers)
– Cloud providers follow/make the trends (e.g. OCI Ansible modules)
– Tooling trends over the lifetime of a typical enterprise product, say 10
years)
27

28. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Maintenance
28
We must have
Infrastructure
as Code
DevOps will
make us great
again
Image credit:
https://dogstrust.org.uk
Automation software is only valuable
if it is constantly maintained
Everything I’ve built could be improved in
at least small ways
Be careful of what you wish for:

29. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Automation software needs commitment
Sadly, all are real:
• “We think it would have still worked, but we didn’t want
to something to break so we did it manually”
• “The new environments are the same but we have to do
a lot of work to get them right”
• “Yes, your code was really good – it helped us work out
all the commands we needed to put into a shell script”
29
All of these destroy value of automation investment

30. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
What DevOps isn’t…
• DevOps is not just automation – it includes culture &
ways of working
• DevOps is not a single role/team
– Desire to do Ops in a more dev/codified manner
– Mindset for changing systems
– Lowering the barrier between Dev & Ops
but there’s still a distinction IMO
• Organisations take time to change
• Autonomy facilitates speed, but only up to a point
30

31. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
First implication of DevOps
• Complexity of environment provisioning (and non-functional
requirements) moved towards Dev
– Historically some had been hidden by IDEs & dev envs supplied separately
– Docker started as a developer productivity aid & is fairly simple
– The non-functional requirements drive multiple running copies of software
(and led to Kubernetes)  complexity
– Agility requirements led to more loosely-coupled software  complexity
– Security is more important than ever (‘Shift Left’ in the SDLC)  complexity
31

32. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Scope vs Organisational Efficiency
• The complexity of environment provisioning moved into dev…
• Amazon’s “two pizza” team size (& similar studies elsewhere)
• How much work can such a team handle whilst focussing on
functionality? What common work can be externalised from
the team, without introducing coupling?
32

33. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Team Topologies
• Introduces concepts of (value) stream-
aligned teams and platform team
• Follows on from research in Accelerate and
elsewhere about cognitive load
33

34. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
What the Platform Team delivers
• Researched, developed & tested platforms (provisioning code)
• Environments are built, self-service, by dev team
• “Opinionated” platforms to save evaluation time by multiple teams
• Feedback loop for continuous improvement
 Internal Developer Platforms
• Product roadmap (with a product manager)
• “Minimum viable platform” (don’t wrap cloud services if no value!)
• Recognising that development teams have choice
34

35. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Recommended Reading
35

36. 36
The Way Ahead for Oracle Platforms

37. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Suggestions for Maintenance
• Long-lived platforms:
– Build, document, test procedures by rebuilding test env (at least) annually
• Stateless apps:
– Push hard for swappable envs (very possible with cloud)
– Don’t patch in place – switch to new
• Cloud envs:
– Invest in automation from start (relatively easy)
– Recreate from TF templates; have a build job that can create, check, delete a test env and run at
least every quarter (cost is low so could run weekly, but someone needs to monitor/fix)
• (APEX: still Work In Progress for us… separate app from data & identity)
37

38. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
General Thoughts
• Beware of making your platform / ops team a dependency on dev productivity
– If not fast enough, or not receptive to new ideas, will just encourage shadow IT
– Loaning someone to build pipelines for dev team will likely end badly
• Don’t re-invent the wheel
– There are lots of organisations doing the same either OS or paid providers – chances are they will
do it better than you can
• Be Open Source-led
– Software engineering has moved on – some of the best IP is now open source
– Be careful who you back though
– If you are really dependent on some key OS products, pay back into community – deepens skills,
improves morale and reduces risk
38

39. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Final Thoughts
• Don’t do any platform automation without CI/CD
– Doesn’t need to be fancy – use same as dev use for apps or cloud
provided (ADO, OCI DevOps, Github Actions, etc)
– (except if Kubernetes, where there’s a bigger debate!)
• Even relatively simple “high level” PaaS will benefit from using
Terraform for standardisation
• May need a mindset change within Ops
39

40. 40
Summary

41. Copyright © 2022, eProseed and/or its affiliates. All rights reserved. | Public
Summary
• The last decade has seen a lot of change in provisioning
• Ansible & Terraform are main contenders
(outside of cloud-native)
• All automation needs maintenance & Operations/BAU buy-in
…otherwise don’t do it!
• DevOps is much more than automation
 new ways of working
41

42. Mobile Process /
Integration
Analytics /
Big Data
Database /
Data
Management
Identity &
Access
Management
Cloud