The document describes the author's journey of automating various Oracle database administration tasks using Python. Some key points: 1) The author started using Python to automate routine database maintenance tasks like log file management and backups that were previously done manually or via scripts. 2) Over time, the author applied object-oriented principles in Python to develop reusable modules and classes to further standardize and simplify administration of multiple databases. 3) Large projects like database migrations were made possible by building on the Python codebase, applying concepts like packages and modules to organize the code into a reusable framework. Proper documentation of the code was also emphasized.

1. The Pythonic Way crosses
Oracle‘s Exadata
by rainer schuettengruber

2. about me
• IT employee since 1998
• main focus on Oracle databases
• various positions as Oracle DBA(DMA), developer, devops
• currently employed as Exadata Administrator

3. itinerary

4. continous integration
• version control
• automated build
• unit tests
• automated unit tests
• automated deployment

5. pre python era
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• heterogeneous environment
• Oracle on AIX, Linux on VMware, Exadata
• Oracle Version 10.2.0.4, 11.2.0.1,11.2.0.2, 11.2.0.3, 11.2.0.4
• Oracle Cloud Control 11g on Linux
• Oracle Cloud Control 12c on AIX
• backup based on Cloud Control jobs
• backup based on TSM scheduler
• backup based on ksh scripts
• .. to take arms against a sea of troubles ..

6. pre python era
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• automation and standardisation are siamese twins
• standardisation by migrating all databases to Exadata
• migrated on database after the other
• took almost a year
• set the stage for automation

7. the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
oracle@disguised:misc (TEST1) > ./tracemaint.sh -h
usage: tracemaint [-h] -d DAYS -m MAXLOGSIZE [-s SID] [-k] [--debug]
tidy up database trace and log files
optional arguments:
-h, --help show this help message and exit
-d DAYS, --days DAYS number of days that logs/traces need to be kept
-m MAXLOGSIZE, --maxlogsize MAXLOGSIZE
threshold in MB, specifying if a log files needs
rotation
-s SID, --sid SID database SID, if omitted all databases configured in
/etc/oratab are considered
-k, --keeptempfiles keep files generated during logrotation, use this for
debugging purposes only
--debug log debug output

8. the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• Python comes with Linux, however
• versions, depending on the distribution, between 2.4 – 2.7
• changing installation might do harm, especially on Exadata
• dedicated installation under /opt in accord with FHS
• moreover Exadata does not support customized RPM‘s
• building from source and installation via tarball

9. the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/bash/osSetup>./build_python.sh
usage : build_python.sh -p <python source tarball> -r -c
-r .. remove existing installation
-c .. create tarball from the installation
disguised:/root/svnRepo/scripts/bash/osSetup>
disguised:/root/svnRepo/scripts/bash/osSetup>./install_python.sh
usage : install_python.sh -t <python tarball>
disguised:/root/svnRepo/scripts/bash/osSetup>

10. the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• SVN rather obvious
• used for deployments in combination with make files
• installation under /opt requires command line wrappers
• which comes in handy for cx_Oracle

11. the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
[root@disguised traceMaintenance]# cat Makefile
INSTALL_DIR = /home/oracle/scripts/misc
wrapperScripts = $(INSTALL_DIR)/tracemaint.sh
all: $(wrapperScripts)
$(INSTALL_DIR)/%.sh: %.sh
install -o oracle -g oinstall -m 755 $? $@
[root@disguised traceMaintenance]# cat tracemaint.sh
#!/bin/bash
export LD_LIBRARY_PATH=/opt/oracle/product/12.2.0.1/instantclient
unset NLS_LANG
/opt/python/bin/tracemaint "$@"
[root@disguised traceMaintenance]#

12. the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
import logging
from logging.handlers import RotatingFileHandler
import argparse
import sys
import os
import subprocess
import glob
import re
from datetime import datetime
import cx_Oracle

13. OOP appears on the stage
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• SAP requires a dedicated toolset for database maintenance
• basically 4 compenents
• separate for each database
• installation and patching rather tedious for 20+ databases
• implementation reminiscent of the DRY principle

14. OOP appears on the stage
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

15. OOP appears on the stage
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• one python file per class, used modules without recognizing it
• patching became a matter of minutes as opposed to hours
• off the beaten script track
• peer cluster nodes taken into consideration

16. OOP appears on the stage
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
import shutil
from pwd import getpwuid
from pexpect import pxssh
import paramiko
from paramiko import SSHClient
from scp import SCPClient

17. the master‘s class
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• became aware of python‘s modul concept
• formed the idea of bundling modules in a package that provides console
scripts
• amongst improving python skills, set the stage for a clean and reusable code
base
• good riddance bash
• well established principles in software development won‘t do harm in
operations

18. the helping hand
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• Cloud Control migration/consilidation had become inevitable
• upgrade path for only one of the two systems available
• implies that 50 per cent of the monitored databases need to be configured
manually, tedious and prone to error
• further use of configuration scripts justify the effort and contribute to
standardisation
• in essence a wrapper for Cloud Control‘s python based emcli utility
• migration/consolidation a matter of hours
• adding a database became a matter of seconds

19. exadata migration on steriods
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• existing Exadata systems had done their duty
• essentially the same migration path as for the migration from AIX to Exadata
• however, due to strigent licence terms only a parallel phase of 3 months in
sharp contrast to 1 year
• further cut down to 4 weekends due to staff availability
• no reason to despair since python is around ...
• built on top of Oracle‘s dataguard, cutting down downtime to a couple of
minutes

20. exadata migration on steriods
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• typed python code instead of oracle commands on the weekends
• structured code into modules
• created a package to ease deployment
• manual build with python setup.py bdist_wheel
• implemented console scripts
• toyed with the idea to use ant as build tool

21. exadata migration on steriods
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/python>find rlb -maxdepth 1
rlb
rlb/common
rlb/__init__.py
rlb/__pycache__
rlb/occ
rlb/sap
rlb/tsm
rlb/db
rlb/dg
rlb/dp
rlb/occut
rlb/dput
rlb/commonut
rlb/exa
rlb/tmut
rlb/dbut
rlb/dgut
rlb/tm

22. exadata migration on steriods
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
from setuptools import setup
from setuptools import find_packages
setup(
name='rlb',
version='6.8.1',
author='rainer schuettengruber',
author_email='rainer.schuettengruber@rrz.co.at',
packages=find_packages(),
package_data={
'': ['*.template'],
},
entry_points={
'console_scripts': [
'massdeploy=rlb.common.massdeploy:main_for_script',
'tracemaint=rlb.tm.tracemaint:main_for_script',

23. automatisation of the automatisation
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• automated database setup
• build and deployment appeared to be rather tedious
• opted for ant
• faced the same issues as already discussed for the pyhton installation
• addressed by installing ant under /opt
• build specific environment variables by means of the build.env script

24. automatisation of the automatisation
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
[root@disguised python]# source build.env
PATH :
/opt/python/bin:/opt/ant/bin:/u01/app/12.1.0.2/grid/bin:/usr/local/sbin:/usr/local/bin:/sbi
n:/bin:/usr/sbin:/usr/bin:/root/bin
LD_LIBRARY_PATH : /opt/oracle/product/12.2.0.1/instantclient
ANT_ARGS : -emacs -logger org.apache.tools.ant.NoBannerLogger
PYTHON_PATH : /root/svnRepo/scripts/python
[root@disguised python]# ant -p
Buildfile: /root/svnRepo/scripts/python/build.xml
Build and deploy rlb's pyhton modules
Main targets:
clean clean up all files created by either dist or docs target
dist build wheel
distclean clean up files created by the dist target
install install wheel on this host
Default target: usage
[root@disguised python]#

25. prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• given a SLOC of 3600, documentation became a necessisty
• opted for sphinx
• created UML diagrams with pyreverse which is part of the pylint package
• added ant target docs

26. prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
<target name="docs"
description="generate sphinx documentation, including pylint and pyreverse
output"
depends="clean">
<!-- create sphinx-apidoc -->
<exec executable="${python.binpath}/sphinx-apidoc">
<arg value="-f" />
<arg value="-o" />
<arg value="${doc.sphinx.source.dir}" />
<arg value="${package.dir}" />
</exec>
<!-- creating UML diagrams -->
<exec executable="${python}" >
<arg value="builduml.py" />
</exec>
<!-- render html docu -->
<exec executable="/usr/bin/make">
<env key="LD_LIBRARY_PATH" path="/opt/oracle/product/12.2.0.1/instantclient" />
<arg value="html" />
</exec>

27. prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
<!-- create pylint output -->
<mkdir dir="${doc.pylint.dir}" />
<echo message="writing pylint's output to ${doc.pylint.dir}/packages_pylint.txt" />
<exec executable="${pylint}"
dir="${package.dir}"
output="${doc.pylint.dir}/packages_pylint.txt" >
<arg value="--rcfile=pylint.rc" />
<arg value="-f" />
<arg value="text" />
<arg value="rlb" />
</exec>
<!-- deploy documentation -->
<delete>
<fileset dir="${doc.deploy.dir}" includes="**/*.*" />
</delete>
<echo message="deploying docu to ${doc.deploy.dir} ..." />
<copy todir="${doc.deploy.dir}">
<fileset dir="${doc.sphinx.build.dir}/html">
<include name="**/*" />
</fileset>
</copy>
</target>

28. prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

29. prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

30. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• functionality shared by all modules has been refactored into the common sub
package
• breaking functionality within the common sub package results in breaking the
whole code base
• unit tests absolutely vital
• additional sub package with suffix ut
• one test file for each module, prefixed with test_

31. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
[root@disguised rlb]# find common* -name '*.py'
common/os.py
common/template.py
common/__init__.py
common/oradb.py
common/massdeploy.py
commonut/test_oradb.py
commonut/__init__.py
commonut/test_template.py
commonut/test_os.py
commonut/test_massdeploy.py
[root@disguised rlb]#

32. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• version control, ant build and unit tests have been implemented
• which sets the stage for a decent jenkins installation
• Subversion plugin for obvious reasons
• Cobertura plugin aimed at automatic unit tests
• SLOCCount plugin in case somebody asks
• Violations plugin since pylint reports are already in place
• Static analysis collector plugin-in as a dependency

33. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/python>cat requirements_test.txt
astroid>=1.5.3
autopep8>=1.3.2
coverage>=4.4.1
idna>=2.1
isort>=4.2.5
lazy_object_proxy>=1.2.2
logilab_common>=1.3.0
mccabe>=0.5.2
pep8>=1.7.0
py>=1.4.34
pyenchant>=1.6.8
pygount>=0.9
pylint>=1.7.2
pytest>=3.2.1
pytest-cov>=2.5.1
sphinx_pyreverse>=0.0.12
wrapt>=1.10.8
Pygments>=2.2

34. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
<target name="code-metrics" description="gather code metrics">
<!-- obtain source code lines by using pygount -->
<echo message="writing pygount's output to cloc.xml" />
<exec executable="${pygount}" dir="../..">
<arg value="--format=cloc-xml" />
<arg value="--suffix=py,sh,sql,xml,pks,pkb,php,java" />
<arg value="--out=cloc.xml" />
</exec>
<!-- assess code quality with pylint -->
.
.
<!-- evaluate code quality with pep8 -->
<echo message="writing pep8's output to pep8.out" />
<exec executable="${pep8}"
dir="${package.dir}"
output="../../pep8.out">
<arg value="--filename=*.py" />
<arg value="--max-line-length=99" />
<arg value="." />
</exec>
</target>

35. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/python>ant -p
Buildfile: /root/svnRepo/scripts/python/build.xml
Build and deploy rlb's pyhton modules
Main targets:
all build wheel and documentation
clean clean up all files created by either dist or docs target
code-metrics gather code metrics
dist build wheel
distclean clean up files created by the dist target
docs generate sphinx documentation, including pylint and pyreverse output
install install wheel on this host
jenkins build targets required by continuous integration
Default target: usage
disguised:/root/svnRepo/scripts/python>
<target name="jenkins" description="build targets required by continuous integration"
depends="clean, code-metrics, dist" />

36. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

37. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

38. a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

39. a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• given that jenkins and unit tests are in place, test automatisation appears to
be quite obvious
• isolated build by means of venv
• documented required modules, which comes in handy when
upgrading/installing python
• run test suite by means of pytest
• added coverage reports

40. a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
<target name="test" description="run test suite" depends="venv">
<exec executable="${venv.python}" failonerror="true">
<arg value="setup.py" />
<arg value="install" />
</exec>
<exec executable="/home/jenkins/scripts/pre_unittest.sh" />
<exec executable="${venv.pytest}">
<!-- No failonerror="true" because on failed tests, pytest exits with 1. -->
<env key="LD_LIBRARY_PATH" path="/opt/oracle/product/12.2.0.1/instantclient" />
<arg value="--junit-xml=unittest.xml" />
<arg value="--cov=${package.dir}" />
<arg value="--cov-report=html" />
<arg value="--cov-report=xml" />
<arg value="--ignore=${venv.dir}" />
<arg value="--ignore=./rlb/commonut/test_cp1252_encoding.txt" />
<arg file="${package.dir}" />
</exec>
<exec executable="/home/jenkins/scripts/post_unittest.sh" />
</target>

41. a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

42. a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

43. a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• complete test coverage would imply a dedicated exadata environment
• a case for unittest.mock
• as a precondition Exadata/database related calls need to be separated, which
has not been considered in the initial design
• might be subject to another talk
• however, the foundations are laid
• played a vital role during the upgrade vom python 3.5 to 3.6

44. the finishing touch
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• due to stringent security policies no access to pypi.python.org
• has been circumvented by using pip install http://...
• however, resolving dependencies appeared to be a bit tedious
• so pypiserver it is
• enables continous deployment

45. the finishing touch
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
[Unit]
Description=pypi-server
After=network.target
[Service]
Type=simple
ExecStart=/opt/python/bin/pypi-server -p 8081 --disable-fallback -v --log-file
/var/log/pypi-server.log -u /var/www/html/pypi
[Install]
WantedBy=multi-user.target

46. the finishing touch
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/python>ant -p
Buildfile: /root/svnRepo/scripts/python/build.xml
Build and deploy rlb's pyhton modules
Main targets:
all build wheel and documentation
clean clean up all files created by either dist or docs target
code-metrics gather code metrics
deploy install wheel on production hosts
dist build wheel
distclean clean up files created by the dist target
docs generate sphinx documentation, including pylint and pyreverse output
install install wheel on this host
jenkins build targets required by continuous integration
test run test suite
venv setup virtual Python environment
Default target: usage
disguised:/root/svnRepo/scripts/python>

47. where do we go now?
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• deal with testing issues
• become proactive by forecasting capacity and load with appropriate models
• tweak pylint/pep8
• improve on documentation

48. questions and answers
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17