Herd your chickens: Ansible for DB2 configuration management

1. #IDUG#IDUG
Herd your chickens: Ansible
for DB2 configuration management.
Frederik Engelen
RealDolmen
Session Code: E06
Tuesday, 11 November 2014 | Platform: DB2 for Linux and Unix

2. #IDUG
“If you want to plow a field with
64 chickens instead of one mule, then yes
the unit cost per chicken is less
but herding chickens becomes the new
frontier.”
-- DB2-L discussion

3. #IDUG
Presentation overview
• Introduction to Configuration Management
• Ansible Overview
• Inventory
• Modules
• Playbooks
• Applying for DB2
• Inventory proposal
• Custom module
• Use cases
• Warning: Linux/Unix focused

4. #IDUG
Introduction to Configuration Management
• Who recognises this?
• Or this?
#!/bin/bash
# loop over all databases
for DB in `db2 list db directory | awk '/Database alias/{db = $4}
/Directory entry type/&&/Indirect/{print db}'`; do
db2 +o connect to $DB
db2 <do something>
db2 +o connect reset
done
#!/bin/bash
# stop all instances
for SERVER in dbserver1, dbserver2, db2server3
ssh db2inst1@$SERVER "db2stop force"
done

5. #IDUG
Introduction to Configuration Management
• Who has written similar procedures?

6. #IDUG

7. #IDUG
Introduction to Configuration Management
• We improved by creating scripts
• And improved our scripts
#!/bin/bash
# TODO let's hope this doesn't exist yet
echo "net.ipv4.tcp_fin_timeout = 20" >> /etc/sysctl.conf
sysctl -p
#!/bin/bash
# Change line in sysctl.conf, it’s always there on Linux
grep "net.ipv4.tcp_fin_timeout = 20" /etc/sysctl.conf > /dev/null
if [ ! $? ]; then
sed -i 's/net.ipv4.tcp_fin_timeout = .*/net.ipv4.tcp_fin_timeout =
20/g' /etc/sysctl.conf
sysctl -p
fi

8. #IDUG

9. #IDUG
Almost there…
• Everyone writes their own
• You end up with a ton of scripts
• Not always easily readable
• Different versions scattered everywhere
• What about different values for
net.ipv4.tcp_fin_timeout? Different database names
and instance users?
• Are you sure the right servers have got the right config?
• SSH loops are so much fun…

10. #IDUG
Sneak peak into Ansible
---
- name: Perform the standard Linux configuration for webservers
hosts:
- webservers
tasks:
- name: Lower TCP timeout
sysctl: name=net.ipv4.tcp_fin_timeout value=20 state=present

11. #IDUG
Introduction to Configuration Management
• Describe, create and manage your environment from a central
location
• Code-oriented approach to configuration
• Goals:
• Consistency
• Automation
• Idempotency
• Scalability
• Tools on the market: Puppet, Chef, Salt, CFEngine, Fabric,
Capistrano, Ansible, …

12. #IDUG
Why Ansible?
• It’s one of the younger, leaner members of the family
• Very low learning curve, start in minutes
• Some companies have started migrating from Puppet to
Ansible which I consider a good sign

13. #IDUG
ANSIBLE INTRODUCTION

14. #IDUG
Ansible overview
• Agent-less
• Clients require only minimal software (mainly SSH and
Python)
• Inventory and playbooks are simple text files
• Modules are written in Python

15. #IDUG
Ansible overview - Inventory
• Main file: hosts.cfg
[webservers]
web1.acme.org
web2.acme.org
[databases:children]
db2
informix
[db2]
db2-primary.acme.org
db2-standby.acme.org
[informix]
ids.acme.org
[main-site]
db2-primary.acme.org
web1.acme.org
ids.acme.org
[dr-site]
db2-standby.acme.org
web2.acme.org
webservers
web1 web2
databases
db2
db2-primary db2-standby
informix
ids
dr-sitemain-site

16. #IDUG
Ansible overview - Inventory
• Hosts.cfg is used for selection of target hosts
• Patterns
• Regular expressions
• Set operators (union, except, intersect)
• Support for variables
• Ad-hoc execution examples
# ping all hosts
ansible all –i hosts.cfg –m ping
# get instance status of DB2 hosts
ansible db2 –i hosts.cfg –m command –a 'db2pd –'
# stop DB2 instances on DR site
ansible 'db2:&dr-site' –i hosts.cfg –m command –a 'db2stop force'
# Use regular expression to make powerful selections
ansible '~db2-.*[tst|dev].*' –i hosts.cfg –m ping

17. #IDUG
Ansible overview - Variables
• Describe components in your environment
• Declare at host, group, playbook, command-line, …
• precedence rules apply
• In hosts.cfg file:
[db2]
db2-primary.acme.org db2instance=db2instp
db2-standby.acme.org db2instance=db2insts
[main-site:vars]
default_gateway=10.1.1.1

18. #IDUG
Ansible overview - Variables
• Keep hosts file clean, declare variables in special directories:
• host_vars
• group_vars
• Format is YAML (Yet Another Markup Language)

19. #IDUG
Ansible overview - Variables
• Example
---
# file: host_vars/db2-primary.acme.org
ansible_ssh_user: db2instp # Change Ansible
ansible_ssh_host: db2-primary.acme.org # log-in behaviour
db2_instance_port: 50004
db2_major: 9
db2_minor: 7
db2_fixpack: 3
dbnames:
- "SAPPRD"
- "OPTIMPRD"
regvars:
- {name: "DB2_PARALLEL_IO", value: "*"}
- {name: "DB2AUTH", value: "OSAUTHDB"}

20. #IDUG
Ansible overview - Referencing variables
• Basic variables
• {{ variable }} or ${variable}
• Last style is deprecated but still useful when nesting variables
• Complex variables
• {{ ansible_eth0.ipv4.address }}
• {{ dbnames[0] }}
• Jinja2 filters
• {{ dbname | upper }}
• {{ autostorage_paths | join(", ") }}
• {{ path | basename }}
• {{ instance_port | default(50000) }}
• …

21. #IDUG
Ansible overview - Special variables
• Facts
• Collected for each host at the start of a playbook
• Referenced like any other variable
• Possible to extend with your own
"ansible_distribution": "Ubuntu",
"ansible_distribution_release": "precise",
"ansible_distribution_version": "12.04",
"ansible_fqdn": "db2-primary.acme.org ",
"ansible_hostname": "db2-primary",
"ansible_os_family": "Debian",
"ansible_system": "Linux",
"ansible_env": {
"COLORTERM": "gnome-terminal",
"DISPLAY": ":0",
…
}

22. #IDUG
Ansible overview - Special variables
• Magic variables
• group_names: list of hosts in specific group
• groups: groups this host is part of
• hostvars: variables for another host
{{hostvars[remote_host].db2_instance}}
• Behavioral variables
• ansible_ssh_host
• ansible_ssh_port
• ansible_ssh_user
• ansible_ssh_pass
• ansible_ssh_private_key_file
• …

23. #IDUG
Ansible overview - Modules
• Encapsulate domain-specific functionality
• Executed on remote hosts or locally
• Idempotent (try to avoid changes)
• Library of modules included
• Usually Python, but any scripting language is possible when
rolling your own
• Ad-hoc
$ ansible * –m command –a db2stop
• In playbook
- name: Stop DB2 instance
command: db2stop

24. #IDUG
Ansible overview - Interesting modules
• Package modules (yum, apt, zypper, ...)
- name: Install prereq packages for DB2
yum: name={{item}} state=present
with_items:
- libaio
- compat-libstdc++-33
• Command (command, shell, script, …)
- name: install DB2 Software
command: ~/installdb2.sh {{db2_major}} {{db2_minor}} {{db2_fixpack}}
creates={{ db2_install_dir }}/bin/db2
when: db2_major is defined
- name: Query log chain ID
shell: db2pd -logs -db {{dbname}} | awk '/Log Chain ID/{print $4}'
• Lineinfile
- name: declare the TMPDIR variable in userprofile
lineinfile: dest="~/sqllib/userprofile" regexp="export TMPDIR.*"
line="export TMPDIR=${INSTHOME}/db2temp"

25. #IDUG
Ansible overview - Interesting modules
• Mount
- name: Mount software repository
mount: name=/mnt/repo fstype=nfs src={{repository_server}}:/repo
state=mounted
• File modules (file, copy, template, …)
- name: Push DB2 instance response file
template:
src=/.../{{db2_version}}_{{instance_type}}_aese.rsp.j2
dest=/root/db2_{{db2environment}}_aese.rsp
when: instance_type is defined and db2_major is defined
#v10.5_default_aese.rsp.j2
FILE = /opt/IBM/db2/V{{db2_major}}.{{db2_minor}}_FP{{db2_fixpack}}
LIC_AGREEMENT = ACCEPT
INSTANCE = inst1
inst1.TYPE = ese
inst1.NAME = {{db2instance}}
inst1.GROUP_NAME = {{db2instance_group}}
inst1.HOME_DIRECTORY = /data/db2/{{db2instance}}
inst1.PORT_NUMBER = {{db2_instance_port | default(50004)}}

26. #IDUG
Ansible overview - Interesting modules
• Database modules
- name: Drop MySQL database once migrated to DB2
- mysql_db: name=slowdb state=absent
• Cron
- name: Configure runstats script to run at 23:00
cron: name="runstats" hour="23" job="/bin/my_runstats.sh {{item}}"
with_items: dbnames
• Other
• user/group
• lvg/lvol/
• service
• subversion/git
• mail
• uri
• …

27. #IDUG
Ansible overview - Playbooks
• Configuration, deployment, and orchestration language
• Describe configuration policy or operational steps
- hosts: webservers
vars:
http_port: 80
max_clients: 200
remote_user: root
serial: 1
tasks:
- name: ensure apache is at the latest version
yum: pkg=httpd state=latest
- name: write the apache config file
template: src=/srv/httpd.j2 dest=/etc/httpd.conf
notify:
- restart_apache
- name: ensure apache is running and started on boot
service: name=httpd enabled=yes state=started
handlers:
- name: restart_apache
service: name=httpd state=restarted

28. #IDUG
Ansible overview - Playbook execution
• Scenario:
• Only one webserver in the farm has been upgraded and reconfigured.
• The senior administrator uses Ansible to rectify this oversight of his
trainee.

29. #IDUG
Ansible overview - Playbook execution
# ansible-playbook –i hosts.cfg playbooks/web-config.yml –v
PLAY [webservers]
*************************************************************
GATHERING FACTS
***************************************************************
ok: [web1.acme.org]
ok: [web2.acme.org]
TASK: [Make sure httpd is at the latest version]
******************************
ok: [web1.acme.org] => {"changed": false, "msg": "", "rc": 0, "results":
["All packages providing httpd are up to date"]}
changed: [web2.acme.org] => {"changed": true, "msg": "", "rc": 0, "results":
["Loaded plugins:
--%<--
Updated:n httpd.x86_64 0:2.2.15-30.el6.centos
nnDependency Updated:n httpd-tools.x86_64 0:2.2.15-30.el6.centos
openssl.x86_64 0:1.0.1e-16.el6_5.7 nnComplete!n"]}

30. #IDUG
Ansible overview - Playbook execution
TASK: [Push httpd.conf]
*******************************************************
ok: [web1.acme.org] => {"changed": false, "gid": 0, "group": "root", "mode":
"0644", "owner": "root", "path": "/tmp/httpd.conf", "size": 6, "state":
"file", "uid": 0}
changed: [web2.acme.org] => {"changed": true, "dest": "/tmp/httpd.conf",
"gid": 0, "group": "root", "md5sum": "8b5cee2ea4d4ae8ee17fda49946c13df",
"mode": "0644", "owner": "root", "state": "file", "uid": 0}
TASK: [Make sure Apache is running and started on boot]
*******************************************************
ok: [web1.acme.org] => {"changed": false, "name": "httpd", "state":
"started"}
changed: [web2.acme.org] => {"changed": true, "name": "httpd", "state":
"started"}
NOTIFIED: [restart_httpd]
*****************************************************
changed: [web2.acme.org] => {"changed": true, "name": "httpd", "state":
"started"}
PLAY RECAP
********************************************************************
web1.acme.org : ok=4 changed=0 unreachable=0 failed=0
web2.acme.org : ok=5 changed=4 unreachable=0 failed=0

31. #IDUG
Ansible overview - Playbook task options
• Register (store output of command)
- name: get tablespace list
db2sql: dbname="{{dbname}}" sql="select tbspace from syscat.tablespaces"
register: tablespaces
• Loops
- name: remove empty space from tablespaces
db2sql: dbname="{{dbname}}" sql="db2 alter tablespace {{item}} reduce"
with_items: tablespaces
- name: update instance registry
command: db2set {{item.name}} {{item.value}}
with_items:
- {name: "DB2_PARALLEL_IO", value: "*"}
- {name: "DB2AUTH", value: "OSAUTHDB"}
• Other loops: with_nested, with_sequence, …

32. #IDUG
Ansible overview - Playbook task options
• Conditionals
when: db2_version is defined
when: ansible_os_family == 'AIX'
when: sqloutput.rc != 1
• Output handling
• Ignore_errors
- name: Stop database, ignore errors, we'll kill it later
command: db2stop force
ignore_errors: yes
• Failed_when/changed_when
- name: Execute update
db2sql: dbname="{{dbname}}" sql="update x where y = z"
register: sqlout
failed_when: sqlout.rc > 2
changed_when: sqlout.rc <> 1

33. #IDUG
Ansible overview - Playbook task options
• Delegate_to / tags
- name: Execute sql on secondary host
db2sql: dbname={{ dbname }} sql="{{ sql }}"
delegate_to: ${instance_host_alternate}
- name: Disabling notifications in Nagios for current host
nagios: action=silence services=all host=${ansible_hostname}
delegate_to: ${nagios_srv}
tags: nagios
- name: Create the ODR server (execute on Deployment Mgr)
command: /opt/IBM/WebSphere/AppServer_v${was_version}/bin/wsadmin.sh
${node_nodename} odr ${server_name}
creates=/…/${node_nodename}/servers/${server_name}
delegate_to: ${dmgr_ansible_host}
when_string: ${server_type} == 'odr'
tags:
- odr
- install

34. #IDUG
Ansible overview - Running playbooks
ansible-playbook –i <hosts file> <playbook>
• --list-hosts
Don't execute but show target hosts
• --limit / -l
Limit target hosts to specified list
• --extra-vars / -e
Supply variables command-line
• --tags / -t
Run only specified tags
• --skip-tags
Skip certain tags
• --step
Ask verification for each step
• -v / -vv / -vvv / -vvvv
Verbose to very verbose

35. #IDUG
APPLYING FOR DB2

36. #IDUG
Applying for DB2 – Inventory proposal
# hosts/hosts_prd.cfg
[db2-instances:children]
db2-db2isap
[db2-db2isap]
sapprd-primary_db2isap
sapprd-standby_db2isap
db-sapprd-prd
[db-all:children]
db-sap
[db-sap]
db-sapprd-prd dbname=SAPPRD
instance_host=sapprd-primary_db2isap
instance_host_alternate=sapprd-standby_db2isap

37. #IDUG
Applying for DB2 – Inventory proposal
# hosts/group_vars/db2-instances
ansible_ssh_user: "{{ db2instance }}"
db2instance_home_dir: "/home/{{ db2instance }}"
# hosts/group_vars/db2-db2isap
db2instance: db2isap
db2instance_group: db2iasap
db2fenced: db2fsap
db2fenced_group: db2fasap
ansible_ssh_user: db2isap
db2environment: sap

38. #IDUG
Applying for DB2 – Inventory proposal
# hosts/host_vars/sapprd-primary_db2isap
ansible_ssh_host: sapprd-primary.acme.org
db2_instance_port: 50004
db2_major: 10
db2_minor: 1
db2_fixpack: 3
dbnames:
- "SAPPRD“
# hosts/group_vars/db-sap
use_tsm: True
db_cfg:
- {name: "LOGBUFSZ", value: "2048"}
- {name: "SOFTMAX", value: "50"}

39. #IDUG
Applying for DB2 – Custom module
• Module db2sql : allows execution of statements to a database,
optionally reading from file (full script in speaker's notes)
argument_spec = dict(
dbname=dict(required=True),
schema=dict(),
sql=dict(),
--%<--
)
dbname = module.params['dbname']
schema = module.params['schema']
sql = module.params['sql']
file = os.path.expanduser(
os.path.expandvars(xstr(module.params['file'])))

40. #IDUG
Applying for DB2 – Custom module
# Connect to database, optionally specifying user
if user:
(rc, out, err) = module.run_command("db2 connect to %s user %s using %s" %
(dbname, user, password))
else:
(rc, out, err) = module.run_command("db2 connect to %s" % (dbname))
# Evaluate any non-0 return codes from connect command
# For HADR Standby databases, send Skipped
if rc <> 0:
words = out.split()
# standby database
if words[0] == "SQL1776N":
module.exit_json(skipped=True, msg="Ignored standby database")
else:
module.fail_json(rc=rc, msg=out)

41. #IDUG
Applying for DB2 – Custom module
# Execute either SQL text or file (should be present)
if sql:
(rc, out, err) = module.run_command('db2 -xp "%s"' % (sql))
elif file:
(rc, out, err) = module.run_command('db2 -x -tvf %s' % (file))
# Evaluate output
# Return separate output lines
if rc <> 0:
module.fail_json(rc=rc, msg=out.split("n"))
else:
module.exit_json(rc=rc, changed=True, msg=[ line.strip() for line in
out.split("n") if line.strip()<>""])

42. #IDUG
Applying for DB2 – Custom module
• A database is not a host, ad hoc execution is impossible
• we need to execute this via a playbook.
- hosts: db-all
gather_facts: no
tasks:
- name: Execute sql on primary host
db2sql: dbname={{ dbname }} sql="{{ sql }}"
delegate_to: ${instance_host}
register: sqloutput
failed_when: sqloutput.rc > 2
- name: Execute sql on secondary host
db2sql: dbname={{ dbname }} sql="{{ sql }}"
delegate_to: ${instance_host_alternate}
when: sqloutput|skipped # only execute when first server is standby
register: sqloutput
failed_when: sqloutput.rc > 2

43. #IDUG
Applying for DB2 – Custom module
$ ansible-playbook -i hosts/db_hosts_dev.cfg playbooks/db-execsql.yml -l db-
*-tst-* -e "sql="select min(stats_time) from syscat.tables where stats_time
> '1970-01-01-00.00.00.000000'"" –v
PLAY [db-all]
*****************************************************************
TASK: [Execute sql on primary host]
*******************************************
changed: [db-abc-tst-b] => {"changed": true, "failed": false,
"failed_when_result": false, "msg": ["2013-03-08-14.49.25.738562"], "rc": 0}
changed: [db-def-tst-a] => {"changed": true, "failed": false,
"failed_when_result": false, "msg": ["2013-02-18-19.25.39.656577"], "rc": 0}
TASK: [Execute sql on secondary host]
*****************************************
skipping: [db-abc-tst-a]
skipping: [db-def-tst-b]
PLAY RECAP
********************************************************************
db-abc-tst-a : ok=1 changed=1 unreachable=0 failed=0
db-def-tst-b : ok=1 changed=1 unreachable=0 failed=0

44. #IDUG
Applying for DB2 – Custom module - Bash
• Example of heading for BASH script (ex. db2state)
#!/bin/bash
# The name of a file containing the arguments to the module is
# given as first argument. Source the file to load the variables:
source ${1}
if [ -z "$dbname" ]; then
echo 'failed=True msg="Module needs dbname= argument"'
fi
if [ -z "$state" ]; then
echo 'failed=True msg="Module needs state= argument"'
fi
if [[ $state == "hadr_primary" ]]; then
…

45. #IDUG
Applying for DB2 – HADR use-case
• Three additional modules, all trivial to implement
• db2dbcfg: change DB CFG parameters
• db2backup: perform backup/restore operations
• db2state: controls database state (deactive, active, hadr_primary, ...)
• Add one parameter to your DB definition
[db-sap]
db-sapprd-prd dbname=SAPPRD instance_host=sapprd-primary_db2isap
instance_host_alternate=sapprd-standby_db2isap hadr_service=50040
• Use the information available in inventory
- name: Set Primary database cfg
db2dbcfg: dbname={{dbname}} param={{item.param}} value={{item.value}} connect=False
delegate_to: ${instance_host}
with_items:
-{ param:"HADR_LOCAL_HOST", value:{{hostvars[instance_host].ansible_ssh_host}}"}
-{ param:"HADR_REMOTE_HOST", value:{{hostvars[instance_host_alternate].ansible_ssh_host}}"}
-{ param:"HADR_REMOTE_INST", value:"{{db2instance}}"}
-{ param:"HADR_LOCAL_SVC", value:"{{hadr_service}}"}
-{ param:"HADR_TIMEOUT", value:"{{hadr_timeout|default(120)}}"}
- ...

46. #IDUG
Wrapping up
• Strong points
• Scales easily to any number of servers
• Short payback period
• Quality of environments has increased
• I can't imagine going back to my old way of working
• Weak points
• Playbook language could be more powerful (loops, calculations, …)
• Variable substitution when using delegate_to isn't working well
• Multi-line command output can be difficult to read
• Serial doesn't define a rolling window but a batch size
• All actions are done per step – a slow server holds back a rollout
• Need to define database-instance relation twice

47. #IDUG
Documentation
• http://www.ansible.com/home
• https://github.com/ansible
• http://www.slideshare.net/johnthethird/ansible-presentation-
24942953
• https://speakerdeck.com/jpmens/ansible-an-introduction

48. #IDUG#IDUG
Frederik Engelen
RealDolmen
frederik.engelen@realdolmen.com
Session Code: E06
Herd your chickens: Ansible for DB2
configuration management
Please fill out your session
evaluation before leaving!