This document summarizes a presentation about hacking Ansible to make it more customizable. It discusses how Ansible's plugin system allows it to be extended through modules, filters, lookups, callbacks and caches. Examples are provided of extending Ansible's core functionality by modifying files in the lib directory and writing custom plugins. The presentation also outlines how Ansible's object model works and provides an overview of its growth in modules and plugins over time.

1. Hacking Ansible
make it do more
ansiblefest 10/2014

2. Who am I?
programmer
DBA
system administrator
tinkerer
devops
tech-janitor
____________
< irc lurker >
------------
, ,
/( )`
___ / |
/- _ `-/ '
(// /
/ / | `
O O ) / |
`-^--'`< '
(_.) _ ) /
`.___/` /
`-----' /
<----. __ / __
<----|====O)))==) ) /====
<----' `--' `.__,'
| |
/
______( (_ / ______
,' ,-----' |
`--{__________) /

3. ________________
/ It runs a TASK
on a HOST /
----------------
^__^
(oo)_______
(__) )/
||----w |
|| ||
What is ansible?
● Configuration management?
● Automation platform?
● Release management?
● Orchestration system?

4. The Guts: ansible internal objects
● inventory: defines my targets
● runner: actually does the work
● connection: how to get to my targets
● playbook: all plays in current invocation
○ play
■ host: where to do it
■ task: “it” aka “what to do”
● callback: show me what was done

5. Example hack - the core
--- a/lib/ansible/playbook/__init__.py
+++ b/lib/ansible/playbook/__init__.py
@@ -346,6 +346,8 @@ class PlayBook(object):
run_hosts=hosts
)
+ runner.module_vars.update({'play_hosts': hosts})
+
if task.async_seconds == 0:
results = runner.run()
else:

6. Example hack - the core
● 10 second patch to add play_hosts
● 3h to find correct spot to patch
● learned a lot about playbook/host iteration
● I did not take “serial:” into account
● runner rewrite in progress

7. The
plugin
system

8. PluginLoader
...
lookup_loader = PluginLoader(
'LookupModule',
'ansible.runner.lookup_plugins',
C.DEFAULT_LOOKUP_PLUGIN_PATH,
'lookup_plugins'
)
vars_loader = PluginLoader(
'VarsModule',
'ansible.inventory.vars_plugins',
C.DEFAULT_VARS_PLUGIN_PATH,
'vars_plugins'
)
lib/ansible/utils/plugins.py
__________________
/ all plugins
except inventory /
------------------
,__,
(oo)____
(__) )
||--|| *

9. Plugins
● Library: host tasks/actions/modules
● Action: master side tasks/actions/modules
● Cache: fact caching
● Callback: play output
● Connection: host connections

10. Plugins
● Shell: what shell to use to execute tasks
● Lookup: master side info lookup
● Vars: variable imports
● Inventory: aside from inventory scripts
● Filter: jinja2 filters for data modification
● Doc Fragment: shared docs for library

11. Library/Action - May 2013 (72)
add_host,debug,get_url,mount,postgresql_user,slurp,apt,
django_manage,git,mysql_db,rabbitmq_parameter,
subversion,apt_key,easy_install,group,mysql_user,
rabbitmq_plugin,supervisorctl,apt_repository,ec2,group_by,
nagios,rabbitmq_user,svr4pkg,assemble,ec2_facts,hg,ohai,
rabbitmq_vhost,sysctl,async_status,ec2_vol,ini_file,opkg,
raw,template,async_wrapper,facter,libr,pacman,script,uri,
authorized_key,fail,lineinfile,pause,seboolean,user,
cloudformation,fetch,lvol,ping,selinux,virt,command,file,
macports,pip,service,wait_for,copy,fireball,mail,pkgin,setup,
yum,cron,gem,mongodb_user,postgresql_db,shell,zfs

12. Library/Action - October 2014 (175)
a10_server,a10_service_group,a10_virtual_server,accelerate,acl,add_host,airbrake_deployment,alternatives,apache2_module,
apt,apt_key,apt_repository,apt_rpm,assemble,assert,async_status,at,authorized_key,azure,bigip_facts,bigip_monitor_http,
bigip_monitor_tcp,bigip_node,bigip_pool,bigip_pool_member,bigpanda,boundary_meter,bzr,campfire,capabilities,cloudformation,
command,composer,copy,cpanm,cron,datadog_event,debconf,debug,digital_ocean,digital_ocean_domain,digital_ocean_sshkey,
django_manage,dnsimple,dnsmadeeasy,docker,docker_image,easy_install,ec2,ec2_ami,ec2_ami_search,ec2_asg,ec2_eip,
ec2_elb,ec2_elb_lb,ec2_facts,ec2_group,ec2_key,ec2_lc,ec2_metric_alarm,ec2_scaling_policy,ec2_snapshot,ec2_tag,ec2_vol,
ec2_vpc,ejabberd_user,elasticache,facter,fail,fetch,file,filesystem,fireball,firewalld,flowdock,gc_storage,gce,gce_lb,gce_net,
gce_pd,gem,get_url,getent,git,github_hooks,glance_image,group,group_by,grove,hg,hipchat,homebrew,homebrew_cask,
homebrew_tap,hostname,htpasswd,include_vars,ini_file,irc,jabber,jboss,jira,kernel_blacklist,keystone_user,layman,
librato_annotation,lineinfile,linode,lldp,locale_gen,logentries,lvg,lvol,macports,mail,modprobe,mongodb_user,monit,mount,mqtt,
mysql_db,mysql_replication,mysql_user,mysql_variables,nagios,netscaler,newrelic_deployment,nexmo,nova_compute,
nova_keypair,npm,ohai,open_iscsi,openbsd_pkg,openvswitch_bridge,openvswitch_port,opkg,osx_say,ovirt,pacman,pagerduty,
pause,ping,pingdom,pip,pkgin,pkgng,pkgutil,portage,portinstall,postgresql_db,postgresql_privs,postgresql_user,
quantum_floating_ip,quantum_floating_ip_associate,quantum_network,quantum_router,quantum_router_gateway,
quantum_router_interface,quantum_subnet,rabbitmq_parameter,rabbitmq_plugin,rabbitmq_policy,rabbitmq_user,rabbitmq_vhost,
raw,rax,rax_cbs,rax_cbs_attachments,rax_cdb,rax_cdb_database,rax_cdb_user,rax_clb,rax_clb_nodes,rax_dns,rax_dns_record,
rax_facts,rax_files,rax_files_objects,rax_identity,rax_keypair,rax_meta,rax_network,rax_queue,rax_scaling_group,
rax_scaling_policy,rds,rds_param_group,rds_subnet_group,redhat_subscription,redis,replace,rhn_channel,rhn_register,riak,
rollbar_deployment,route53,rpm_key,s3,script,seboolean,selinux,service,set_fact,setup,shell,slack,slurp,sns,stackdriver,stat,
subversion,supervisorctl,svr4pkg,swdepot,synchronize,sysctl,template,twilio,typetalk,ufw,unarchive,uri,urpmi,user,virt,
vsphere_guest,wait_for,win_feature,win_get_url,win_group,win_msi,win_ping,win_service,win_stat,win_user,xattr,yum,
zabbix_maintenance,zfs,zypper,zypper_repository

13. Filters: Jinja2’ism
● best way to change data
● chainable pipes
● simple to expand
● can hide complexity
__________________________
< [hello, world]|join(‘ ‘) >
--------------------------
^__^
(oo)_______
(__) )/
||----w |
|| ||

14. Example hack - filter
import random
def randomize_list(mylist):
random.shuffle(mylist)
return mylist
...
def filters(self): # this maps filter names to functions
return {
‘shuffle’: randomize_list,
...
}
lib/ansible/runner/filter_plugins/core.py

15. Example hack - filter
- hosts: all
connection: local
gather_facts: false
vars:
- mylist: [1,2,3,4,5]
tasks:
- debug: msg=”{{item}}”
with_items: “{{mylist|shuffle}}”

16. Example hack - filter
#>ansible-playbook test.yml -i ‘localhost,’
__________________________
< TASK: debug msg={{item}} >
--------------------------
ok: [localhost] => (item=1) => {
"item": 1,
"msg": "1"
}
ok: [localhost] => (item=3) => {
"item": 3,
"msg": "3"
}
ok: [localhost] => (item=4) => {
"item": 4,
"msg": "4"
}
ok: [localhost] => (item=5) => {
"item": 5,
"msg": "5"
}
ok: [localhost] => (item=2) => {
"item": 2,
"msg": "2"
}
__________________________
< TASK: debug msg={{item}} >
--------------------------
ok: [localhost] => (item=3) => {
"item": 3,
"msg": "3"
}
ok: [localhost] => (item=2) => {
"item": 2,
"msg": "2"
}
ok: [localhost] => (item=5) => {
"item": 5,
"msg": "5"
}
ok: [localhost] => (item=2) => {
"item": 2,
"msg": "2"
}
ok: [localhost] => (item=1) => {
"item": 1,
"msg": "1"
}
__________________________
< TASK: debug msg={{item}} >
--------------------------
ok: [localhost] => (item=2) => {
"item": 2,
"msg": "2"
}
ok: [localhost] => (item=4) => {
"item": 4,
"msg": "4"
}
ok: [localhost] => (item=1) => {
"item": 1,
"msg": "1"
}
ok: [localhost] => (item=3) => {
"item": 3,
"msg": "3"
}
ok: [localhost] => (item=5) => {
"item": 5,
"msg": "5"
}

17. Lookups
● you are already using them: with_<lookup>
● they execute on the “master”
● a way to access external files and/or data
● normally returns a list

18. Example hack - lookup
lib/ansible/runner/lookup_plugins/etcd.py
class LookupModule(object):
def __init__(self, basedir=None, **kwargs):
self.basedir = basedir
self.etcd = etcd() # initializes etcd class
def run(self, terms, inject=None, **kwargs):
terms = utils.listify_lookup_plugin_terms(terms, self.basedir, inject)
if isinstance(terms, basestring):
terms = [ terms ]
ret = []
for term in terms:
key = term.split()[0]
value = self.etcd.get(key) # gets the data from etcd class
ret.append(value)
return ret

19. Example hack - lookup
lib/ansible/runner/lookup_plugins/etcd.py
class etcd(): # simplified version
def __init__(self, url=ANSIBLE_ETCD_URL):
self.url = url
self.baseurl = '%s/v1/keys' % (self.url)
def get(self, key):
url = "%s/%s" % (self.baseurl, key)
r = urllib2.urlopen(url)
data = r.read()
item = json.loads(data)
value = item['value']
return value
_______________
/ written by
Jan-Piet Mens /
---------------
,__,
(oo)____
(__) )
||--|| *

20. Example hack - lookup
#>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key1”)}}’ -i ‘localhost,’
localhost | success >> {
"msg": "value1"
}
#>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key2”)}}’ -i ‘localhost,’
localhost | success >> {
"msg": "value2"
}

21. Example hack - callback
callback_plugins/profile_tasks.py
class CallbackModule(object): # simplified example
def playbook_on_task_start(self, name, is_conditional):
_______________
/ written by
Jharrod LaFon /
---------------
,__,
(oo)____
if self.current is not None: # Record the total time of the previous
self.stats[self.current] = time.time() - self.stats[self.current]
self.current = name # Record the start time of the current task
self.stats[self.current] = time.time()
def playbook_on_stats(self, stats): # Sort the tasks by their running time
...
(__) )
||--|| *
results = sorted(self.stats.items(), key=lambda value: value[1], reverse=True)
for name, elapsed in results: # Print the timings
print "{0:-<70}{1:->9}".format('{0} '.format(name),'{0:.02f}s'.format(elapsed))

22. Example hack - callback
#>ansible-playbook test.yml -i ‘localhost,’
# re-running the previous filter/shuffle playbook
__________________________
< TASK: debug msg={{item}} >
--------------------------
ok: [localhost] => (item=2) => {
"item": 2,
"msg": "2"
}
...
ok: [localhost] => (item=4) => {
"item": 4,
"msg": "4"
}
PLAY RECAP ********************************************************************
debug msg={{item}} ------------------------------------------------------ 0.02s
localhost : ok=1 changed=0 unreachable=0 failed=0

23. Example hack - cache
import exceptions
class BaseCacheModule(object):
def get(self, key):
lib/ansible/cache/base.py
raise exceptions.NotImplementedError
def set(self, key, value):
raise exceptions.NotImplementedError
def keys(self):
raise exceptions.NotImplementedError
def contains(self, key):
raise exceptions.NotImplementedError
def delete(self, key):
raise exceptions.NotImplementedError
def flush(self):
raise exceptions.NotImplementedError
def copy(self):
raise exceptions.NotImplementedError
________________
/ I abandoned it
| and Josh Drake |
revived it /
----------------
/ ___ /
// / /
(( O O ))
/ //
/ | | /
| | | |
| | | |
| o |
| | | |
|m| |m|

24. Example hack - cache
from ansible.cache.base import BaseCacheModule
class CacheModule(BaseCacheModule):
def __init__(self, *args, **kwargs):
self._cache = {}
def get(self, key):
return self._cache.get(key)
def set(self, key, value):
self._cache[key] = value
def keys(self):
return self._cache.keys()
def contains(self, key):
return key in self._cache
def delete(self, key):
del self._cache[key]
def flush(self):
lib/ansible/cache/memory.py

25. Example hack - cache
from redis import StrictRedis
class CacheModule(BaseCacheModule): # simplified version
self._cache = StrictRedis(*connection)
...
def get(self, key):
value = self._cache.get(self._make_key(key))# make_key creates correct prefix
# guard against the key not being removed from the zset
if value is None:
self.delete(key)
raise KeyError
return json.loads(value)
def set(self, key, value):
value2 = json.dumps(value)
if self._timeout > 0: # a timeout of 0 is handled as meaning 'never expire'
self._cache.setex(self._make_key(key), int(self._timeout), value2)
else:
self._cache.set(self._make_key(key), value2)
self._cache.zadd(self._keys_set, time.time(), key)
...
lib/ansible/cache/redis.py

26. Tests: test/
● sadly the presenter sucks at test coverage
● no really, tests are good, regressions!
● assert: the test module
● destructive, non destructive, integration
● ‘make tests’, runs unit tests

27. Before you submit
● update documentation (it is also in the repo)
● create tests, assert! (also in the repo)
● prepare a clear usage/example to post in the PR
● its a dialog, be ready to make your ‘use case’
● patience … … … …
● not everything belongs in core