Embedded Ansible In Cloudforms Deep Dive

1. Embedded Ansible in
Cloudform
Prasad Mukhedkar

2. Agenda
● Introduction to Embedded Ansible
● Enabling Embedded Ansible
● Tuning Embedded Ansible
● Adding Playbook Repositories and Credentials
● Run Playbook From Service
● Run Playbook from button
● Run playbook as part of Automate State Machine ( Automate)
● Troubleshooting
● Future

3. Introduction
Embedded Ansible is optional automation engine in
cloudforms to simplify automation workflows. Alternative
to Ruby based automation framework.
Headless Ansible Tower is installed and configured on the
cloudforms appliance for implementation of Ansible
automation engine.

4. Activating Embedded Ansible
- Enabling ‘Embedded Ansible’ role activates Ansible
automation engine.
- Upon activating the role, tower setup command is
triggered to setup the tower, this activity may take several
minutes.
- All the packages of ansible tower are shipped in the

5. Activating Embedded Ansible
Ansible Tower Setup playbook
-> #ansible-tower-setup
- > #/var/lib/awx/setup/setup.sh
-> # /var/lib/awx/setup/install.yml

6. Activating Embedded Ansible
/var/lib/awx/setup/inventory
[tower]
localhost ansible_connection=local
[database]
[all:vars]
admin_password=''
pg_host=''
pg_port=''
pg_database='awx'
pg_username='awx'
pg_password=''
rabbitmq_username=tower
rabbitmq_password=''
rabbitmq_cookie=cookiemonster
admin_password=''
rabbitmq_password=''
admin_password=''
All passwords generated
dynamically !
def find_or_create_admin_authentication
miq_database.ansible_admin_authentication ||
miq_database.set_ansible_admin_authentication(:
password => generate_password)
end

7. Activating Embedded Ansible
EmbeddedAnsible Processes ^
Core Embedded Ansible Services (Supervisord)

8. Embedded Ansible Logs
Setup Logs :
LOG_DIR="/var/log/tower"
LOG_FILE="${LOG_DIR}/setup-${TIMESTAMP}.log"
Other Logs :
/var/log/tower/tower.log
/var/log/tower/task_system.log

9. Embedded Ansible Logs
Ansible Tower Core Logs

10. Embedded Ansible APIs
Endpoint : https://cloudformsIP/ansibleapi/
Note: Ansible Tower API cannot be accessed from a
browser since the browser based authentication plug-
in is not installed. You have to use API clients like
postman or insomnia or curl command

11. Embedded Ansible APIs : Authentication
Admin Password is generated randomly at the time of ansible Setup. To
get the password you need to query rails.
# vmdb ; echo "MiqDatabase.first.ansible_admin_authentication.password " | rails c
Admin Password ^
# vmdb ; echo "MiqDatabase.first.ansible_aansible_rabbitmq_authentication.password
" | rails c
# vmdb ; echo "MiqDatabase.first.ansible_database_authentication.password " | rails
c

12. Embedded Ansible Database
Postgres Database Named : AWX
Credentials : No credentials need to pass if you login locally. Peer Authentication
[root@dhcp130-127 data]# cat pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
local all all peer map=usermap
local replication all peer map=usermap
hostssl all all all md5
host replication all all md5
#psql -U postgres -d awx

13. Advanced Configuration : Proxy
:embedded_ansible:
:host:
:password:
:port:
:user:
:scheme:
:job_data_retention_days

14. Consideration
- Enabling Embedded Ansible role in highly available region is not
supported in 4.6 < lower version, support is added in 4.7.
- Maximum concurrent workers for embedded ansible is 1.
- 2 GB Memory overhead
- EmbeddedAnsibleWorker, When looking at the diagnostics > workers, no
information about CPU and memory is displayed for Embedded
Ansible worker.
- Embedded ansible is not zone aware.
- Support playbooks executed from the scheduler not supported.

15. Adding Playbook
Repositories

16. Adding Repository

17. Adding Repository
- ems_operations role is responsible to perform all the repository add/remove actions.
- Need network connectivity to sync repos.
- Make Sure embedded ansible role is running state before adding repo, how to check
that : https://cloudformsIP/ansibleapi/v1/ping/
- To activate proxy for embedded ansible, you need to turn off and on the embedded
ansible role.
- Syncing git repository over SSL protocol fails if the SSL certificate is self-singed. To
workaround, disable SSL verification for git. Add following line /etc/tower/settings.py file
and restart the service.
"AWX_TASK_ENV": { "GIT_SSL_NO_VERIFY": "True" }
- YAML three dashes (---) are important in playbook. Playbooks which are not starting with --- will not be
detected by cloudforms.

18. Offline Repository | Not Officially Supported
vmdb; rails c
conn = EmbeddedAnsible.new.api_connection
conn.api.projects.create!(:name => "My Project", :scm_type => "git", :scm_url =>
"file:///opt/ansible_playbooks", :scm_update_on_launch => false, :organization =>
ManageIQ::Providers::EmbeddedAnsible::Provider.first.default_organization)

19. Adding Credentials

20. Adding Credential
Automation > Ansible > Credentials > Add New Credential

21. Run Playbook As Service

22. Playbook As Service
Playbook services are created in the same way as other service catalog
items, via the Catalog Items accordion in the Services -> Catalogs
section of the WebUI.

23. 1. Navigate to Services/Catalogs/Catalog Items.
2. Create "Ansible Playbook" catalog item with "Provision" and
"Retirement" tabs filled, add some extra vars.
3. Open the catalog item detail screen, click on Retirement tab.
4. Navigate to Services/Service Catalogs.
5. Order the service.
6. Wait until it will be provisioned.
7. Navigate to Services/My Services.
8. Click on the service in the tree.
9. Click Lifecycle/Retire this Service.
10. Wait until retirement will be finished.
11. Open Retirement tab of the service in My Services.

24. Run Playbook from Button

25. Run Playbook from Button
The Ansible Playbook button type still calls the
/System/Request/Order_Ansible_Playbook instance to launch the playbook service, but it
simplifies the process of creating the parameters that the order_ansible_playbook method
uses.

26. Run Playbook from
Control Policy

27. Run Playbook from Control Policy
1. Create a ansible Service catalog that you wish to
execute as part of control policy.
2. Navigate to Control/Explorer.
3. Expand Actions accordion.
4. Click Configuration/Add a new Action.
5. In action type choose "Run Ansible Playbook".
6. In Playbook Catalog Item choose just created catalog item.
7. Assign this action to some event in a host or vm control policy.
8. Assign policy profile which contains that policy to some host or vm.
9. Trigger the event which assigned to the policy.

28. Run Playbook from
Automation

29. Run Playbook from Automation (State
Machine)
1. Create a domain, a namespace, a class and an
instance.
2. Click on Methods tab of the instance.
3. Click Configuration/Create new method.
4. In the dropdown menu pick "Ansible Playbook".
5. Fill the required fields.
6. Navigate to Automate Simulation.

30. Troubleshooting
1 . Restart Ansible Tower by turning on / off Embedded Ansible Role.
2. Restart Embedded Ansible from Rails Console.
# vmdb ; echo " EmbeddedAnsibleWorker.destroy_all " | rails c

31. TroubleShooting : Re-Setup
systemctl restart evmserverd #to be sure memory is freeed up
rm /etc/tower/SECRET_KEY
vmdb
bin/rails r 'MiqDatabase.first.ansible_secret_key = nil'
bin/rails r 'MiqDatabase.first.ansible_database_authentication.destroy'
bin/rails r 'ManageIQ::Providers::EmbeddedAnsible::Provider.first.destroy'
psql -d vmdb_production -c 'DROP DATABASE awx' # run on the db appliance
psql -d vmdb_production -c 'DROP ROLE awx' # run on the db appliance
Login to to the UI and enable Ansible role
check /var/log/tower/setup-.log for status

32. TroubleShooting : Installing pyhton modules
# source /var/lib/awx/venv/ansible/bin/activate
# umask 0022
# pip install --upgrade module
# deactivate

33. TroubleShooting : Windows Machines
ansible_connection: winrm
ansible_winrm_server_cert_validation: ignore
ansible_winrm_read_timeout_sec: 150
ansible_winrm_operation_timeout_sec: 120

34. TroubleShooting : Playbook Execution output
grep ServiceAnsiblePlaybook#log_stdout
/var/www/miq/vmdb/evm.log
[----] I, [2019-04-18T05:03:29.338193 #5474:db06c8] INFO -- : Q-
task_id([r10000000000001_service_template_provision_task_10000000000001])
MIQ(ServiceAnsiblePlaybook#log_stdout) Stdout from ansible job miq_Ping_provision: ansible-playbook 2.7.9

35. TroubleShooting : Playbook Logging Verbosity

36. Future
Headless tower will be replaced by Ansible Runner , Ansible Runner 1.1.2
(https://ansible-runner.readthedocs.io/en/latest/) is included in 4.7 with
appliance and used for some provider-related operations.

37. Reference
https://manageiq.gitbook.io/mastering-cloudforms-automation-addendum/embedded