Splunk Upgrade Using Ansible by Sandeep Sarkar

1. Bengaluru User Group
WELCOME
5th Sep 2020
स्वागत
স্বাগত
ಸ್ವಾಗತ
स्वागत आहे
స్వాగతவரவவற்பு
സ്വാഗതം
ਸਵਾਗਤ ਹੈ
સ્વાગત છે
‫آمدید‬ ‫خوش‬
ସ୍ୱାଗତ
‫آیا‬ ‫ڪري‬ ‫ڀلي‬

2. https://conf.splunk.com/
https://www.youtube.com/watch?v=C8UzEaF2OwQ

3. https://events.splunk.com/the-splunkies-2020
The Data Heroes Award
The Home-Office-Hero Award
The Innovation Award The Developer Award
The Community Award
The Ecosystem Award
Get your nominations in by September 18th

4. Housekeeping
Join #splunk_bengaluru_usergroup on Slack http://splk.it/slack
Use #splunk_bengaluru_usergroup for Q&A during the session
Please keep your lines muted when not speaking
Slides, recording & feedback form will be posted to the Events page
Splunk Bengaluru User Group
https://usergroups.splunk.com/bengaluru-splunk-user-group/

5. © 2019 SPLUNK INC.
Sandeep Sarkar
Senior Consultant (Mercedes-Benz India)
Splunk Upgrade Through Ansible
5th Sept 2020

6. Agenda
Topics for today
Why Splunk Upgrade is Important ?
How to Plan your Upgrade ?
What is Ansible ?
Configure your Ansible environment
Code Walkthrough – Ansible
Demo
Q & A

7. Why Splunk Upgrades are required!
Mitigate the Security Risks of older versions!
Meet the Auditing requirement
Version Out of Support
We want new features!
Fix some known bugs by moving to a newer version

8. Plan Plan & again Plan!
How to find a needle in multiple haystacks?
(choose your tool)
Discover Prepare Test in Dev/QA Upgrade Verify
• Understand your
Architecture .
• Create Inventory with
Splunk versions.
• Determine your
destination versions
• Create app compatibility
matrix with your intended
Upgrade version
• Backup Splunk
configurations.
• System Health Check.
• Check for any SSL
connectivity issues with
current & intended version.
Plan your upgrade thoroughly
• Test your upgrade scripts in
QA or Dev environment.
• Verify your planning.
• Upgrade in the order
described by the Splunk
docs.
• Upgrade Cluster Master.
• Upgrade Search head
tier.
• Upgrade Peer node tier.
• Upgrade Forwarder tier
• Verify the system health.
• Verify the log streams.
• Verify the roles/functions of
each server.

9. Upgrade
Master Node
a. Stop the master
i. /opt/splunk/bin/splunk stop
b. Take backup
i. tar -zcvf backup_splunk.tar.gz /opt/splunk/etc/
c. Copy new package in in /opt directory as root
d. Run rpm command to install
i. rpm -U --nodeps --prefix=/opt/splunk-<version>-<build>-
linux-2.6-x86_64.rpm
or ii. tar -xvzf splunk-<version>-<build>-Linux-x86_64.tgz -
C /opt/
e. Start splunk now accepting license as root user
i. /opt/splunk/bin/splunk start --accept-license --answer-yes
f. Enable boot start
i. /opt/splunk/bin/splunk enable boot-start -user splunk
g. Stop Splunk as root user
i. /opt/splunk/bin/splunk stop
h. Change user to Splunk
i. su splunk
I. Start Splunk
i. /opt/splunk/bin/splunk start
j. View the master dashboard to verify that all cluster nodes are up and
running.

10. Upgrade
Search Head
Tier
a. Stop all cluster members
i. /opt/splunk/bin/splunk stop
b. Take backup
i. tar -zcvf backup_splunk.tar.gz /opt/splunk/etc
c. Take backup of KV Store
i. /opt/splunk/bin/splunk backup kvstore –archiveName
<archive>
d. Upgrade all members
i. Follow steps c to h from “Upgrade the master node” section
e. Stop the deployer
i. /opt/splunk/bin/splunk stop
f. Take backup
i. tar -zcvf backup_splunk.tar.gz /opt/splunk/
g. Upgrade the deployer
i. Follow steps c to h from “Upgrade the master node” section
h. Start the deployer
i. Start the members

11. Upgrade
Peer Node
tier
a. Run splunk enable maintenance-mode on the master
b. Confirm the above step using splunk show maintenance-mode
c. Stop all the peer nodes
d. Take backup
e. Upgrade the peer nodes
i. Follow steps c to h from “Upgrade the master node” section
f. Start the peer nodes
g. Run splunk disable maintenance-mode
h. Confirm the above step using splunk show maintenance-mode

12. Ansible
Ansible is an open-source software
provisioning & configuration
management tool.
Ansible is agentless, works via
connecting remotely through SSH or
Windows Remote Management
(allowing remote PowerShell execution)
to do its tasks.
Ansible uses push mechanism
Ansible uses YAML syntax to
describe the automation tasks.

13. Ansible
Setup
Install Ansible
• sudo apt install ansible –yes
• https://docs.ansible.com/ansible/latest/install
ation_guide/intro_installation.html
Configure Your environment to Use
Ansible
• Create your ssh-keygen & share it with all
your target hosts
Create the directory Structure
• Create the variable files required for the
playbooks to run
• Copy the installer files into a specific
directory
• Install or verify python version (pexpect
module)

14. Ansible
Setup
More Videos Refer to Session Recording for video walkthrough: https://youtu.be/UkbfTjIovjw?t=968

15. Ansible
Setup
More Videos
Refer to Session Recording for video walkthrough: https://youtu.be/UkbfTjIovjw?t=1063
Refer to Session Recording for video walkthrough: https://youtu.be/UkbfTjIovjw?t=1258

16. Code
walkthrough
Master YAML

17. Code
Walkthrough
Cluster Master YAML
1
2 3

18. Code
Walkthrough
Search Head YAML
12
3

19. Code
walkthrough
Indexers YAML

20. Code
Walkthrough
Windows YAML
1
2
3

21. Demo

22. Upgrade Splunk!
Refer to Session Recording for video walkthrough: https://youtu.be/UkbfTjIovjw?t=1948

23. © 2020 SPLUNK INC.
Further
resources Splunk Upgrade Steps –
https://docs.splunk.com/Documentation/Splunk/latest/Instal
lation/HowtoupgradeSplunk
Register for upcoming .conf20 session –
TRU1504C - Ansible Starter Pack for Automating Splunk
Administration
Mason Morales, Sr. Manager, Splunk@Splunk, Splunk
Installing Ansible –
https://docs.ansible.com/ansible/latest/installation_guide/int
ro_installation.html#selecting-an-ansible-version-to-install

24. © 2020 SPLUNK INC.
Q&A
Raise hand to be unmuted Post questions in WebEx
Chat
Join Slack for Q&A
http://splk.it/slack

25. © 2020 SPLUNK INC.
Contribute, Collaborate and win
#splunk_bengaluru_usergroup
• Token of appreciation for the Speakers in the Community
event
Sandeep Sarkar
• Monthly reward for winners of Challenges posted in Slack
Sanjeev Reddy
http://splk.it/slack

26. © 2020 SPLUNK INC.
Challenges on Slack
#splunk_bengaluru_usergroup
Rule for participation*
• Must have attended User Group Session and Checked In.
• In the Slack thread only mention challenge# attempted (do not answer in Slack chat).
• Send personal note on Slack with actual answer or email.
• Winner will be based on first one to get max. correct answer.
• If you have already won previously in last 12 sessions, preference will be given to second best
answer.
• Answers to challenges from August Bengaluru User Group session posted on Slack.
• Challenges from September session posted on Slack.
http://splk.it/slack

27. © 2020 SPLUNK INC.
Community Resources
Splunk Community Resources (Both Official and Unofficial)
Splunk > Clara-fication: Splunk Community: https://www.splunk.com/en_us/blog/tips-
and-tricks/splunk-clara-fication-splunk-community.html

28. We plan to meet 1st Saturday of every month at 11:00 AM IST.
Please provide feedback for :
• Sessions and improvements.
• Topics to be covered in future sessions.
• Let us know if you are interested in presenting in User Group.
Keep the comradery through Slack and Splunk Answers>
What’s Next
http://splk.it/slack http://community.splunk.com
https://conf.splunk.com
Splunk .Conf 2020 registrations are open: Oct 20th and 21st (Virtual)

29. Thank You