Charles Wheelus gave a presentation on using Splunk to ensure compliance with service level agreements (SLAs) at a wireless carrier. He described developing a system using Splunk to measure key performance indicators, ingest data from various systems, build reports on SLA compliance, and provide real-time analytics and dashboards. Wheelus emphasized how Splunk helped solve challenges like integrating diverse data sources, performing load testing analysis, and alerting operations teams.

1. SLA Compliance Assurance
Charles Wheelus
Senior Data Scientist, Cequint
Splunk LIVE Baltimore
December 12th, 2013
1
1

2. 2
2

3. About me:
Charles Wheelus, MSCS
• Senior Data Scientist, Cequint
• Ph.D. Candidate, Florida Atlantic University
research interests: Data Mining and Machine Learning
• 2012 Splunk Ninja Revolution award recipient
• Splunk Certified Architect
• Technology consultant for 20 years
• Splunk user and evangelist for three years
• Started with version 4.3
3
3

4. About
Cequint provides handset and Carrier data services to
most major wireless carriers in the U.S.
http://cequint.com
4
4

5. About
5
5

6. About
6
6

7. Service Level Agreement (SLA)
Compliance Assurance
Charles Wheelus
December 12th, 2013
7
7

8. ...or
How to kill a flock of birds with one stone
Charles Wheelus
December 12th, 2013
8
8

9. Disclaimer: No birds were injured during the
production of this presentation. :)
Charles Wheelus
December 12th, 2013
9
9

10. SLA Compliance
(on a Wireless Carrier network)
The project:
Develop a system that provides proof
of our SLA compliance with our
carrier customer
Time is of the essence!
Charles Wheelus
December 12th, 2013
10
10

11. SLA Compliance
Determine the Key Performance Indicators
• Numerous subsystems
• Different development teams
• Different programming languages
• Different operating systems
• Wide variety of hardware types
Charles Wheelus
December 12th, 2013
11
11

12. SLA Compliance
The “Cat Herder”
Charles Wheelus
December 12th, 2013
12
12

13. SLA Compliance
Determine what data to get
Study the SLA
Engage others in the process
• Developers
• Management
• Product team
• Operations
Charles Wheelus
December 12th, 2013
13
13

14. SLA Compliance
Determine the best place(s) to get the data from
Charles Wheelus
December 12th, 2013
14
14

15. SLA Compliance
Establish best practice for data input
What simple step can you take
in the beginning that will save time later?
Best practices document
Verify the data is in the expected format!
Charles Wheelus
December 12th, 2013
15
15

16. SLA Compliance
Determine transport method for getting the data
into Splunk
syslog
UDP
Charles Wheelus
December 12th, 2013
16
16

17. SLA Compliance
Determine transport method for getting the data
into Splunk
Universal
Forwarder
Charles Wheelus
December 12th, 2013
17
17

18. SLA Compliance
Determine transport method for getting the data
into Splunk
Charles Wheelus
December 12th, 2013
18
18

19. SLA Compliance
Determine transport method for getting the data
into Splunk
Charles Wheelus
December 12th, 2013
18
18

20. SLA Compliance
Determine transport method for getting the data
into Splunk
Charles Wheelus
December 12th, 2013
18
18

21. SLA Compliance
KPI Data flow diagram
Charles Wheelus
December 12th, 2013
19
19

22. SLA Compliance
SLA report (RECAP):
• Establish KPI
• Get KPI data into Splunk
• KPI counter aggregation and
reconciliation
• Use Splunk REST API to build the
report
Charles Wheelus
December 12th, 2013
20
20

23. SLA Compliance
SLA report (RECAP):
• Establish KPI
• Get KPI data into Splunk
• KPI counter aggregation and
reconciliation
• Use Splunk REST API to build the
report
Charles Wheelus
December 12th, 2013
20
20

24. SLA Compliance
Charles Wheelus
December 12th, 2013
21
21

25. Charles Wheelus
December 12th, 2013
22
22

26. Charles Wheelus
December 12th, 2013
22
22

27. “Black-box” testing
The problem:
Performance information about the
Carrier’s self provisioning gateway is
unavailable. We have to run our own tests
to determine the expected performance
Time is of the essence!
Charles Wheelus
December 12th, 2013
23
23

28. Charles Wheelus
December 12th, 2013
24
24

29. Black Box Testing
Charles Wheelus
December 12th, 2013
25
25

30. Black Box Testing
Charles Wheelus
December 12th, 2013
25
25

31. Charles Wheelus
December 12th, 2013
26
26

32. Charles Wheelus
December 12th, 2013
26
26

33. Load test results analysis
The problem:
We need a quick way to evaluate the
results of load testing.
Time is of the essence!
Charles Wheelus
December 12th, 2013
27
27

34. Load test results analysis
Charles Wheelus
December 12th, 2013
28
28

35. Load test results analysis
Charles Wheelus
December 12th, 2013
28
28

36. Load test results analysis
Charles Wheelus
December 12th, 2013
29
29

37. Load test results analysis
Charles Wheelus
December 12th, 2013
29
29

38. Charles Wheelus
December 12th, 2013
30
30

39. Charles Wheelus
December 12th, 2013
30
30

40. Event Reporting
The problem:
Thousands of subsystem events may be
generated into the log files, some events are
inter-dependent. We need a comprehensive
and robust system for detecting, correlating,
and reporting these events to the correct
development team.
Time is of the essence!
Charles Wheelus
December 12th, 2013
31
31

41. Event Reporting
The solution:
Splunk saved and scheduled searches!
With very brief training, the developers are
building their own queries, saving and scheduling
Charles Wheelus
December 12th, 2013
32
32

42. Event Reporting
Charles Wheelus
December 12th, 2013
32
32

43. Event Reporting
Charles Wheelus
December 12th, 2013
32
32

44. Event Reporting
Charles Wheelus
December 12th, 2013
32
32

45. Charles Wheelus
December 12th, 2013
33
33

46. Charles Wheelus
December 12th, 2013
33
33

47. Event Monitoring and Alarming
The problem:
The operations team requires that the
KPI produce alarm output into their preexisting monitoring and alarm system
Time is of the essence!
Charles Wheelus
December 12th, 2013
34
34

48. Event Monitoring and Alarming
• Operations has pre-existing alarming software
• Splunk was connected to OPS alarm system
using the Splunk API
Charles Wheelus
December 12th, 2013
35
35

49. Event Monitoring and Alarming
Charles Wheelus
December 12th, 2013
35
35

50. Charles Wheelus
December 12th, 2013
36
36

51. Charles Wheelus
December 12th, 2013
36
36

52. Performance Analysis
The problem:
The entire team needs to have up to the
minute business intelligence.
Time is of the essence!
Charles Wheelus
December 12th, 2013
37
37

53. Performance Analysis
The answer:
Splunk Dashboards and Apps!
Charles Wheelus
December 12th, 2013
38
38

54. Performance Analysis
• Customized tools for Developers
• Dashboards for Operations
• Trouble shooting for Developers and
Operations
• Business Intelligence for Management
Charles Wheelus
December 12th, 2013
39
39

55. 40
40

56. 40
40

57. 40
40

58. 40
40

59. 40
40

60. 40
40

61. Cut to the chase
Splunk’s greatest benefits:
•Time savings
•Ability to react quickly (SPL)
•Real time analytics
•Rapid dashboard production
•The Splunk Community !!
Charles Wheelus
December 12th, 2013
41
41

62. What’s next?
• New metrics & dashboards
• Modular inputs
• More use of Splunk 3rd party Apps
• Predictive Analytics
• Data Models / Pivots
Charles Wheelus
December 12th, 2013
42
42

63. Charles Wheelus
December 12th, 2013
43
43

64. Charles Wheelus
http://about.me/charleswheelus
http://facebook.com/charleswheelus
charles@wheelus.com
44
44