SLA Compliance Assurance
Charles Wheelus

Senior Data Scientist, Cequint

Splunk LIVE Baltimore
December 12th, 2013

1
1
2
2
About me:
Charles Wheelus, MSCS

• Senior Data Scientist, Cequint
• Ph.D. Candidate, Florida Atlantic University

research...
About
Cequint provides handset and Carrier data services to
most major wireless carriers in the U.S.

http://cequint.com
4...
About

5
5
About

6
6
Service Level Agreement (SLA)
Compliance Assurance

Charles Wheelus

December 12th, 2013
7
7
...or

How to kill a flock of birds with one stone
Charles Wheelus

December 12th, 2013
8
8
Disclaimer: No birds were injured during the
production of this presentation. :)

Charles Wheelus

December 12th, 2013
9
9
SLA Compliance
(on a Wireless Carrier network)

The project:
Develop a system that provides proof
of our SLA compliance wi...
SLA Compliance
Determine the Key Performance Indicators

• Numerous subsystems
• Different development teams
• Different p...
SLA Compliance
The “Cat Herder”

Charles Wheelus

December 12th, 2013
12
12
SLA Compliance
Determine what data to get
Study the SLA
Engage others in the process

• Developers
• Management
• Product ...
SLA Compliance
Determine the best place(s) to get the data from

Charles Wheelus

December 12th, 2013
14
14
SLA Compliance
Establish best practice for data input
What simple step can you take
in the beginning that will save time l...
SLA Compliance
Determine transport method for getting the data
into Splunk

syslog
UDP

Charles Wheelus

December 12th, 20...
SLA Compliance
Determine transport method for getting the data
into Splunk

Universal
Forwarder

Charles Wheelus

December...
SLA Compliance
Determine transport method for getting the data
into Splunk

Charles Wheelus

December 12th, 2013
18
18
SLA Compliance
Determine transport method for getting the data
into Splunk

Charles Wheelus

December 12th, 2013
18
18
SLA Compliance
Determine transport method for getting the data
into Splunk

Charles Wheelus

December 12th, 2013
18
18
SLA Compliance
KPI Data flow diagram

Charles Wheelus

December 12th, 2013
19
19
SLA Compliance
SLA report (RECAP):

• Establish KPI
• Get KPI data into Splunk
• KPI counter aggregation and
reconciliatio...
SLA Compliance
SLA report (RECAP):

• Establish KPI
• Get KPI data into Splunk
• KPI counter aggregation and
reconciliatio...
SLA Compliance

Charles Wheelus

December 12th, 2013
21
21
Charles Wheelus

December 12th, 2013
22
22
Charles Wheelus

December 12th, 2013
22
22
“Black-box” testing
The problem:
Performance information about the
Carrier’s self provisioning gateway is
unavailable. We ...
Charles Wheelus

December 12th, 2013
24
24
Black Box Testing

Charles Wheelus

December 12th, 2013
25
25
Black Box Testing

Charles Wheelus

December 12th, 2013
25
25
Charles Wheelus

December 12th, 2013
26
26
Charles Wheelus

December 12th, 2013
26
26
Load test results analysis
The problem:
We need a quick way to evaluate the
results of load testing.
Time is of the essenc...
Load test results analysis

Charles Wheelus

December 12th, 2013
28
28
Load test results analysis

Charles Wheelus

December 12th, 2013
28
28
Load test results analysis

Charles Wheelus

December 12th, 2013
29
29
Load test results analysis

Charles Wheelus

December 12th, 2013
29
29
Charles Wheelus

December 12th, 2013
30
30
Charles Wheelus

December 12th, 2013
30
30
Event Reporting
The problem:
Thousands of subsystem events may be
generated into the log files, some events are
inter-depen...
Event Reporting
The solution:
Splunk saved and scheduled searches!
With very brief training, the developers are
building t...
Event Reporting

Charles Wheelus

December 12th, 2013
32
32
Event Reporting

Charles Wheelus

December 12th, 2013
32
32
Event Reporting

Charles Wheelus

December 12th, 2013
32
32
Charles Wheelus

December 12th, 2013
33
33
Charles Wheelus

December 12th, 2013
33
33
Event Monitoring and Alarming
The problem:
The operations team requires that the
KPI produce alarm output into their preex...
Event Monitoring and Alarming

• Operations has pre-existing alarming software
• Splunk was connected to OPS alarm system
...
Event Monitoring and Alarming

Charles Wheelus

December 12th, 2013
35
35
Charles Wheelus

December 12th, 2013
36
36
Charles Wheelus

December 12th, 2013
36
36
Performance Analysis
The problem:
The entire team needs to have up to the
minute business intelligence.
Time is of the ess...
Performance Analysis
The answer:
Splunk Dashboards and Apps!

Charles Wheelus

December 12th, 2013
38
38
Performance Analysis

• Customized tools for Developers
• Dashboards for Operations
• Trouble shooting for Developers and
...
40
40
40
40
40
40
40
40
40
40
40
40
Cut to the chase
Splunk’s greatest benefits:

•Time savings
•Ability to react quickly (SPL)
•Real time analytics
•Rapid das...
What’s next?

• New metrics & dashboards
• Modular inputs
• More use of Splunk 3rd party Apps
• Predictive Analytics
• Dat...
Charles Wheelus

December 12th, 2013
43
43
Charles Wheelus
http://about.me/charleswheelus
http://facebook.com/charleswheelus
charles@wheelus.com

44
44
Upcoming SlideShare
Loading in...5
×

SplunkLive! Customer Presentation - Cequint

492

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
492
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

SplunkLive! Customer Presentation - Cequint

  1. 1. SLA Compliance Assurance Charles Wheelus Senior Data Scientist, Cequint Splunk LIVE Baltimore December 12th, 2013 1 1
  2. 2. 2 2
  3. 3. About me: Charles Wheelus, MSCS • Senior Data Scientist, Cequint • Ph.D. Candidate, Florida Atlantic University research interests: Data Mining and Machine Learning • 2012 Splunk Ninja Revolution award recipient • Splunk Certified Architect • Technology consultant for 20 years • Splunk user and evangelist for three years • Started with version 4.3 3 3
  4. 4. About Cequint provides handset and Carrier data services to most major wireless carriers in the U.S. http://cequint.com 4 4
  5. 5. About 5 5
  6. 6. About 6 6
  7. 7. Service Level Agreement (SLA) Compliance Assurance Charles Wheelus December 12th, 2013 7 7
  8. 8. ...or How to kill a flock of birds with one stone Charles Wheelus December 12th, 2013 8 8
  9. 9. Disclaimer: No birds were injured during the production of this presentation. :) Charles Wheelus December 12th, 2013 9 9
  10. 10. SLA Compliance (on a Wireless Carrier network) The project: Develop a system that provides proof of our SLA compliance with our carrier customer Time is of the essence! Charles Wheelus December 12th, 2013 10 10
  11. 11. SLA Compliance Determine the Key Performance Indicators • Numerous subsystems • Different development teams • Different programming languages • Different operating systems • Wide variety of hardware types Charles Wheelus December 12th, 2013 11 11
  12. 12. SLA Compliance The “Cat Herder” Charles Wheelus December 12th, 2013 12 12
  13. 13. SLA Compliance Determine what data to get Study the SLA Engage others in the process • Developers • Management • Product team • Operations Charles Wheelus December 12th, 2013 13 13
  14. 14. SLA Compliance Determine the best place(s) to get the data from Charles Wheelus December 12th, 2013 14 14
  15. 15. SLA Compliance Establish best practice for data input What simple step can you take in the beginning that will save time later? Best practices document Verify the data is in the expected format! Charles Wheelus December 12th, 2013 15 15
  16. 16. SLA Compliance Determine transport method for getting the data into Splunk syslog UDP Charles Wheelus December 12th, 2013 16 16
  17. 17. SLA Compliance Determine transport method for getting the data into Splunk Universal Forwarder Charles Wheelus December 12th, 2013 17 17
  18. 18. SLA Compliance Determine transport method for getting the data into Splunk Charles Wheelus December 12th, 2013 18 18
  19. 19. SLA Compliance Determine transport method for getting the data into Splunk Charles Wheelus December 12th, 2013 18 18
  20. 20. SLA Compliance Determine transport method for getting the data into Splunk Charles Wheelus December 12th, 2013 18 18
  21. 21. SLA Compliance KPI Data flow diagram Charles Wheelus December 12th, 2013 19 19
  22. 22. SLA Compliance SLA report (RECAP): • Establish KPI • Get KPI data into Splunk • KPI counter aggregation and reconciliation • Use Splunk REST API to build the report Charles Wheelus December 12th, 2013 20 20
  23. 23. SLA Compliance SLA report (RECAP): • Establish KPI • Get KPI data into Splunk • KPI counter aggregation and reconciliation • Use Splunk REST API to build the report Charles Wheelus December 12th, 2013 20 20
  24. 24. SLA Compliance Charles Wheelus December 12th, 2013 21 21
  25. 25. Charles Wheelus December 12th, 2013 22 22
  26. 26. Charles Wheelus December 12th, 2013 22 22
  27. 27. “Black-box” testing The problem: Performance information about the Carrier’s self provisioning gateway is unavailable. We have to run our own tests to determine the expected performance Time is of the essence! Charles Wheelus December 12th, 2013 23 23
  28. 28. Charles Wheelus December 12th, 2013 24 24
  29. 29. Black Box Testing Charles Wheelus December 12th, 2013 25 25
  30. 30. Black Box Testing Charles Wheelus December 12th, 2013 25 25
  31. 31. Charles Wheelus December 12th, 2013 26 26
  32. 32. Charles Wheelus December 12th, 2013 26 26
  33. 33. Load test results analysis The problem: We need a quick way to evaluate the results of load testing. Time is of the essence! Charles Wheelus December 12th, 2013 27 27
  34. 34. Load test results analysis Charles Wheelus December 12th, 2013 28 28
  35. 35. Load test results analysis Charles Wheelus December 12th, 2013 28 28
  36. 36. Load test results analysis Charles Wheelus December 12th, 2013 29 29
  37. 37. Load test results analysis Charles Wheelus December 12th, 2013 29 29
  38. 38. Charles Wheelus December 12th, 2013 30 30
  39. 39. Charles Wheelus December 12th, 2013 30 30
  40. 40. Event Reporting The problem: Thousands of subsystem events may be generated into the log files, some events are inter-dependent. We need a comprehensive and robust system for detecting, correlating, and reporting these events to the correct development team. Time is of the essence! Charles Wheelus December 12th, 2013 31 31
  41. 41. Event Reporting The solution: Splunk saved and scheduled searches! With very brief training, the developers are building their own queries, saving and scheduling Charles Wheelus December 12th, 2013 32 32
  42. 42. Event Reporting Charles Wheelus December 12th, 2013 32 32
  43. 43. Event Reporting Charles Wheelus December 12th, 2013 32 32
  44. 44. Event Reporting Charles Wheelus December 12th, 2013 32 32
  45. 45. Charles Wheelus December 12th, 2013 33 33
  46. 46. Charles Wheelus December 12th, 2013 33 33
  47. 47. Event Monitoring and Alarming The problem: The operations team requires that the KPI produce alarm output into their preexisting monitoring and alarm system Time is of the essence! Charles Wheelus December 12th, 2013 34 34
  48. 48. Event Monitoring and Alarming • Operations has pre-existing alarming software • Splunk was connected to OPS alarm system using the Splunk API Charles Wheelus December 12th, 2013 35 35
  49. 49. Event Monitoring and Alarming Charles Wheelus December 12th, 2013 35 35
  50. 50. Charles Wheelus December 12th, 2013 36 36
  51. 51. Charles Wheelus December 12th, 2013 36 36
  52. 52. Performance Analysis The problem: The entire team needs to have up to the minute business intelligence. Time is of the essence! Charles Wheelus December 12th, 2013 37 37
  53. 53. Performance Analysis The answer: Splunk Dashboards and Apps! Charles Wheelus December 12th, 2013 38 38
  54. 54. Performance Analysis • Customized tools for Developers • Dashboards for Operations • Trouble shooting for Developers and Operations • Business Intelligence for Management Charles Wheelus December 12th, 2013 39 39
  55. 55. 40 40
  56. 56. 40 40
  57. 57. 40 40
  58. 58. 40 40
  59. 59. 40 40
  60. 60. 40 40
  61. 61. Cut to the chase Splunk’s greatest benefits: •Time savings •Ability to react quickly (SPL) •Real time analytics •Rapid dashboard production •The Splunk Community !! Charles Wheelus December 12th, 2013 41 41
  62. 62. What’s next? • New metrics & dashboards • Modular inputs • More use of Splunk 3rd party Apps • Predictive Analytics • Data Models / Pivots Charles Wheelus December 12th, 2013 42 42
  63. 63. Charles Wheelus December 12th, 2013 43 43
  64. 64. Charles Wheelus http://about.me/charleswheelus http://facebook.com/charleswheelus charles@wheelus.com 44 44

×