The document discusses how CSAA Insurance Group expanded their use of Splunk from just application logs to also incorporate application performance monitoring (APM) data from Dynatrace. They were able to get real-time APM data into Splunk to create customizable performance reports. This allowed various teams like development to more accurately test applications and identify performance issues. Integrating these additional data sources into Splunk proved more valuable than the previous manual process and helped reduce errors by 75%.

1. 1© 2016 CSAA Insurance Group. Confidential and proprietary.© 2016 CSAA Insurance Group. Confidential and proprietary.
CSAA Insurance Group
Splunking the User Experience
Going Beyond Application Logs
with APM data
Doug Erkkila

2. 2© 2016 CSAA Insurance Group. Confidential and proprietary.
Agenda
Introduction
How We Introduced Splunk – Logs
Expanding Beyond Logs with Dynatrace
Feeding Knowledge Back into Development
Lessons Learned
Questions?

3. 3© 2016 CSAA Insurance Group. Confidential and proprietary.
Who am I?
Doug Erkkila
Capacity Management Analyst
• 10 years as an application
developer
• Specialized in data related
products
• 3 years with CSAA Insurance
Group
• Application Performance
Optimization
• Operational Monitoring
• DevOps

4. 4© 2016 CSAA Insurance Group. Confidential and proprietary.
What is CSAA Insurance Group?
• Insurance company offering
automobile, homeowners and
other personal lines of
insurance to AAA members
through AAA clubs
• More than 3600 employees
coast to coast
• Reaching nearly 17 million
AAA members in 23 states
and Washington DC

5. 5© 2016 CSAA Insurance Group. Confidential and proprietary.
What is PAS?
PAS is the central Policy
Administration System
• Consolidate policy administration
from across different insurance
categories and across different
states into one central system
• A Team of over 300 developers
and analysts
 Optimization Team is 5 Analysts

6. 6© 2016 CSAA Insurance Group. Confidential and proprietary.
In The Beginning:
There were Logs…..

7. 7© 2016 CSAA Insurance Group. Confidential and proprietary.
What Started It All?
Customer complaints about Application Errors (Error 500s)
• Was there an increase in exceptions?
• Difficult to understand error patterns across a cluster of servers
 Was an exception only occurring on a single server?
 Was an exception only occurring at a certain time of day?
• Is this a matter of perception or application quality?

8. 8© 2016 CSAA Insurance Group. Confidential and proprietary.
What Started It All?
A lot of questions, but finding answers was difficult
• Logs were rolled off so exception history was hard to gather
• Difficult to understand error patterns across a cluster of servers
 Any patterns had to come from intimate knowledge of logs
 Wasted support time reading logs every day
• Customer feedback can be sporadic and you tend to hear from the same
small group

9. 9© 2016 CSAA Insurance Group. Confidential and proprietary.
Error 500s
A single measurable use case
• Basic Aggregation and
Trends

10. 10© 2016 CSAA Insurance Group. Confidential and proprietary.
Error 500s
A single measurable use case
• Basic Aggregation and
Trends
• Added categorization

11. 11© 2016 CSAA Insurance Group. Confidential and proprietary.
Error 500s
A single measurable use case
• Basic Aggregation and Trends
• Added categorization
• Segment by Server

12. 12© 2016 CSAA Insurance Group. Confidential and proprietary.
Error 500s
A single measurable use case
• Basic Aggregation and Trends
• Added categorization
• Segment by Server
• Merge user information

13. 13© 2016 CSAA Insurance Group. Confidential and proprietary.
Error 500s
A single measurable use case
• Basic Aggregation and Trends
 Helped us see the increase in exceptions
• Added categorization
 Narrowed down timeframes for specific
errors
• Segment by Server
 Easy to see if the issue was server specific
• Merge user information
 Answered our Perception vs Quality
question

14. 14© 2016 CSAA Insurance Group. Confidential and proprietary.
What Started It All?
Customer complaints about Application Errors (Error 500s)
• Was there an increase in exceptions?
• Difficult to understand error patterns across a cluster of servers
 Was an exception only occurring on a single server?
 Was an exception only occurring at a certain time of day?
• Is this a matter of perception or application quality?

15. 15© 2016 CSAA Insurance Group. Confidential and proprietary.
What Started It All?
Customer complaints about Application Errors (Error 500s)
• Was there an increase in exceptions?
• Difficult to understand error patterns across a cluster of servers
 Was an exception only occurring on a single server?
 Was an exception only occurring at a certain time of day?
• Is this a matter of perception or application quality?

16. 16© 2016 CSAA Insurance Group. Confidential and proprietary.
What Started It All?
Customer complaints about Application Errors (Error 500s)
Was there an increase in exceptions?
Difficult to understand error patterns across a cluster of servers
 Was an exception only occurring on a single server?
 Was an exception only occurring at a certain time of day?
Is this a matter of perception or application quality?

17. 17© 2016 CSAA Insurance Group. Confidential and proprietary.
What Started It All?
Customer complaints about Application Errors (Error 500s)
Was there an increase in exceptions?
Difficult to understand error patterns across a cluster of servers
 Was an exception only occurring on a single server?
 Was an exception only occurring at a certain time of day?
Is this a matter of perception or application quality?

18. 18© 2016 CSAA Insurance Group. Confidential and proprietary.
Changing The Reality
75% reduction in errors
5076 0 1524 1074 5150
1000
2000
3000
4000
5000
6000
March 2014 March 2015 July 2015 January 2016
Without Splunk With Splunk

19. 19© 2016 CSAA Insurance Group. Confidential and proprietary.
Moving Beyond Logs with Dynatrace APM

20. 20© 2016 CSAA Insurance Group. Confidential and proprietary.
The Power of APM
We were already using Dynatrace for
APM
• Method level execution times
• User Experience Management
 Server/Network/Client side load times
 Browser data
• Database Response times
• JVM Heap Stats
Agent
Info
Dynatrace
Performance
Data
User-centered
Performance
Reports

21. 21© 2016 CSAA Insurance Group. Confidential and proprietary.
Exporting Data out of Dynatrace
• Using the Dynatrace REST API
• Nightly scheduled ETL
• Database storage
• Excel based reporting
Before Splunk
Nightly
ETL
Nightly
ETL
Performance
Warehouse
Weekly
Reporting
Excel
Reports
DynatraceVM

22. 22© 2016 CSAA Insurance Group. Confidential and proprietary.
Exporting Data out of Dynatrace
• Not real time
• Limited user population
• Fragile
 Scheduled tasks failed to trigger
 User account credentials changed
frequently
 VM configuration issues
 REST API calls timed out
• Hours spent cleaning up Excel
reports every week
But there were flaws…

23. 23© 2016 CSAA Insurance Group. Confidential and proprietary.
Exporting Data out of Dynatrace
• Real time push from Dynatrace
• Anyone with Splunk access
could get performance data
• Limited areas for interruption
• No manual report modification
• Added additional report types
We addressed those flaws
Flume
Dynatrace

24. 24© 2016 CSAA Insurance Group. Confidential and proprietary.
Real Time Performance View

25. 25© 2016 CSAA Insurance Group. Confidential and proprietary.
Real Time Performance View

26. 26© 2016 CSAA Insurance Group. Confidential and proprietary.
Club Performance Reports

27. 27© 2016 CSAA Insurance Group. Confidential and proprietary.
Club Performance Reports

28. 28© 2016 CSAA Insurance Group. Confidential and proprietary.
Club Performance Reports

29. 29© 2016 CSAA Insurance Group. Confidential and proprietary.
Individual User Performance Trends

30. 30© 2016 CSAA Insurance Group. Confidential and proprietary.
Individual User Performance Trends

31. 31© 2016 CSAA Insurance Group. Confidential and proprietary.
Making Data Available Through Models

32. 32© 2016 CSAA Insurance Group. Confidential and proprietary.
Feeding Back Into Development

33. 33© 2016 CSAA Insurance Group. Confidential and proprietary.
Going Beyond Just Monitoring
• Easily searchable data repository
• Quick long term trending
• Extensible…

34. 34© 2016 CSAA Insurance Group. Confidential and proprietary.
Going Beyond Just Monitoring
• Easily searchable data repository
• Quick long term trending
• Extensible…

35. 35© 2016 CSAA Insurance Group. Confidential and proprietary.
Updating Testing Plans
• Using production
transaction frequency
• More realistic performance
testing

36. 36© 2016 CSAA Insurance Group. Confidential and proprietary.
Performance Comparisons
• Typically baseline to
baseline comparison in
the save environment
• Differences between
production and
performance
environments
• More accurate
understanding forecasts

37. 37© 2016 CSAA Insurance Group. Confidential and proprietary.
Lessons Learned

38. 38© 2016 CSAA Insurance Group. Confidential and proprietary.
Our Splunk Architecture
Splunk started out as a PAS only tool.
PAS

39. 39© 2016 CSAA Insurance Group. Confidential and proprietary.
Our Splunk Architecture
But quickly other teams in the enterprise began using it too.
PAS
SOA Shared Services

40. 40© 2016 CSAA Insurance Group. Confidential and proprietary.
Our Splunk Architecture
PAS
SOA Shared Services Enterprise Analytics
But quickly other teams in the enterprise began using it too.

41. 41© 2016 CSAA Insurance Group. Confidential and proprietary.
Our Splunk Architecture
PAS
SOA Shared Services Enterprise Analytics
UBI
But quickly other teams in the enterprise began using it too.

42. 42© 2016 CSAA Insurance Group. Confidential and proprietary.
Our Splunk Architecture
PAS
SOA Shared Services Enterprise Analytics
UBI
Digital Services
But quickly other teams in the enterprise began using it too.

43. 43© 2016 CSAA Insurance Group. Confidential and proprietary.
Our Splunk Architecture
PAS
SOA Shared Services Enterprise Analytics
UBI
Digital ServicesMRM
But quickly other teams in the enterprise began using it too.

44. 44© 2016 CSAA Insurance Group. Confidential and proprietary.
Keeping up with the times
Splunk added the HTTP Event Collector in Splunk 6.3
Flume
Dynatrace
Dynatrace

45. 45© 2016 CSAA Insurance Group. Confidential and proprietary.
Questions?

46. 46© 2016 CSAA Insurance Group. Confidential and proprietary.
Thank you!