This document discusses how big data analytics tools like Splunk can be used to accelerate the software development lifecycle for large public sector applications. It provides examples of how Splunk was used to improve productivity by enabling immediate log access across many servers and files. Splunk also created real-time performance dashboards to help identify root causes of issues. Additional analytics revealed insights like peak usage times and patterns, user behaviors on forms, and browser/device details. The summary concludes that these tools can improve IT and business while providing lessons on proper Splunk setup and logging the right application data.

1. Accelerating SDLC for Large Public Sector
Enterprise Applications
Using Big Data analytics tools to build and operate
Large Enterprise Applications
Vinky Mehta
Specialist Leader
Technology/System Integration/Public Sector
October 15th, 2015

2. 2 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Contents
Introduction 3
Deloitte Public Sector practice 4
Large Public Sector Project profile 5
i. Improve Productivity 6
ii. Optimize Performance 9
iii. Enhance Business Functionality 12
Summary 20

3. 3 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Background and Experience
• 30 Years Software industry Experience
• 11 Years with Deloitte
• Industry focus : Public sector High Performance Large Enterprise Applications.
• Technology Focus :
• Enterprise Architecture, Performance analysis and optimization
• SOA /BPM : Published papers, Three U.S. patents, Editor for two SOA standards
• Bachelor of Technology in Mechanical Engineering from IIT, Bombay.
Vinky Mehta
Specialist Leader, Deloitte
Public Sector / System Integration
vmehta@deloitte.com
Architect and Deliver High Performance Large Custom applications for Public Sector Clients
Introduction

4. 4 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Deloitte Public Sector Technology practice

5. 5 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Complex business critical applications, Complex stack
◦ 1 to 8 million lines of code
◦ Security Server, HTTP Server, Application server, Web services server, MQ server, Process
Server, Oracle database (all servers clustered)
Parallel development
◦ Up to 4 Parallel releases
◦ 10 - 200 builds / month
◦ 4 – 19 development environments
Large development team:
◦ 50 – 200 developers
High volume, High performance, High availability, Large user base
◦ 10 – 160 million transactions/month
◦ ~0.25 to 2 second average response time
◦ >99.9% availability
◦ 100 to 10,000+ internal users; 1000 to 1,000,000+ external users
Deloitte Large Public Sector Projects profile

6. Improve Productivity

7. 7 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
SDLC Challenge for Large Applications
Challenges
Application logging in a large clustered environment gets very voluminous
• 10 (log files / clone) x (10 to 128) clones ≈ 100 to 1,000+ log files
• Performance logging ≈ 0.3 million to 10 million transactions / day
Log file access can take up to a day. Why?
• Identify correct log file among 1,000s of files
• Copy logs from production to developers’ desktop
• Coordination between development and operations team
• Manual work
• Overwhelming network bandwidth
Approach Splunk deployed in Development, Test and Production environment
Results
Log data accessible through Splunk
• Immediate log access (1-3 minute delay) through secure, role-based login
• Information correlated across 1,000+ log files
• Overall significantly improved production operations productivity!

8. 8 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Search - Visualize, Filter, Analyze
What techniques are available to perform root cause analysis?
• Visualize trends to spot abnormal patterns
– Compare change across production code releases
– Usage trend over time
• Search filters targeted to specific pages, URLs or error type
• Drill down and analyze details down to the raw log data entry

9. Optimize Performance

10. 10 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Application Performance Dashboard
What kind of real-time metrics can we view?
• Key Splunk feature–Built-in support to quickly develop dashboard views
• Real-time executive level view
• Extension points with Splunk WebSDK

11. 11 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Performance Dashboard
How can it help isolate root cause?
• Key feature of Splunk – event metadata such as time, source, host, etc. help to isolate root cause
• Is slowness on particular node or server instance?
• Is slowness because of network, database or application server?
• Is performance comparable to the past?

12. Enhance Business
Functionality

13. 13 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
When is the application used?
Analyze : Transaction Volume
Peak load is during lunch times. But usage at all hours (24x7)

14. 14 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
What kind of machines are used by users?
Analyze : Users Machines
Windows machine are prevalent
But users are using Mobile devices (Android, iPhone)
Even Mac and Linux machines

15. 15 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
What are the types and versions of browsers used by users?
Analyze : Users Browser Versions
Many versions of IE in use from IE 7.0 to IE 11.0
Almost all major browsers and versions in use
Users are using Mobile Safari 7.0 and Android Webkit 4.0

16. 16 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Completing application
Analyze : User Behavior
Most submit the form in first attempt
Some users saved their work and exited to work later
Some users (~25%) did not bother to explicitly save and exit

17. 17 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
How long does it take to submit a new application?
Analyze – Form submission patterns
Form input and submission time vary from 5 minutes to several hours.
• Next drill down, complexity, number if records,

18. 18 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
What stage are users leaving the Form?
Analyze : Application abandonment pattern
Most forms make it all the way to the last submission page.

19. 19 Big Data Analytics Tools to Accelerate SDLC Copyright © 2015 Deloitte Development LLC. All rights reserved.
Summary
• Big Data Analysis Tools of Large Enterprise Applications benefits IT & Business
• Accelerate SDLC - Improved productivity for Dev, Test, Operate
• Business Insight - ask new questions to challenge old business models
• Lessons learned
• Splunk has low learning overhead for base functionality
• Plan for adequate time for initial Splunk setup
• Creating dashboards and sophisticated reports is quick but requires specialized
skills (custom syntax)
• Deep understanding of your application and logging the right data is essential to
achieve the real benefits from this technology.

20. Copyright © 2014 Deloitte Development LLC. All rights reserved.
36 USC 220506
Member of Deloitte Touche Tohmatsu Limited