1.
GAINING APPLICATION LIFECYCLE
INTELLIGENCE WITH SPLUNK
By Damien Dallimore , Dev Evangelist @ Splunk
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/

2.
Who am I ?
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
2

3.
From Middle Earth
JVM background
Make things
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 3

4.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 4

5.
apps.splunk.com
github.com/damiendallimore
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 5

6.
Agenda
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
6

7.
Overview of Splunk and build a simple app
How Splunk can help in the Application Development Lifecycle
Various ways to get data into Splunk and demos
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 7

8.
Data Data Everywhere
VELOCITY
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
8
VOLUME
VARIETY
VERACITY

9.
How can Splunk help ?
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/

10.
Spelunking
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
10

11.
Platform for machine data
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
11
Developer
Platform
Splunk storage Other Big Data stores
Data collection
and indexing
Report
and
analyze
Custom
dashboards
Monitor
and alert
Ad hoc
search

12.
Any amount, any location, any source.
Schema at read time, not write time
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
Developer
Platform
Report
and
analyze
Custom
dashboards
Monitor
and alert
Ad hoc
search
Platform for machine data
12
Splunk storage Other Big Data stores
Data
collection
and indexing
Data in any format
No RDBMS
Very Extensible

13.
What Does Machine Data Look Like?
Order Processing
Middleware
Error
Care IVR
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
13
Sources
Twitter

14.
Machine Data Contains Critical Insights
Order Processing
Middleware
Error
Care IVR
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
14
Customer ID Order ID
Customer’s Tweet
Time Waiting On Hold
Twitter ID
Product ID
Company’s Twitter ID
Sources
Twitter
Order ID Customer ID
Customer ID

15.
Machine Data Contains Critical Insights
Order Processing
Middleware
Error
Care IVR
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
15
Order ID
Customer’s Tweet
Time Waiting On Hold
Product ID
Company’s Twitter ID
Sources
Twitter
Order ID
Customer ID
Twitter ID
Customer ID
Customer ID

16.
How are we best
going wrangle this
data ?
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 16

17.
Release the Developers
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
17

18.
Very Extensible Platform for Developers
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
18
REST API
Django
Web
Framework
Java
JavaScript
Python
Ruby
C#
PHP
Data Models
Search
Extensibility
Modular Inputs
SDKs

19.
Lets build something simple
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
19

20.
Simple Swarm App (ex Foursquare)
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
20
Get my actual checkin data in via REST
Search over this data
Visualize

21.
Application Lifecycle Data
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
21

22.
Application Development Challenges
Lack of visibility across the product
development lifecycle
Build
Unit Testing
Code
Pressure to increase velocity and
agility with DevOps
Limited insights into behavior and
Check-in
performance from application logs
Integration
Testing Deploy
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
Staging
22

23.
Splunk for Application Lifecycle Intelligence
FIND AND FIX
ISSUES FASTER
Quickly trace and identify errors anywhere
in the codebase with real-time search
and monitoring
Instrument your app logs to gain
application intelligence
GAIN END-TO-END VISIBILITY
ACROSS THE DEV TOOL CHAIN
Break down dev tool silos with real-time
insights from machine data
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
PUSH BETTER
CODE USING
ANALYTICS
23

24.
Real-time dashboards show error
rate in production and impact of
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
pushing new builds
Developers can search and
visualize web logs, Java logs—
without production access
Alerts notify developers as soon as
a problem arises
24
Find and Fix Issues Faster

25.
Push Better Code Using Analytics
Gain end-to-end visibility to make
informed decisions
Analytics insights without the need
for additional analytics tools
Ask questions while exploring and
collecting data
void submitPurchase(purchaseId)
{
log.info("action=submitPurchaseSt
art, purchaseId=%d", purchaseId)
//these calls throw an exception on error
submitToCreditCard(...)
generateInvoice(...)
generateFullfillmentOrder(...)
log.info("action=submitPurchaseCo
mpleted, purchaseId=%d", purchaseId)
}
25
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/

26.
End-To-End Visibility Across The Dev Tool Chain
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
26
CI / Build
Servers
Project and
Issue Tracking
Code
Repository
QA / Testing
Tools
Deployment
Servers

27.
App Development Lifecycle Demo
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
27

28.
Getting your data into Splunk
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
28

29.
Log Data
Log Files
Splunk Logging Appenders
Coding
Splunk Java SDK
Splunk Spring Integration Adaptors
JMX
Messaging
JMS
AMQP w/Rabbit
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 29

30.
Log Data
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
30

31.
Standard Log Files
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
31
Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1153 ms
Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Application logs that are part of the product
Developer logs for any code that was deployed
Written to local disk or network storage

32.
Structured and Unstructured Data
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
32
Ideally events are in a best practice semantic format
key=value format , JSON
You can perform index time and search time extractions in Splunk

33.
Logging best practices
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
33
Clearly timestamp every event , human readable at beginning of line
Log in text , binary needs decoding
Categorize – Use INFO, WARN, ERROR, DEBUG, Event type etc...
Log unique identifiers
Log anything that can add value when aggregated, charted or further
2012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar"

34.
SplunkJavaLogging
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
34
Sometimes you can’t write to file
Appenders for Java Util Logging , Log4J , Logback
Simply add a logging appender to your logging configuration
file

35.
LogBack Appender Example
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
35

36.
Code
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
36

37.
Better Exception Logging
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
37

38.
Easier to work with in Splunk
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
38

39.
Coding
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
39

40.
Splunk SDK for Java
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
40
Use the SDK from any JVM Language , Java / Groovy / Scala etc….
Send log events via REST , UDP or TCP directly to Splunk from your
code
Search over data in Splunk
SDK available from dev.splunk.com

41.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 41

42.
Spring Integration Adaptors
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
42
Inbound Adapter
Used to execute Splunk searches and get data out
Outbound Adapter
Write data to Splunk via REST, TCP , UDP
Write to a named index, submit a REST request, write to a data input bound
to a server TCP port
Get the code on Github

43.
Inbound Adaptor
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
43

44.
Outbound Adaptor
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
44

45.
JMX
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
45

46.
What is JMX
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
46
JMX = Java Management Extensions
Monitor JVM via MBean attributes , operations and
notifications
JVM MBeans
Vendor MBeans
Custom Coded MBeans

47.
Getting this data into Splunk
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
47
Runs on all supported Splunk platforms
Works with all main JVM variants
100% Free and Open Source

48.
Simple to Configure
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
48

49.
Many Connectivity Options
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
49

50.
Messaging
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
50

51.
JMS
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
51
Not a messaging protocol , but a programming interface to many
different underlying message providers
WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc…

52.
AMQP
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
52
Built with Rabbit Java Client Library
AMQP 0.9.1, 0.9, 0.8

53.
But wait , there’s more…..
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
53

54.
Poll data from any REST API
Pull data directly off the wire
Capture output from executing any commands
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 54

55.
Closer look at capturing command output
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
55
Let’s see what the host Operating System can tell us : top
External programs that provide additional JVM insights : jstat
Index this data in Splunk and correlate

56.
top
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
56

57.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 57

58.
jstat
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
58

59.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 59

60.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/ 60

61.
Splunk options galore
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
61
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.
Splunk> AMI : BYOL versions for Amazon AWS Cloud.
Splunk> Sandbox : Spin up a cloud instance in minutes.Load in data.
Hunk> : Splunk for data in Hadoop HDFS , MongoDB

62.
More Info
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/
62
Splunk Docs , Downloads , Vids : http://www.splunk.com
Download Splunk Apps : http://apps.splunk.com
Ask : http://answers.splunk.com
Watch the App Dev / Devops Video : http://www.splunk.com/goto/appdev
Splunk Developer Platform : http://dev.splunk.com
Splunk on Github : https://github.com/splunk

63.
Thankyou.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/

64.
Questions ?
@damiendallimore
ddallimore@splunk.com
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Sof tware, Inc. and licensed under a
Creat ive Commons At tribut ion-NonCommercial license: ht tp: / /creat ivecommons.org/ licenses/by-nc/3.0/