Almost all developers face the challenge of reactively debugging failed business transaction processes. Not only does this require extensive navigation of enormous volumes of log data, but determining root cause becomes a laborious and time-consuming task. Additionally, business managers often request developers and operations to provide analytics on applications, resulting in the tedious task of charting the information, most usually from intangible data. Learn how to capture, extract and analyze your event data by having analytics embedded in the application. Download the white-paper that details how to gain Application Intelligence through effective logging. Check out the webinar here: http://www.splunk.com/goto/analytics_webcast

1. Don’t
Rewrite
Code
to
Get
BeCer
AnalyEcs
Archana
Ganapathi
Research
Engineer
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

2. AnalyEcs
Can
Be
Challenging!
• Modern
systems
are
distributed
and
heterogeneous
• Consolidate
informaEon
• Analyzing
across
a
distributed
architecture
• AnalyEcs
is
limited
to
informaEon
that
is
made
“available”
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

3. Typical
Architecture
ApplicaEons
Data
Direct
Insert
Warehouse
BI,
AnalyEcs,
ReporEng
Tool
ETL
Database
Connector
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

4. Development
Cycle
Early
Structure
Binding
Decide
the
quesEons
you
want
to
ask
Design
the
Schema
Normalize
the
data
and
Write
DB
inserEon
code
SELECT
customers.*
FROM
customers
WHERE
Create
SQL
&
feed
into
AnalyEcs
Tool
customers.customer_id
NOT
IN(SELECT
customer_id
FROM
orders
WHERE
year(orders.order_date)
=
2004)
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

5. A
Paradigm
Change:
Use
Your
Log
Files
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

6. Using
Log
Files
!Log.debug(“orderstatus=error,errorcode=454,!
!user=%s,transactionid=%d”, userId, transId)!
ü
You
already
log
key
informaEon
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

7. Using
Log
Files
They
contain
a
gold
mine
of
informaEon
• DefiniEve
record
of
acEvity
and
behavior
• Ensure
system
security
• Meet
compliance
mandates
User
IP
AcEon
Login
Result
10.2.1.44 - [25/Sep/2009:09:52:30 -0700]
type=USER_LOGIN msg=audit(1253898008.056:199891):
auid=4294967295 msg='acct="TAYLOR": exe="/usr/sbi
addr=10.2.1.48, terminal=sshd res=failed)'
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

8. Using
Log
Files
They
contain
a
gold
mine
of
informaEon
• Important
insight
for
IT
and
the
business
• Customer
behavior
and
experience
• Product
and
service
usage
User
IP
Product
Category
• End-­‐to-­‐end
transacEon
visibility
10.2.1.80 - - [25/Jan/2010:09:52:30 -0700]
"GET /petstore/product.screen
?product_id=AV-CB-01 HTTP/1.1" 200 9967 "http://10
category.screen?category_id=BIRDS" "Mozilla/5.0 (co
Linux)”"JSESSIONID=xZDTK81Gjq9gJLGWnt2NXrJ2tpGZb1Hy
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

9. They
Help
You
Find
Problems
Apr 29 19:13:01 45.2.98.7 SentriantGenericAlert: Time="04/29/06 07:12 PM
PDT",Host="roach_motel.enet.interop.net",Category="fabric_network_activity",Generato
r="Response:Slow
Scan",Type="NOTICE",Priority="High",Body="Appliance=roach_motel.enet.interop.net,Rep
orting Segment=ENET network,Action=Response disabled,Response=Slow Scan,Duration=90
seconds,Source Segment=Unprotected,Source IP=88.73.39.200,Source MAC=00:01:30:BC:
93:90,Current Target Count=0"!
45.2.98.7
Apr 29 19:13:01 45.2.98.7 SentriantGenericAlert: Time="04/29/06 07:12 PM
SentriantGe
PDT",Host="roach_motel.enet.interop.net",Category="fabric_network_activity",Generato
r="Response:Slow nericAlert:
Time="04
Scan",Type="NOTICE",Priority="High",Body="Appliance=roach_motel.enet.interop.net,Rep
orting Segment=ENET network,Action=Response disabled,Response=Slow Scan,Duration=69
seconds,Source Segment=Unprotected,Source IP=68.163.20.95,Source MAC=00:01:30:BC:
93:90,Current Target Count=0"!
Apr 29 19:13:01 45.2.98.7 SentriantGenericAlert: Time="04/29/06 07:12 PM
PDT",Host="roach_motel.enet.interop.net",Category="fabric_network_activity",Generato
r="Response:Slow!
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

10. Machine-­‐generated
Events
are
Everywhere
AddiEonal
Sources
Core
IT
Customer-­‐facing
IT
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

11. Splunk:
The
Plaiorm
for
Machine
Data
Customer
Outside
the
Facing
Data
Datacenter
" Click-­‐stream
data
" Manufacturing,
" Shopping
cart
data
logisEcs…
" Online
transacEon
" CDRs
&
IPDRs
data
" Power
consumpEon
Logfiles
Configs
Messages
Traps
Metrics
Scripts
Changes
Tickets
" RFID
data
Alerts
" GPS
data
Windows
Linux/Unix
Virtualiza7on
Applica7ons
Databases
Networking
" Registry
" ConfiguraEons
&
Cloud
" Web
logs
" ConfiguraEons
" ConfiguraEons
" Event
logs
" syslog
" Hypervisor
" Log4J,
JMS,
JMX
" Audit/query
logs
" syslog
" File
system
" File
system
" Guest
OS,
Apps
" .NET
events
" Tables
" SNMP
" sysinternals
" ps,
iostat,
top
" Cloud
" Code
and
scripts
" Schemas
" neilow
Burlingame,
March
8,
2012
Copyright
©
2011,
Splunk
Inc.
Listen
to
your
data.
Copyright
©
2012,
Splunk
Inc.

12. Splunk
Collects
and
Indexes
Any
Machine
Data
Customer
Outside
the
Facing
Data
Datacenter
" Click-­‐stream
data
" Manufacturing,
" Shopping
cart
data
logisEcs…
• Any
amount,
any
locaEon,
any
source.
" Online
transacEon
" CDRs
&
IPDRs
data
" Power
consumpEon
Logfiles
Configs
Messages
Traps
Metrics
Scripts
Changes
Tickets
" RFID
data
GPS
data
No
upfront
schema
Alerts
"
No
custom
connectors
Windows
Linux/Unix
Virtualiza7on
Applica7ons
Databases
Networking
" Registry
" ConfiguraEons
&
Cloud
DBMS
Web
logs
No
R " ConfiguraEons
" " ConfiguraEons
" Event
logs
" syslog
" Hypervisor
Log4J,
JMS,
JMX
" Audit/query
logs
" " syslog
"
"
File
system
sysinternals
"
"
File
system
ps,
iostat,
top
No
nS,
Apps
to
filter/forward
Tables
"
"
Guest
O eed
Cloud
.NET
events
"
Code
and
scripts
"
"
Schemas
"
"
"
SNMP
neilow
Burlingame,
March
8,
2012
Copyright
©
2011,
Splunk
Inc.
Listen
to
your
data.
Copyright
©
2012,
Splunk
Inc.

13. A
Single
Plaiorm
for
OperaEonal
Intelligence
Single
Data
Store
Single
UI
Across
Use
Cases
Three
Primary
CapabiliEes
Search
/
Naviga7on
Real-­‐7me
Visibility
Historical
Analy7cs
• Data
drilldown
• Live
dashboards
• Baseline
and
thresholds
• “Needle
in
a
haystack”
• Event
correlaEon
• Trending
• Root
cause
analysis
/
• Monitoring
and
alerEng
• OperaEonal
insights
troubleshooEng
• Performance
issues
• Historical
paCerns
• Incident
invesEgaEons
• TransacEon
levels
• Compliance
reports
• SLA
tracking
Burlingame,
March
8,
2012
Copyright
©
2011,
Splunk
Inc.
Listen
to
your
data.
Copyright
©
2012,
Splunk
Inc.

14. Real
Business
Value
with
OperaEonal
Metrics
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

15. Intelligence
on
your
ApplicaEons
with
Splunk
Log
Files
ApplicaEon
OperaEonal
Database
Intelligence
Java
EE
Server
+
AnalyEcs
Unix
based
OS
Unix
based
OS
+
ReporEng
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

16. An
AlternaEve
Development
Cycle
Apr 29 19:13:01 45.2.98.7 entriantGenericAlert:
Late
Structure
Binding
Time="04/29/06 07:12 PM PDT”,
Host="roach_motel.enet.interop.net",Category="fabric_
network_activity",Generator="Response:Slow
Write
events
to
your
log
files
Scan",Type="NOTICE",Priority="High",Body="Appliance=r
oach_motel.enet.interop.net,Reporting Segment=ENET
network,Action=Response disabled,Response=Slow
Collect
log
files
Scan,Duration=90 seconds,Source
Segment=Unprotected,Source IP=88.73.39.200,Source
MAC=00:01:30:BC:93:90,Current Target Count=0"!
Apr 29 19:13:01 45.2.98.7 SentriantGenericAlert:
Create
searches,
graphs
and
reports
Time="04/29/06 07:12 PM
PDT",Host="roach_motel.enet.interop.net",Category="fa
bric_network_activity",Generator="Response:Slow
Scan",Type="NOTICE",Priority="High",Body="Appliance=r
oach_motel.enet.interop.net,Reporting!
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

17. “SemanEc
Logging”
Events
which
are
wriCen
explicitly
for
the
gathering
of
analyEcs
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

18. A
Simple
Example
void submitPurchase(transctionID) !
{ !
log.info("action=submitPurchaseStart, transactionId=%d",
transactionID, “ productId=%s”, productId, “ listPrice=%dn”, listPrice)!
!
//these calls throw an exception on error!
submitToCreditCard(...)!
generateInvoice(...)!
generateFullfillmentOrder(...)!
!
log.info("action=submitPurchaseStop, transactionID=%dn",
transactionID)!
} !
!
!
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

19. AnalyEcs
QuesEons
Enabled
ü Purchase
volume
by
hour,
by
day,
by
month
ü How
long
are
purchases
taking?
ü Are
my
purchases
taking
longer
than
they
did
last
month?
ü Are
my
systems
geong
slower?
ü How
many
purchases
are
failing?
ü Which
specific
purchases
are
failing?
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

20. AnalyEcs
Dashboard
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

21. Streaming
Radio
Example
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

22. Group
TransacEons
sourcetype=radiolog | transaction IPAddress
startswith="play" endswith="stop"
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

23. Calculate
Concurrency
" sourcetype=radiolog | transaction IPAddress
startswith="play" endswith="stop" | concurrency
duration=duration
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

24. Add
Lookups
and
StaEsEcs
> sourcetype=radiolog | transaction IPAddress startswith="play"
endswith="stop" | concurrency duration=duration | eval key=1 | lookup songs
key | stats first(song) as song max(concurrency) as concurrency by id | stats
sum(concurrency) by song
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.

25. Developer
Concerns
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

26.
Developer
Concern:
Performance
92
sec
15
sec
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

27. Developer
Concern:
Infrastructure
Cost
ü Splunk
Requires
standard
hardware
ü Start
with
an
easy
download
ü Free
Apps
for
domain
specific
analyEcs
ü Proven
in
Big
Data
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

28. Developer
Concern:
Refactoring
Code
ü Start
gradually
and
grow
organically
ü Develop
future
applicaEons
with
analyEcs
and
Splunk
in
mind
ü Build
closer
relaEonships
with
Ops,
Support
and
QA
ü ROI
can
be
priceless
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

29. Developer
Concern:
How
Much
to
Log
Two
approaches
to
event
logs:
ü Log
what
is
evidently
required
ü Open
the
flood-­‐gates
QuanEty
and
granularity
can
vary
based
on
task:
-­‐ Diagnosis
-­‐ ReporEng
-­‐ AnalyEcs
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

30. Logging
Best
PracEces
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

31. Create
Human
Readable
Events
ü Log
in
Text
ü Make
it
easy
for
humans
ü Categorize
ü Avoid
XML
or
JSON
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

32. Clearly
Time
Stamp
Every
Event
ü Do
not
use
Fme
offsets
ü Use
human
readable
Fmestamps
ü Favor
the
beginning
of
the
line
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

33. Use
Clear
Key/Value
Pairs
Example
(Bad):
!Log.debug(“error 454 - %s %d”, userId, transId)!
Example
(Good):
!Log.debug(“orderstatus=error,errorcode=454,!
!user=%s,transactionid=%d”, userId, transId)!
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

34. Break
MulE-­‐Value
InformaEon
Into
Separate
Events
Example
(Bad):
<TS>
phonenumber=415-­‐555-­‐1212,app=angrybirds,facebook
Example
(Good):
<TS>
phonenumber=415-­‐555-­‐1212,
app=angrybirds,
installdate=xx/xx/xx
<TS>
phonenumber=415-­‐555-­‐1212,
app=facebook,
installdate=yy/yy/yy
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

35. Log
Unique
IdenEfiers
ü Allows
to
track
transacEons
in
detail
ü Use
TransiEve
Closure
if
you
need
to:
transid=abcdef,
Transac7on
transid=abcdef,
otherid=
qrstuv,
.
.
.
.
.
otherid=qrstuv
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

36. Using
Header
Lines
for
Keys
<TS>
USER
PID
%CPU
%MEM
VSZ
RSS
TT
STAT
STARTED
TIME
COMMAND
root
41
21.9
1.7
3233968
143624
??
Rs
7Jul11
48:09.67
/System/Library/foo
rdas
790
4.5
0.4
4924432
32324
??
S
8Jul11
9:00.57
/System/Library/baz
.
.
.
.
.
.
.
.
• Splunk
will
interpret
the
column
headers
as
keys
and
each
line
as
values
Copyright
©
2012,
Splunk
Inc.
March
8,
2012
Burlingame,
Listen
to
y©
2012,
Splunk
Inc.
Copyright
our
data.

37. Top
Takeaways
Log
anything
that
can
add
value
when
aggregated
and/or
visualized
Copyright
©
2012,
Splunk
Inc.
37
Listen
to
your
data.

38. Top
Takeaways
Simplify
your
life…
Splunk
logs
for
AnalyEcs
Copyright
©
2012,
Splunk
Inc.
38
Listen
to
your
data.

39. Thanks!
QuesEons?
Copyright
©
2012,
Splunk
Inc.
Listen
to
your
data.