Open Source Monitoring for Java with JMX and Graphite - Softshake 20131. Open Source Monitoring for Java
with Graphite
Cyrille Le Clerc
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
3. You ?
DEV ? QA ? OPS ?
Monitoring Experience ?
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
3
5. Agenda
• The demo application
• Monitoring performance: Why? What?
• Java webapp monitoring with JMX
• Monitoring with Graphite
• Conclusion
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
5
7. Your mission, should you decide to accept it, ...
e-commerce SOS Cocktails
http://www.flickr.com/photos/23791504@N08/2607814349
Setup technical and business monitoring
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
7
10. The Indicators to Monitor
http://www.langevin.com/blog/2013/08/29/3-ways-to-maximize-brainstorming/
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
10
11. The Indicators to Monitor
• Operating System ➔ Sysload
• JVM ➔ GC duration
• Tomcat ➔ activeSessions (active visitors)
• Application ➔ sales revenue & items
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
11
13. Open Source Monitoring for Java
Dashboard - wiki
e-commerce web site
Graphite
Email
Seyren
Nagios
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
13
15. Why JMX ?
• JMX vs. web pages vs. logs vs. ...
• Simple and secured
• Displayable with web pages and logs
• Alternatives to JMX ?
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
15
16. JMX vs. logs vs. web pages vs. ...
•
Monitoring logs
•
Parsing GB of text files ?
•
Per invocation details vs. average per minute
• Web pages
•
History ?
•
Clustered applications ?
•
Security ?
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
16
17. JMX with Spring Framework
<beans
...>
...
<context:mbean-‐export/>
...
@ManagedResource("cocktail:name=ShoppingCartController,type=...")
</beans>
class
ShoppingCartController
{
final
AtomicInteger
revenueInCentsCounter
=
new
AtomicInteger();
void
purchase(...){
...
revenueInCentsCounter.addAndGet(priceInCents);
}
@ManagedAttribute
public
int
getRevenueInCentsCounter()
{
return
revenueInCentsCounter.get();
}
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
17
19. JMX with JSP
Human readable & script friendly
Watch out for security !
$
curl
http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-‐sales-‐
metrics.txt.jsp
Epoch
SalesRevenueInCentsCounter
SalesItemsCounter
SalesOrdersCounter
1363217376
4500
7
2
http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp
curl
http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/...
{
"Epoch":1363217653,
"SalesRevenueInCentsCounter":4050,
"SalesItemsCounter":7,
...
}
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
19
20. JMX with JSP
/5ADrUWr5/jmx-sales-metrics.jsp
Human readable & script friendly
Watch out for security !
$
curl
http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-‐sales-‐
metrics.txt.jsp
Epoch
SalesRevenueInCentsCounter
SalesItemsCounter
SalesOrdersCounter
1363217376
4500
7
2
http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp
curl
http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/...
{
"Epoch":1363217653,
"SalesRevenueInCentsCounter":4050,
"SalesItemsCounter":7,
...
}
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
19
23. Alternatives to JMX
r();
ervo
ntege
R)
S
{
micI
UNTE
oller ",
type=CO
=
new
Ato
Contr Cents
Cart
nter
pping revenueIn
tsCou
s
Sho ame="
InCen
clas
enue
r
rev
or(n
onit
ntege
@M
micI
l
Ato
fina
nts);
...){
eInCe
ase(
pric
purch
dGet(
void
addAn
ter.
..
sCoun
.
nCent
nueI
reve
}
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
21
24. Alternatives to JMX
Metrics
class
Shopp
ingCartCont
r();
roller
{
ervo
ntege
R)
S
{
micI
UNTE
fController ",
type=CO
=
new
Ato
inal
Countes
rt
Ca
ueInCentr
revenueInCents
nter
pping reven
InCentsCou
Counter
=
M
ShoppingCa
s
Sho ame="
etrics.newC
clas
enue
rtControlle
r
rev
ounter(
or(n
e
t
r,
"revenue
teg
@Moni tomicIn
void
-‐in-‐cents")
purchase(..
;
al
A
n
.){
fi
...
nts);
...){
eInCe
ase(
revenu
pric
purch
eInCedAndGet(
void
.adntsCounter.inc(pr
} Counter
iceInCents)
...
ts
;
InCen
enue
rev
}
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
21
25. JMX
• OS: Sysload
• JVM: Garbage Collector
• Tomcat: activeSessions
• Application: Web Site Revenue
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
22
30. standalone vs. embedded vs. agent
standalone
embedded
agent
Packaging
Model
Manually connected apps
Pull
Standalone apps
Push
Standalone middleware
Push
Cursor
OPS -♦----------- DEV
OPS ----------♦-- DEV
OPS ---♦--------- DEV
Use case
Net unfriendly monitoring,
OPS
cloud, batch
DEV/QA
cloud, batch
OPS
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
27
32. embedded-jmxtrans configuration
<dependency>
<groupId>org.jmxtrans.embedded</groupId>
<artifactId>embedded-‐jmxtrans</artifactId>
<version>1.0.8</version>
</dependency>
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
28
33. embedded-jmxtrans configuration
<dependency>
<groupId>org.jmxtrans.embedded</groupId>
<artifactId>embedded-‐jmxtrans</artifactId>
<version>1.0.8</version>
</dependency>
<beans
...
xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"
xsi:schemaLocation="...
http://www.jmxtrans.org/schema/embedded
...”>
<jmxtrans:jmxtrans>
<jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>
...
</jmxtrans:jmxtrans>
</beans>
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
28
34. embedded-jmxtrans configuration
{
<dependency>
"queries":
[
<groupId>org.jmxtrans.embedded</groupId>
{
<artifactId>embedded-‐jmxtrans</artifactId>
"objectName":
"cocktail:name=ShoppingCartController",
<version>1.0.8</version>
"resultAlias":
"sales",
</dependency>
"attributes":
["SalesRevenueInCentsCounter"]
},
],
"outputWriters":
[
{
<beans
...
"@class":
"org.jmxtrans.embedded.output.GraphiteWriter",
xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"
"settings":
{
xsi:schemaLocation="...
"host":
"${graphite.host:localhost}",
http://www.jmxtrans.org/schema/embedded
...”>
"port":
"${graphite.port:2003},"
"enabled":
"${graphite.enabled:true}"
<jmxtrans:jmxtrans>
}
<jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>
}
...
]
</jmxtrans:jmxtrans>
}
</beans>
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
28
36. Graphite
• Open Source
• Time series database and graphing tool
• Used by tech giants
• Similar
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
30
37. Graphite
• Open Source
Simplicity and Self Service
• Time series database and graphing tool
• Used by tech giants
• Similar
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
30
38. Metrics injection / Write Access
• Automatic metrics creation
• Socket protocols: “plain text” & Python Pickle serialization
echo
"softshake.happyCounter
1.2
1364338989"
|
nc
localhost
2003
• Precision and storage duration by configuration
[sales_1min_for_30days_5min_for_1year]
pattern
=
^sales.
retentions
=
60s:30d,300s:365d
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
1.7MB
31
39. Metrics rendering / Read Access
• Graph Composer & URL API
• Everybody can create graphs
• Wide range of functions
•
•
•
sum, scale, derivative, timeshift, deviation, filter, ...
Various exports
png, svg, csv, json, raw text
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
32
40. The Art of Graphing
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
33
41. Monitoring on the Cloud
•
Graphite as a Service
•
http://hostedgraphite.com/
•
Addons: Tasseo
2
Zero setup
•
1
email: admin@jmxtrans.org
password: SOFTSHAKE
3
• No-OPS
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
34
42. Metrics Types
Gauge
Ever Increasing Counter
• Examples:
• Information available without transformation
• Examples:
• “per minute” aggregation required
activeRequests, dataSource.activeConnection, ...
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
requestsCount, revenue, ...
35
43. Graphs and Maths
?
Ever Increasing Counter
Revenue per Second
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
36
45. Graphs and Maths
?
Per Second → Per Hour
Revenue per Hour
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
37
47. Graphs and Maths
?
2 servers
Total Revenue per Hour
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
38
49. Graphs and Maths
?
server restart
Ignore reset to zero
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
39
52. Graphs and Maths
Compare to
last week
timeShift(..., “7d”)
?
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
40
53. JMX
• OS: Sysload
• JVM: Garbage Collector
• Tomcat: activeSessions
• Application: Web Site Revenue
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
41
54. Graphite URL API
http://localhost:8081/render/?
from=-‐7days&
title=Revenue%20per%20Hour&
vtitle=Dollars&
lineWidth=3&
xFormat=%25a%20%25H%25p&
target=
alias(
scale(
summarize(
sumSeries(
nonNegativeDerivative(
edu.servers.*.sales.revenueInCentsCounter)...)&
...
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
42
55. Sharing metrics and graphs
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
43
56. Dashboards and Wikis
dashboard.prod.md
#
Website
Traffic
<table>
<tr>
<td>
<img
src="http://graphite.prod.mycompany/graphite/render/?
width=400&height=300&title=Revenue&xFormat=%25a%20%25d
%25p&vtitle=USD&lineWidth=3&from=-‐5days&target=alias(scale(summarize(sumSeries(non
NegativeDerivative(edu.servers.*.sales.revenueInCentsCounter))%2C%221h
%22)%2C0.01)%2C%22Revenue%20per%20Hour%22)...">
</td>
<td>
<img
src="http://graphite.prod.mycompany/graphite/render/?...">
</td>
<tr>
</table>
Wiki
https://github.com/jmxtrans/embedded-jmxtrans-samples/wiki/Dashboard-PROD
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
44
59. Graphite Integration with Alerting
• Based on Graphite URL API
• RawText or JSON format
• Pattern
/render?from=-‐11minutes&until=-‐1minutes&format=raw&target=****
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
47
60. Graphite Integration with Alerting
• Based on Graphite URL API
• RawText or JSON format
• Pattern
/render?from=-‐11minutes&until=-‐1minutes&format=raw&target=****
Example
$
curl
"http://graphite.example.com/render?
from=-‐11minutes&until=-‐1minutes&format=raw&
target=keepLastValue(servers.cloudbees.jvm.os.SystemLoadAverage)"
my-‐metric,1363225680,1363226340,60|0.03,0.01,0.1,4.0,4.0,0.9,0.7,0.8,0.4,0.5,0.5
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
47
61. Graphite Alerting with Seyren
• Alerting Dashboard for Graphite
• Open Source
• Java .war + MongoDB
• Alerts: email, PagerDuty, ...
http://seyren.jmxtrans.cloudbees.net/
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
48
62. Graphite Alerting with Nagios
• Monitoring Infrastructure
• Open Source
• De facto standard
• check_graphite plugin
•
Pierre-Yves Ritschard
•
Jason Dixon / obfuscurity
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
49
64. Conclusion
•
Self-service monitoring changes everything
•
Technical and business monitoring
•
Monitoring integrated in the Continuous Delivery pipeline
•
Open Source solutions are available
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
51
65. Hosted Graphite Promo
• http://hostedgraphite.com
• 45 days trial with “SOFTSHAKE” promo code
(3) Promo Code “SOFTSHAKE”
(1) Signup
(2) Options / Enter Promo Code
help: cleclerc@cloudbees.com
©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13
52