Open Source Monitoring for Java with JMX and Graphite - Softshake 2013

  • 2,530 views
Uploaded on

Fast feedback from monitoring is a key of Continuous Delivery. JMX is the right Java API to do so but it unfortunately stayed underused and underappreciated as it was difficult to connect to …

Fast feedback from monitoring is a key of Continuous Delivery. JMX is the right Java API to do so but it unfortunately stayed underused and underappreciated as it was difficult to connect to monitoring and graphing systems. Throw in the sin bin the poor solutions based on log files and weakly secured web interfaces!

A new generation of Open Source tooling based on JMX and Graphite makes it easy to graph java application metrics, including business indicators, and integrate them to traditional monitoring systems like Nagios.

Following the logic of DevOps, we will look together how best to integrate the monitoring dimension in a project: from design to development, to QA and finally to production on both traditional deployment and in the Cloud. Come and discover how the JmxTrans-Graphite ticket can make your life easier.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,530
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
31
Comments
0
Likes
6

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Open Source Monitoring for Java with Graphite Cyrille Le Clerc ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13
  • 2. /me @cyrilleleclerc CTO Solution Architect Cyrille Le Clerc DevOps, Infra as Code Open Source Cloud ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 2
  • 3. You ? DEV ? QA ? OPS ? Monitoring Experience ? ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 3
  • 4. Agenda ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 4
  • 5. Agenda • The demo application • Monitoring performance: Why? What? • Java webapp monitoring with JMX • Monitoring with Graphite • Conclusion ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 5
  • 6. The Application ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 6
  • 7. Your mission, should you decide to accept it, ... e-commerce SOS Cocktails http://www.flickr.com/photos/23791504@N08/2607814349 Setup technical and business monitoring ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 7
  • 8. SOS Cocktail ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 8
  • 9. Monitoring Why ? What ? ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 9
  • 10. The Indicators to Monitor http://www.langevin.com/blog/2013/08/29/3-ways-to-maximize-brainstorming/ ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 10
  • 11. The Indicators to Monitor • Operating System ➔ Sysload • JVM ➔ GC duration • Tomcat ➔ activeSessions (active visitors) • Application ➔ sales revenue & items ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 11
  • 12. Open Source Monitoring for Java ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 12
  • 13. Open Source Monitoring for Java Dashboard - wiki e-commerce web site Graphite Email Seyren Nagios ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 13
  • 14. JMX & Metrics ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 14
  • 15. Why JMX ? • JMX vs. web pages vs. logs vs. ... • Simple and secured • Displayable with web pages and logs • Alternatives to JMX ? ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 15
  • 16. JMX vs. logs vs. web pages vs. ... • Monitoring logs • Parsing GB of text files ? • Per invocation details vs. average per minute • Web pages • History ? • Clustered applications ? • Security ? ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 16
  • 17. JMX with Spring Framework <beans  ...>      ...      <context:mbean-­‐export/>      ... @ManagedResource("cocktail:name=ShoppingCartController,type=...") </beans> class  ShoppingCartController  {          final  AtomicInteger  revenueInCentsCounter  =  new  AtomicInteger();  void  purchase(...){      ...      revenueInCentsCounter.addAndGet(priceInCents);  }  @ManagedAttribute  public  int  getRevenueInCentsCounter()  {    return  revenueInCentsCounter.get();  } ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 17
  • 18. JMX with VisualVM ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 18
  • 19. JMX with JSP Human readable & script friendly Watch out for security ! $  curl  http://demo-­‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-­‐sales-­‐ metrics.txt.jsp Epoch   SalesRevenueInCentsCounter   SalesItemsCounter   SalesOrdersCounter 1363217376  4500   7   2 http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp curl  http://demo-­‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/... {    "Epoch":1363217653,    "SalesRevenueInCentsCounter":4050,    "SalesItemsCounter":7,  ... } ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 19
  • 20. JMX with JSP /5ADrUWr5/jmx-sales-metrics.jsp Human readable & script friendly Watch out for security ! $  curl  http://demo-­‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-­‐sales-­‐ metrics.txt.jsp Epoch   SalesRevenueInCentsCounter   SalesItemsCounter   SalesOrdersCounter 1363217376  4500   7   2 http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp curl  http://demo-­‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/... {    "Epoch":1363217653,    "SalesRevenueInCentsCounter":4050,    "SalesItemsCounter":7,  ... } ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 19
  • 21. JMX with JSP ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 20
  • 22. Alternatives to JMX ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 21
  • 23. Alternatives to JMX r(); ervo ntege R) S  { micI UNTE oller ",  type=CO  =  new  Ato Contr Cents Cart nter pping revenueIn tsCou s  Sho ame=" InCen clas enue r  rev or(n onit ntege  @M micI l  Ato  fina nts);   ...){ eInCe       ase( pric purch dGet( void   addAn   ter. .. sCoun      . nCent nueI  reve      } ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 21
  • 24. Alternatives to JMX Metrics class  Shopp ingCartCont r(); roller  {         ervo ntege R) S  { micI UNTE  fController ",  type=CO  =  new  Ato inal  Countes rt Ca           ueInCentr  revenueInCents nter pping reven                InCentsCou Counter  =  M  ShoppingCa s  Sho ame=" etrics.newC clas enue rtControlle        r  rev ounter( or(n e t r,  "revenue teg @Moni tomicIn  void     -­‐in-­‐cents") purchase(.. ; al  A n .){  fi      ... nts);   ...){ eInCe       ase(  revenu pric     purch eInCedAndGet( void     .adntsCounter.inc(pr  } Counter iceInCents) ... ts ;       InCen enue    rev    } ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 21
  • 25. JMX • OS: Sysload • JVM: Garbage Collector • Tomcat: activeSessions • Application: Web Site Revenue ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 22
  • 26. jmxtrans ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 23
  • 27. jmxtrans - standalone ecommerce Tomcat logistics Graphite Tomcat accounting Tomcat ... jmxtrans - standalone ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 24
  • 28. embedded-jmxtrans ecommerce Tomcat logistics Graphite Librato Tomcat accounting Tomcat batch ... embedded-jmxtrans ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 25
  • 29. jmxtrans-agent ecommerce ecommerce Tomcat Tomcat JVM JVM JVM ecommerce logistics Tomcat Tomcat JVM JVM ecommerce accounting Tomcat Tomcat JVM JVM Graphite Librato ... batch JVM java -javaagent=jmxtrans-agent.jar=... jmxtrans-agent ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 26
  • 30. standalone vs. embedded vs. agent standalone embedded agent Packaging Model Manually connected apps Pull Standalone apps Push Standalone middleware Push Cursor OPS -♦----------- DEV OPS ----------♦-- DEV OPS ---♦--------- DEV Use case Net unfriendly monitoring, OPS cloud, batch DEV/QA cloud, batch OPS ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 27
  • 31. embedded-jmxtrans configuration ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 28
  • 32. embedded-jmxtrans configuration <dependency>    <groupId>org.jmxtrans.embedded</groupId>    <artifactId>embedded-­‐jmxtrans</artifactId>    <version>1.0.8</version> </dependency> ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 28
  • 33. embedded-jmxtrans configuration <dependency>    <groupId>org.jmxtrans.embedded</groupId>    <artifactId>embedded-­‐jmxtrans</artifactId>    <version>1.0.8</version> </dependency> <beans  ...    xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"    xsi:schemaLocation="...        http://www.jmxtrans.org/schema/embedded  ...”>    <jmxtrans:jmxtrans>        <jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>        ...    </jmxtrans:jmxtrans> </beans> ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 28
  • 34. embedded-jmxtrans configuration { <dependency>    "queries":  [    <groupId>org.jmxtrans.embedded</groupId>            {    <artifactId>embedded-­‐jmxtrans</artifactId>            "objectName":  "cocktail:name=ShoppingCartController",    <version>1.0.8</version>            "resultAlias":  "sales", </dependency>            "attributes":  ["SalesRevenueInCentsCounter"]        },    ],    "outputWriters":  [        { <beans  ...            "@class":  "org.jmxtrans.embedded.output.GraphiteWriter",    xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"            "settings":  {    xsi:schemaLocation="...                "host":  "${graphite.host:localhost}",        http://www.jmxtrans.org/schema/embedded  ...”>                "port":  "${graphite.port:2003},"                "enabled":  "${graphite.enabled:true}"    <jmxtrans:jmxtrans>            }        <jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>        }        ...    ]    </jmxtrans:jmxtrans> } </beans> ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 28
  • 35. Graphite ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 29
  • 36. Graphite • Open Source • Time series database and graphing tool • Used by tech giants • Similar ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 30
  • 37. Graphite • Open Source Simplicity and Self Service • Time series database and graphing tool • Used by tech giants • Similar ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 30
  • 38. Metrics injection / Write Access • Automatic metrics creation • Socket protocols: “plain text” & Python Pickle serialization echo  "softshake.happyCounter  1.2  1364338989"  |  nc  localhost  2003 • Precision and storage duration by configuration [sales_1min_for_30days_5min_for_1year] pattern  =  ^sales. retentions  =  60s:30d,300s:365d ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 1.7MB 31
  • 39. Metrics rendering / Read Access • Graph Composer & URL API • Everybody can create graphs • Wide range of functions • • • sum, scale, derivative, timeshift, deviation, filter, ... Various exports png, svg, csv, json, raw text ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 32
  • 40. The Art of Graphing ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 33
  • 41. Monitoring on the Cloud • Graphite as a Service • http://hostedgraphite.com/ • Addons: Tasseo 2 Zero setup • 1 email: admin@jmxtrans.org password: SOFTSHAKE 3 • No-OPS ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 34
  • 42. Metrics Types Gauge Ever Increasing Counter • Examples: • Information available without transformation • Examples: • “per minute” aggregation required activeRequests, dataSource.activeConnection, ... ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 requestsCount, revenue, ... 35
  • 43. Graphs and Maths ? Ever Increasing Counter Revenue per Second ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 36
  • 44. Graphs and Maths derivative() ? Ever Increasing Counter Revenue per Second ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 36
  • 45. Graphs and Maths ? Per Second → Per Hour Revenue per Hour ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 37
  • 46. Graphs and Maths summarize(..., “1h”) ? Per Second → Per Hour Revenue per Hour ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 37
  • 47. Graphs and Maths ? 2 servers Total Revenue per Hour ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 38
  • 48. Graphs and Maths sumSeries() ? 2 servers Total Revenue per Hour ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 38
  • 49. Graphs and Maths ? server restart Ignore reset to zero ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 39
  • 50. Graphs and Maths nonNegativeDerivative() ? server restart Ignore reset to zero ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 39
  • 51. Graphs and Maths Compare to last week ? ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 40
  • 52. Graphs and Maths Compare to last week timeShift(..., “7d”) ? ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 40
  • 53. JMX • OS: Sysload • JVM: Garbage Collector • Tomcat: activeSessions • Application: Web Site Revenue ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 41
  • 54. Graphite URL API http://localhost:8081/render/?      from=-­‐7days&      title=Revenue%20per%20Hour&      vtitle=Dollars&      lineWidth=3&      xFormat=%25a%20%25H%25p&      target=        alias(          scale(            summarize(              sumSeries(                nonNegativeDerivative(                  edu.servers.*.sales.revenueInCentsCounter)...)&        ... ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 42
  • 55. Sharing metrics and graphs ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 43
  • 56. Dashboards and Wikis dashboard.prod.md #  Website  Traffic <table> <tr>    <td>        <img  src="http://graphite.prod.mycompany/graphite/render/? width=400&height=300&title=Revenue&xFormat=%25a%20%25d %25p&vtitle=USD&lineWidth=3&from=-­‐5days&target=alias(scale(summarize(sumSeries(non NegativeDerivative(edu.servers.*.sales.revenueInCentsCounter))%2C%221h %22)%2C0.01)%2C%22Revenue%20per%20Hour%22)...">        </td>        <td>                <img  src="http://graphite.prod.mycompany/graphite/render/?...">        </td> <tr> </table> Wiki https://github.com/jmxtrans/embedded-jmxtrans-samples/wiki/Dashboard-PROD ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 44
  • 57. Dashboards and Email Send dashboards by email ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 45
  • 58. Alerting with Metrics ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 46
  • 59. Graphite Integration with Alerting • Based on Graphite URL API • RawText or JSON format • Pattern /render?from=-­‐11minutes&until=-­‐1minutes&format=raw&target=**** ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 47
  • 60. Graphite Integration with Alerting • Based on Graphite URL API • RawText or JSON format • Pattern /render?from=-­‐11minutes&until=-­‐1minutes&format=raw&target=**** Example $  curl  "http://graphite.example.com/render? from=-­‐11minutes&until=-­‐1minutes&format=raw&      target=keepLastValue(servers.cloudbees.jvm.os.SystemLoadAverage)" my-­‐metric,1363225680,1363226340,60|0.03,0.01,0.1,4.0,4.0,0.9,0.7,0.8,0.4,0.5,0.5 ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 47
  • 61. Graphite Alerting with Seyren • Alerting Dashboard for Graphite • Open Source • Java .war + MongoDB • Alerts: email, PagerDuty, ... http://seyren.jmxtrans.cloudbees.net/ ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 48
  • 62. Graphite Alerting with Nagios • Monitoring Infrastructure • Open Source • De facto standard • check_graphite plugin • Pierre-Yves Ritschard • Jason Dixon / obfuscurity ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 49
  • 63. Conclusion ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 50
  • 64. Conclusion • Self-service monitoring changes everything • Technical and business monitoring • Monitoring integrated in the Continuous Delivery pipeline • Open Source solutions are available ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 51
  • 65. Hosted Graphite Promo • http://hostedgraphite.com • 45 days trial with “SOFTSHAKE” promo code (3) Promo Code “SOFTSHAKE” (1) Signup (2) Options / Enter Promo Code help: cleclerc@cloudbees.com ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 52
  • 66. http://jmxtrans.org/ http://demo-cocktail.jmxtrans.cloudbees.net/ ©2013 CloudBees, Inc. All Rights Reserved Thursday, October 24, 13 53