Centralized Log Management and Java Application Monitoring

3,196
-1

Published on

What is wrong about accessing logs the usual way?
How to access and search in logs from multiple hosts of distributed system?
How to monitor Java applications?
What is special about application performance monitoring?
How collected metrics can be used to find bottlenecks and improve performance?
How to build gomogenous system for centralized log management and Java application monitoring with Logstash, ElasticSearch and Kibana?

http://developer-should-know.tumblr.com/post/76438808532/centralized-log-management-and-java-application

http://developer-should-know.tumblr.com/post/76434222958/centralized-log-management-and-java-application

https://github.com/evgeniy-khist/jmxtrans-agent-elasticsearch

https://github.com/evgeniy-khist/sampling-agent

Published in: Software

Centralized Log Management and Java Application Monitoring

  1. 1. CENTRALIZED LOG MANAGEMENT AND JAVA APPLICATION MONITORING Author Evgeniy Khist
  2. 2. AGENDA What is wrong about accessing logs the usual way? How to access and search in logs from multiple hosts of distributed system? How to monitor Java applications? What is special about application performance monitoring? How collected metrics can be used to find bottlenecks and improve performance? How to build gomogenous system for centralized log management and Java application monitoring with Logstash, ElasticSearch and Kibana?
  3. 3. WHY DO WE NEED LOGS? To analyze incidents.
  4. 4. WHY DO WE NEED MONITORING? To track current system status to prevent incidents. To analyze collected statistics to improve system even further.
  5. 5. WHAT DO YOU MEAN BY MONITORING? Infrastructure monitoring - keeping track on available resources capacity Application monitoring - collecting statistics of application behaviour to analyze it later
  6. 6. WHAT ARE THE WAYS TO ACCESS LOGS? SSH FTP Network-attached storage (NAS)
  7. 7. WHAT ARE THE WAYS TO SEARCH IN LOGS? grep Search in text editor
  8. 8. WHAT IS BAD ABOUT DEALING WITH LOGS THAT WAY? It becomes hard to access log files on growing number of hosts of distributed systems It is not convenient to track some flow in logs from multiple hosts of distributed systems
  9. 9. WHAT IS BAD ABOUT DEALING WITH LOGS THAT WAY? It is hard to access logs on multiple hosts
  10. 10. WHAT IS BAD ABOUT DEALING WITH LOGS THAT WAY? NAS makes access to logs easier, but not the search
  11. 11. CENTRALIZED LOG MANAGEMENT Logstash, Elasticsearch and Kibana makes both access to logs and search easy
  12. 12. CENTRALIZED LOG MANAGEMENT We have two classes of server. First, one that ships logs. Second, one that collects and indexes logs.
  13. 13. SIMPLIFIED CENTRALIZED LOG MANAGEMENT To make things easier, let's use only shipping Logstash agents
  14. 14. CENTRALIZED LOG MANAGEMENT Single entry point for accessing logs with filtering and searching support. Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching) Elasticsearch - distributed RESTful search and analytics Kibana is a web interface for searching and drilling into all of your logs
  15. 15. LOGSTASH CONFIGURATION i pt { nu fl { ie t p = " e vr lg ye > sr e_ o " p t = [" ot jo ss a dl n/ o/ .o "] ah > / p/ b s/ tn ao el g *l g } } f le { i tr m li ie { u t ln # mr ea l ln st a de n ts at wt d t i ay fr a e g l ie ht o s o t r i h ae n n om t p te n= " (9 2 )d d- /]0 1 9 | [ 1] [ /]0 1 9 | 1 ] 0 9 | [1 ) at r > ^1 |0 [ . ( [ - ]1 0 2) - . ( [ - ] [2 [ - ] 30 ]" n gt = t u ea e > r e w a = " r vo s ht > pe iu " } go { rk # lg ak p te n" dt { S8 0} %5 ee [tr a ]% og r4 } [f l: ln ]% s% o bc a tr %a eI O6 1 - lv l %h ed l ge { 0 %i e% i e mg n m th = [" es g " " ? ){ AA lg ae % SA E% LG EE : ee }{ PC } [{ AA t ac > ms ae, (m %D T: o dt }{ PC } {O LV Ll vl % SA E %D T: } dt { ae m th = [" od t" " yy M- dH : ms ,S "] ac > l ga e, y y -M d Hm :s S S } } o tu { u pt e at ce rh { l s is ac h s = " o ah s" ot > lc lo t c ut r= " os ls e > lg " i dx = "o sa h %+ YY M. d" ne > l gt s- {Y Y .M d}
  16. 16. ELASTICSEARCH Schema-free (document-oriented) Support of different data types: s r n , i t g r l n , tig nee/og fotdul,boen nl la/obe ola, ul Full-Text search Search is done withing index New index can be added periodically (per day, per week, etc) Speed of search doesn't depend on total amount of data Search is done withing index, new index can be created per day for example Outdated indexes can be easily deleted (index deletion is fast operation) Clusterization
  17. 17. KIBANA Browser based analytics and search interface to Logstash and other timestamped data sets stored in ElasticSearch. demo.kibana.org Written entirely in HTML and Javascript Visualizing data in powerful interfaces Powerful Search Syntax Dashboard customization
  18. 18. KIBANA Easily create bar, line and scatter plots, or pie charts and maps
  19. 19. KIBANA Easily create bar, line and scatter plots, or pie charts and maps
  20. 20. KIBANA Easily create bar, line and scatter plots, or pie charts and maps
  21. 21. KIBANA Easily create, save, share, and link your visualized data
  22. 22. KIBANA Powerful Search Syntax
  23. 23. OPERATING SYSTEM AND JVM METRICS System load Free physical memory Open file descriptor count Heap memory usage Loaded class count Thread count All these metrics available in JMX
  24. 24. APPLICATION CUSTOM METRICS Total execution time of each method Number of executions of Spring MVC controller method Other
  25. 25. COMMON TOOLS FOR APPLICATION MONITORING Zabbix Splunk Nagios AppDynamics But it is another standalone tool...
  26. 26. WHAT ABOUT STORING METRICS IN ELASTICSEARCH? Almost always metrics can be present as events Elasticsearch is great tool to store any amount of events Elasticsearch is great tool to search Kibana is great tool for visualizing data
  27. 27. HOW TO COLLECT METRICS FROM JMX AND SEND TO ELASTICSEARCH? jmxtrans-agent jmxtrans-agent-elasticsearch jmxtrans is effectively the missing connector between speaking to a JVM via JMX on one end and any logging/monitoring package on the other end.
  28. 28. JMXTRANS-AGENT DECLARATION J V_ PS "J V_ P S- aa gn : pt /o jx rn - gn .a =m ta s ae tx l A AO T =$ AA OT j v ae t/ ah t/ m ta sa et j rj xr n- gn . m"
  29. 29. JMXTRANS-AGENT CONFIGURATION < mt as ae t j xr n -g n> < ur e> q e is <ur o jc Nm = jv .a gt p= p rt nS se "at iu e " yt m od vr g" qe y b e ta e" aa l n: ye Oe ai g yt m tr bt = Ss eL aA e ae <ur o jc Nm = jv .a gt p= p rt nS se "at iu e " re h sc le oy ie qe y b e ta e" aa l n: ye Oe ai g yt m tr bt = Fe Py ia M mr Sz " <ur o jc Nm = jv .a gt p= p rt nS se "at iu e " re w pp cS z" qe y b e ta e" aa l n: ye Oe ai g yt m tr bt = Fe Sa Sa e ie <ur o jc Nm = jv .a gt p= p rt nS se "at iu e " pn i ee ci tr on qe y b e ta e" aa l n: ye Oe ai g yt m tr bt = Oe Fl Ds r po Cu t <ur o jc Nm = jv .a gt p= e oy a ti ue "e pe o ys g" ky " sd qe y b e ta e" aa l n: ye Mm r" t rb t=H aM mr Ua e e =u e" <ur o jc Nm = jv .a gt p= e oy a ti ue "e pe o ys g" ky " om te " qe y b e ta e" aa l n: ye Mm r" t rb t=H aM mr Ua e e =c mi td <ur o jc Nm = jv .a gt p= e oy a ti ue "o Ha M mr Ua e ky "s d qe y b e ta e" aa l n: ye Mm r" t rb t=N ne pe oy s g" e =u e" <ur o jc Nm = jv .a gt p= e oy a ti ue "o Ha M mr Ua e ky "o mt qe y b e ta e" aa l n: ye Mm r" t rb t=N ne pe oy s g" e =c mi t <ur o jc Nm = jv .a gt p= l sL ai g at iu e " o dd ls Cu t rs lA qe y b e ta e" aa l n: ye Ca so d n" t rb t= La eC a so n" e ut l <ur o jc Nm = jv .a gt p= h ed n"at i ue "h ed on " rs lA is qe y b e ta e" aa l n: ye Tr ai g t rb t=Tr a Cu t eu t la = <ur o jc Nm = jv .a gt p= h ed n"at i ue "a mn he d on "r sl Ai qe y b e ta e" aa l n: ye Tr ai g t rb t=De o Tr aC ut eu tl a <ur o jc Nm = jv .a gt p= h ed n"at i ue "o aS at d he do n" qe y b e ta e" aa l n: ye Tr ai g t rb t=Tt l tr eT ra C ut <ur o jc Nm = jv .a gt p= h ed n"at i ue "e kh ed o n" rs lA is qe y b e ta e" aa l n: ye Tr ai g t rb t=Pa T ra Cu t e ut la <qe is /u re > < up tr tr c as "r .m t as ae tE at c er hu pt rt r > o t uW ie l s=o gj xr n. gn . ls iS ac O tu Wi e" <ls is a cH s >oa h s<ea tc er h ot ea tc er ho t lc lo t/ls i sa cH s > <ls is a cP r >30 / ls is ac Pr > ea tc er ho t 90 <e at ce r ho t <ls is a cC ut ra e ea tc er h /l si sa cC ut r ae ea tc er hl s eN m >ls is a c <ea tc e rh ls eN m > <ls is a cI dx jx r n- {y yM .d <ea tc er hn e > ea tc er hn e >m ta s% yy . Md }/ls i sa cI dx <oe ae n dN m<nd N m> !- mt b d fu tnd Nm > o ea e /o ea e < -e p y y e al - > <sP ei A Tp >r e/s Pe iA Tp > ue rf xs ye t u<ue rf xs y e <ot uW ie > /u pt rt r < ol cI tr a IS cn s2<cl et ne vl n eo d > c l et ne vl ne o d> 0/ ol cI t ra IS cn s <jx rn -g n > /mt a sa et
  30. 30. JAVAAGENT j v . a g i s r m n package provides services that aaln.ntuet allow Java programming language agents to instrument programs running on the JVM The mechanism for instrumentation is modification of the bytecodes of methods
  31. 31. JAVAAGENT The agent class must implement a public static p e a n r m i method similar in principle to the m i application entry point an p bi c as Sm lJ vA e ti pe et C as ie r nf re { u lc l s ip e aa gn m l mn s l s Fl Ta so m r p bi b t[ t as om Ca s od rl ae ,S r n c a sa e C a sc as en R df nd u l c y e] r nf r (l sL ae o d r ti g ls Nm , ls l s Bi ge ei e, S se .u . rn l( ls N m) yt mo tp it n ca sa e; r tr b tc d; eu n y e oe } p bi s ai v i p e an Sr n ag ,I sr m na in is rm n ai n tr w Ca so F u l c t tc o d rm i( t ig rs nt ue tt o n tu et to ) ho s ls Nt i sr mn a in ad rn f re (h s; nt ue tt o. d Ta so mr ti ) } }
  32. 32. JAVAAGENT The manifest of the agent JAR file must contain the attribute PeanCas rmi-ls The value of this attribute is the name of the agent class MT-N/AIETM EAIFMNFS.F P ea nC as c me ap e Sm lJ vA et r mi - ls : o . xm l. ip e aa gn
  33. 33. JAVAAGENT Agent is started by adding j v a e t a a g n option to the commandline j v - a ag n: pt /o s ml jv ae tj r -a j ri ej r a a jv ae t/ a ht /i pe a ag n. a jr a fl . a
  34. 34. WHY JAVAAGENT? It can be added to any environment any time, even by DevOps No relation to application code at all - zero dependencies
  35. 35. WHAT ABOUT APPLICATION CUSTOM METRICS? Often application custom metrics can be present as events too. So they can be stored in Elasticsearch too.
  36. 36. EVEN APPLICATION PERFORMANCE METRICS? For example, how much time was spent for execution of each application method? Yes, easily. Total execution time of each method can be collected with sampler and sent to Elasticsearch.
  37. 37. WHAT IS SAMPLER? Sampler gathers performance and memory data by periodically polling the monitored application for thread dumps or memory histograms.
  38. 38. HOW TO COLLECT SAMPLING INFORMATION? sampling-agent Yes, javaagent once again
  39. 39. SAMPLING-AGENT DECLARATION AND CONFIGURATION J V_ PS "J V_ P S A AO T =$ AA OT - aa gn :a pi gae t j rs m ln It ra =0 , up tn ev l2 0 0o tu Wi e =r . j va e ts ml n-gn . a= ap ig ne v l1 0o tu I tr a= 00,u p tr tr og - ogsm ln ae t ic ue ak g sc me ap e D r.a pi gg n .nl d Pc ae =o .x m l. - ogsm ln ae t ec ue ak g sj v.jv x ,o .u . D r.a pi gg n .xl d Pc ae =a a ,a a.cm sn - ogsm ln ae t ea tc er h ot lc lo t D r.a pi gg n .ls i sa cH s= oa h s - ogsm ln ae t ea tc er h ot 90 D r.a pi gg n .ls i sa cP r= 30 - ogsm ln ae t ea tc er h ls eN m= ls i sa c D r.a pi gg n .ls i sa cC ut ra e ea tc er h - og sm ln ae t ea tc er h ne =a pi g % y y. Md } D r.a pi gg n .ls i sa cI dx sm l n - {y yM . d - ogsm ln ae t nd Nm =o e ae D r.a pi gg n .oe a en dN m"
  40. 40. PUTTING ALL TOGETHER
  41. 41. SINGLE ENTRY POINT FOR EVERYTHING Logs Operating system metrics JVM metrics Application custom metrics From all servers of distributed system
  42. 42. SEPARATE KIBANA DASHBOARDS http://localhost/kibana/#/dashboard/file/logstashs.json http://localhost/kibana/#/dashboard/file/jmxtrans.json http://localhost/kibana/#/dashboard/file/sampling.json
  43. 43. OPERATING SYSTEM AND JVM METRICS Free physical memory and heap memory usage
  44. 44. OPERATING SYSTEM AND JVM METRICS Average system load and thread count
  45. 45. APPLICATION CUSTOM METRICS Total execution time of 2 application methods
  46. 46. HOW TO SET UP LOGSTASH, ELASTICSEARCH AND KIBANA FROM SCRATCH? http://developer-should-know.tumblr.com/post/76434222958/centralized-log-management-and-javaapplication
  47. 47. More slides

×