Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Troubleshooting JIRA andConfluence!While keeping your sanity!Jeff Curry!!Atlassian Support!
Agenda!•  Troubleshooting process!    •  Reviewing logfiles!    •  Reading thread dumps!•  Taking it further!•  Getting sup...
Case Review!
Case Review!•  Entitled Error Message !•  Daily crashes, poor   performance!•  Larger instance!
Case Review – Reviewing Logfiles!   logs/atlassian-jira.log!    ( NOT catalina.out )!
QuartzWorker-0 ERROR ServiceRunner    xxx[service.services.mail.MailFetcherService] xxx[10030]: Error connecting to host x...
org.apache.jk.core.MsgContext action!WARNING: Error sending end packet!java.net.SocketException: Software caused connectio...
DefaultIndexManager - Lock attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Lo...
Case Review – Reviewing Logfiles!   Garbage collection ( GC ) logs!      -verbose:gc -Xloggc:<home>/logs/gc.log - XX:     +...
Case Review – Reviewing Logfiles!                                                                                          ...
Case Review - Workaround!•  GC tuning for stability!    •  Concurrent !       •    -XX:+UseConcMarkSweepGC -XX:+CMSIncreme...
Case Review – Thread dumps!"QuartzWorker-0" prio=10 tid=0x0a2ac400 nid=0xe3e runnable [0x72f04000]!java.lang.Thread.State:...
Case Review – Thread dumps!at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)!at java.io.BufferedInputStream...
Case Review – Thread dumps!at javax.mail.Service.connect(Service.java:288)!- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3...
Case Review – Thread dumps!"IssueIndexer:thread-3" prio=10 tid=0x08ec1000 nid=0x1b64 runnable [0x71f44000]!java.lang.Threa...
Case Review – Thread dumps!at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:605)!at org.ofbiz.core.entity.G...
Case Review – Thread dumps!at com.atlassian.jira.issue.comments.DefaultCommentManager.getComments(DefaultCommentManager.ja...
Case Review - Resolution!JIRA	    POP          Mail      Issue     Database	          service	   inbox	    Key
Case Review - Resolution!
Taking it further                !
Taking it further - VisualVM!•  http://visualvm.java.net! •  Monitor performance! •  Profile memory usage! •  Review thread...
Taking it further – OS Monitoring!Linux 2.6.18-164.11.1.el5PAE (test.jira.com)     !05/17/2012!             01:20:01 AM   ...
Taking it further - SAR!http://www.ibm.com/developerworks/aix/library/au-unix-perfmonsar.html!
Taking it further – HTTP inspection!•  Fiddler! •  http://www.fiddler2.com/fiddler2/!•  Live HTTP Headers ( Firefox addon )!...
Taking it further – HTTP inspection!
Taking it further – HTTP inspection!
Getting support ( quicker )                          !
Getting support – self-help!
Getting support – self-help!•  Knowledge base articles! •  https://confluence.atlassian.com/display/KB/Knowledge+Base+Home!...
Getting support – self-help!
!
Getting support – Atlassian view!•  Initial response time SLA! •  1 – 1 hour! •  2 – 4 hours! •  3 – 8 hours! •  4 – 24 ho...
Getting support – Atlassian view! Open	   Waiting for     Atlassian 	    Waiting for           Customer	   Investigating	 ...
Getting support – file a ticket!•  When I… I expect… however…! •  Application log files ! •  Screenshots! •  Steps to reprod...
Getting support – file a ticket!•  When I use JIRA and try to login!•  I expect JIRA to display a login page!•  however I g...
Getting support – Atlassian view!  Open	     Atlassian 	    Waiting for            Investigating	   Customer	             ...
Summary!•  Basic! •  Application and GC log files! •  Thread dumps!•  Advanced! •  VisualVM! •  HTTP inspection!
I am my own Atlassian support engineer AMAA  #atlassiansummit
Thank you!
Troubleshooting JIRA & Confluence
Troubleshooting JIRA & Confluence
Upcoming SlideShare
Loading in …5
×

Troubleshooting JIRA & Confluence

2,999 views

Published on

With Atlassian Support as your guide learn how to diagnose your JIRA and Confluence installation like a pro, where to look for solutions, and tips for getting a quick response from us when all else fails.

Published in: Technology, News & Politics
  • Be the first to comment

Troubleshooting JIRA & Confluence

  1. 1. Troubleshooting JIRA andConfluence!While keeping your sanity!Jeff Curry!!Atlassian Support!
  2. 2. Agenda!•  Troubleshooting process! •  Reviewing logfiles! •  Reading thread dumps!•  Taking it further!•  Getting support ( quicker )!!
  3. 3. Case Review!
  4. 4. Case Review!•  Entitled Error Message !•  Daily crashes, poor performance!•  Larger instance!
  5. 5. Case Review – Reviewing Logfiles! logs/atlassian-jira.log! ( NOT catalina.out )!
  6. 6. QuartzWorker-0 ERROR ServiceRunner xxx[service.services.mail.MailFetcherService] xxx[10030]: Error connecting to host xxxas user xxx via protocol pop3s: javax.mail.MessagingException: Connect failed;
nested exception is:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIXpath validation failed: java.security.cert.CertPathValidatorException: signature checkfailed
javax.mail.MessagingException: Connect failed;
nested exception is:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIXpath validation failed: java.security.cert.CertPathValidatorException: signature checkfailed!
  7. 7. org.apache.jk.core.MsgContext action!WARNING: Error sending end packet!java.net.SocketException: Software caused connection abort: socket write error!at java.net.SocketOutputStream.socketWrite0(Native Method)!at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)!at java.net.SocketOutputStream.write(SocketOutputStream.java:136)!at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)!at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)!at org.apache.jk.core.MsgContext.action(MsgContext.java:293)!at org.apache.coyote.Response.action(Response.java:182)!at org.apache.coyote.Response.finish(Response.java:304)!at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)!at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)!at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)!at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)!at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)!at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)!at java.lang.Thread.run(Thread.java:534)!
  8. 8. DefaultIndexManager - Lock attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Lock attempt timed out - waited 30000 milliseconds
 !at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:463)
 !at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:286)
 !at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:252)
 !at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:260)
 !at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)
 !at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 !at java.lang.reflect.Method.invoke(Method.java:618)
 !at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:71)
 !at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
 !at $Proxy4.reIndexIssueObjects(Unknown Source)
 !at com.atlassian.jira.event.listeners.search.IssueIndexListener.reIndex(IssueIndexListener.java:138)
 !at com.atlassian.jira.event.listeners.search.IssueIndexListener.issueCommented(IssueIndexListener.java:78)
 !at com.atlassian.jira.event.issue.AbstractIssueEventListener.workflowEvent(AbstractIssueEventListener.java:118)
 !at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:125)
 !at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:83)
 !at com.atlassian.jira.web.action.issue.AddComment.alertSystemOfComment(AddComment.java:101)
 !at com.atlassian.jira.web.action.issue.AddComment.doExecute(AddComment.java:85)
 !at webwork.action.ActionSupport.execute(ActionSupport.java:153)
 !at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:54)
 !at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:132)
 !at com.atlassian.jira.web.dispatcher.JiraServletDispatcher.service(JiraServletDispatcher.java:178)
 !at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
!
  9. 9. Case Review – Reviewing Logfiles! Garbage collection ( GC ) logs! -verbose:gc -Xloggc:<home>/logs/gc.log - XX: +PrintGCTimeStamps -XX:+PrintGCDetails -XX: +PrintGCDateStamps! ! [ http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html ]!
  10. 10. Case Review – Reviewing Logfiles! !•  Plain! user=0.06 sys=0.01, real=0.02 secs]48.630: [GC [PSYoungGen: 112384K->5460K(131072K)] •  Filtered!112384K->5460K(1029888K), 0.0158160 secs] [Times: user=0.03sys=0.01, real=0.03 secs] ! grep Full GC gc.log | awk {print $1, $16}!93.865: [GC [PSYoungGen: 117844K->16468K(131072K)]117844K->16468K(1029888K), 0.0329740 secs] [Times: 9285.005: real=7.80!user=0.06 sys=0.03, real=0.03 secs] ! 9474.161: real=10.99!105.728: [GC [PSYoungGen: 128852K->18302K(131072K)]128852K->18302K(1029888K), 0.0287090 secs] [Times: 10106.270: real=5.57!user=0.07 sys=0.01, real=0.02 secs] !122.928: [GC [PSYoungGen: 130686K->18679K(131072K)] 10355.367: real=12.84!130686K->21123K(1029888K), 0.0221030 secs] [Times: 10758.017: real=15.17!
  11. 11. Case Review - Workaround!•  GC tuning for stability! •  Concurrent ! •  -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:SurvivorRatio=8! •  2 GB allocation!•  Added vCPU!!
  12. 12. Case Review – Thread dumps!"QuartzWorker-0" prio=10 tid=0x0a2ac400 nid=0xe3e runnable [0x72f04000]!java.lang.Thread.State: RUNNABLE!at java.net.SocketInputStream.socketRead0(Native Method)!at java.net.SocketInputStream.read(SocketInputStream.java:129)!at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)!at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)!at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)!- locked <0x744f7a38> (a java.lang.Object) !
  13. 13. Case Review – Thread dumps!at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)!at java.io.BufferedInputStream.read(BufferedInputStream.java:237)!- locked <0x74501140> (a java.io.BufferedInputStream)!at java.io.DataInputStream.readLine(DataInputStream.java:496)!at com.sun.mail.pop3.Protocol.simpleCommand(Protocol.java:360)!at com.sun.mail.pop3.Protocol.<init>(Protocol.java:104)!at com.sun.mail.pop3.POP3Store.getPort(POP3Store.java:214)!- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:157)!- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!
  14. 14. Case Review – Thread dumps!at javax.mail.Service.connect(Service.java:288)!- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!at com.atlassian.jira.service.services.mail.MailFetcherService.run(MailFetcherService.java:146)!at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60)!at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47)!at org.quartz.core.JobRunShell.run(JobRunShell.java:195)!at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72)!at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)!
  15. 15. Case Review – Thread dumps!"IssueIndexer:thread-3" prio=10 tid=0x08ec1000 nid=0x1b64 runnable [0x71f44000]!java.lang.Thread.State: RUNNABLE!...!at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java)!at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)!at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)!at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:92)!at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597)!
  16. 16. Case Review – Thread dumps!at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:605)!at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131)!at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:790)!at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:775)!at org.ofbiz.core.entity.GenericDelegator.getRelated(GenericDelegator.java:1211)!at org.ofbiz.core.entity.GenericDelegator.getRelatedByAnd(GenericDelegator.java:1169)!at org.ofbiz.core.entity.GenericValue.getRelatedByAnd(GenericValue.java:260)!atcom.atlassian.jira.issue.managers.DefaultIssueManager.getEntitiesByIssue(DefaultIssueManager.java:318)!
  17. 17. Case Review – Thread dumps!at com.atlassian.jira.issue.comments.DefaultCommentManager.getComments(DefaultCommentManager.java:135)!at com.atlassian.jira.issue.index.DefaultCommentRetriever.get(DefaultCommentRetriever.java:40)!at com.atlassian.jira.issue.index.DefaultCommentRetriever.get(DefaultCommentRetriever.java:29)!at com.atlassian.jira.issue.index.DefaultIssueIndexer$CommentDocumentBuilder.get(DefaultIssueIndexer.java:460)!at com.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:315)!at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:371)!!!
  18. 18. Case Review - Resolution!JIRA POP Mail Issue Database service inbox Key
  19. 19. Case Review - Resolution!
  20. 20. Taking it further !
  21. 21. Taking it further - VisualVM!•  http://visualvm.java.net! •  Monitor performance! •  Profile memory usage! •  Review thread dumps! •  Browse heap dumps!
  22. 22. Taking it further – OS Monitoring!Linux 2.6.18-164.11.1.el5PAE (test.jira.com) !05/17/2012! 01:20:01 AM all 3.48 0.00 1.14 0.06 0.00 95.33!! 01:30:01 AM all 3.80 0.00 0.95 0.06 0.00 95.19!12:00:01 AM CPU %user %nice %system %iowait %steal %idle! 01:40:01 AM all 3.43 0.00 0.90 0.07 0.00 95.60!12:10:01 AM all 5.22 0.00 1.50 6.50 0.00 86.79! 01:50:01 AM all 3.41 0.00 0.92 0.07 0.00 95.61!12:20:01 AM all 3.42 0.00 0.90 0.08 0.00 95.60! 02:00:01 AM all 3.42 0.00 0.91 0.19 0.00 95.49!12:30:01 AM all 4.77 0.00 1.01 0.29 0.00 93.92! 02:10:01 AM all 3.71 0.00 0.94 3.59 0.00 91.76!12:40:01 AM all 3.47 0.00 0.90 0.07 0.00 95.56! 02:20:01 AM all 3.38 0.00 0.90 0.08 0.00 95.65!12:50:01 AM all 3.43 0.00 1.13 0.07 0.00 95.38! 02:30:01 AM all 3.77 0.00 0.94 0.07 0.00 95.23!01:00:01 AM all 3.48 0.00 1.12 0.06 0.00 95.33! 02:40:01 AM all 3.38 0.00 0.89 0.07 0.00 95.66!01:10:01 AM all 4.27 0.00 0.95 1.74 0.00 93.04! 02:50:01 AM all 3.37 0.00 0.89 0.06 0.00 95.69! 03:00:01 AM all 3.37 0.00 0.89 0.07 0.00 95.67!
  23. 23. Taking it further - SAR!http://www.ibm.com/developerworks/aix/library/au-unix-perfmonsar.html!
  24. 24. Taking it further – HTTP inspection!•  Fiddler! •  http://www.fiddler2.com/fiddler2/!•  Live HTTP Headers ( Firefox addon )! •  https://addons.mozilla.org/en-US/firefox/addon/live-http-headers/!
  25. 25. Taking it further – HTTP inspection!
  26. 26. Taking it further – HTTP inspection!
  27. 27. Getting support ( quicker ) !
  28. 28. Getting support – self-help!
  29. 29. Getting support – self-help!•  Knowledge base articles! •  https://confluence.atlassian.com/display/KB/Knowledge+Base+Home!•  Answers.atlassian.com!•  #atlassian on irc.freenode.com!
  30. 30. Getting support – self-help!
  31. 31. !
  32. 32. Getting support – Atlassian view!•  Initial response time SLA! •  1 – 1 hour! •  2 – 4 hours! •  3 – 8 hours! •  4 – 24 hours! !
  33. 33. Getting support – Atlassian view! Open Waiting for Atlassian Waiting for Customer Investigating Customer IRT
  34. 34. Getting support – file a ticket!•  When I… I expect… however…! •  Application log files ! •  Screenshots! •  Steps to reproduce!
  35. 35. Getting support – file a ticket!•  When I use JIRA and try to login!•  I expect JIRA to display a login page!•  however I get this 404 page! •  log files attached!
  36. 36. Getting support – Atlassian view! Open Atlassian Waiting for Investigating Customer IRT
  37. 37. Summary!•  Basic! •  Application and GC log files! •  Thread dumps!•  Advanced! •  VisualVM! •  HTTP inspection!
  38. 38. I am my own Atlassian support engineer AMAA #atlassiansummit
  39. 39. Thank you!

×