This document outlines an agenda for a MuleSoft troubleshooting workshop. The agenda includes an introduction to MuleSoft support, an overview of the troubleshooting process, a deep dive into siege tactics for troubleshooting applications and environments, common tools used for troubleshooting, and examples of real troubleshooting use cases. Attendees will also participate in hands-on exercises and have an opportunity to ask questions.

1. April 2017
Troubleshooting Workshop

2. Senior Support Engineer - NA
Chirag Narang

3. All contents © MuleSoft Inc.
Agenda
3
• Introduction to MuleSoft Support
• Troubleshooting: General Overview
• Siege tactics in depth
• Tools used
• Real use cases
• Exercise and Q & A

4. All contents © MuleSoft Inc.
MuleSoft Support
4
• Global Support
– San Francisco, Sydney, Buenos Aires and London
– 24x7 coverage for Sev1 cases
– Follow-The-Sun
• Flat Support Organization
– All support engineers are tier 3 support
– Every engineer can take the case to resolution
– Java developers and experts with Mule products
• 75% of Support Cases are during Development
– Engage early and throughout the customer journey
– Products/platform stability

5. All contents © MuleSoft Inc.
MuleSoft Support Center
5
• Public Knowledge Base
• Create/Update and review support cases
• Search articles, cases and ideas
• Auto-suggest articles when creating a
support case
• Account case history
• Software artefacts download
• View subscription and download license
keys

6. All contents © MuleSoft Inc.
Support channels and resources
6
• MuleSoft Support Center
• Recommended Support Process
• Developer forum

7. Troubleshooting Process

8. All contents © MuleSoft Inc.
Troubleshooting Process
8
• Gathering information
- Initial analysis
- Digging deeper
- Nature of the problem (types of evidence)
- Tools
• Attacking the problem
- Tactics
- Reproducing
- Static analysis
- Tools

9. All contents © MuleSoft Inc.
Gathering Information
9
Gather
Information
Initial
Analysis
Identify
Potential
Root Cause
Root Cause
Confirmed?
Knowledge Search
Keywords
Experience KB +
Previous
Cases
Internet
Release
Notes
Problem
Attack
No
Yes

10. All contents © MuleSoft Inc.
Siege Tactics → Problem attack
Have an
answer?
Get back to the
reporter
Try to reproduce
the issue
a. Simplified application +
Mocked dependencies
b. Real application + Mocked
dependencies
Cannot reproduce:
● Environment related?
● Other applications?
● External dependencies?
Yes No
● Isolate
● Eliminate external
dependencies
● Hypothesis → Iterate

11. All contents © MuleSoft Inc.
Siege tactics in depth: Application
11
• Divide and conquer: simplify the application, isolate the failing code
• Reproducing the issue
• Types of analysis: Static (source code) and live (debugging)

12. All contents © MuleSoft Inc.
Siege tactics in depth: Environmental
12
• Environment differences
• Cluster issues
• Cloudhub
• Resources (memory, disk, connectivity to external services)
• SSL/TLS

13. All contents © MuleSoft Inc.
Tools we use
13
Load test
● Jmeter
Out of memory
● VisualVM
● YourKit (commercial license)
● Memory Analyzer Tool
● Jmeter
Application unresponsive
● VisualVM
● YourKit (commercial license)
● Byteman
Web Services issues
● SoapUI
● WireShark
● Wire logs
Networking issues
● WireShark
● TCP dump
Application errors
● Studio debugger
● Java debugger
Environmental issues
● VirtualBox
● VMWare (commercial license)
● CloudShare (commercial license)
Other issues
● Misc tools: keytool, Keystore
explorer, LDAP browser
● Local servers SMTP, FTP,
SFTP, LDAP
● Byteman

14. Real use cases

15. All contents © MuleSoft Inc.
Real use cases
• Take a heap dump
• Analyze the heap dump
• Leak or load?
• Isolate applications
15
Several applications deployed, out of memory errors

16. All contents © MuleSoft Inc.
Real use cases
• Application calls a web service and is showing an error
• Mocking web services
• Before or After web service invocation?
16
Error calling an external webservice

17. All contents © MuleSoft Inc.
Real use cases
• Take a Thread Dump and look for:
- Blocked threads
- Long running threads
- Waiting threads
- Patterns
• Compare the logs with the thread dump
17
Application unresponsive, no messages being processed

18. All contents © MuleSoft Inc.
Real use cases
• Persistent Queues
• Classloading
• VM Resources
• HTTP Ports
• VPC (connectivity issues)
18
Application runs fine in Studio but fails in CloudHub

19. All contents © MuleSoft Inc.
Real use cases
• Use Default Policies
• Avoid changing the payload
• Debug / Trace Logs
• com.mulesoft.module.policies
• com.mulesoft.module.client. APIPlatformRestClient
• Check the policies folder
19
Problems related to policies

20. Q & A

21. Hand-on exercises
Let’s get to work

22. All contents © MuleSoft Inc. 22
• Exercise A: Out of memory
• Exercise B: Application unresponsive
Hands-on exercises