2
Challenges in Troubleshooting On-Prem Applications
1. Missing
Troubleshooting
Artifacts
2. Reliance on Customer Staff
for Timely Data Collection
3. Refusal to Share
Artifacts for Security
Concerns
4. Environmental Instability
& Hidden Customizations
5. Misleading Communication
About Problems
6. Unable to Reproduce
Problem Internally
Open-Source script:
https://github.com/ycrash/yc-360-script
5. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump
13. ps
15. Disk Usage
7. top
6. Extended Data
11. ping
14. Kernel Params
1. App Logs
16. metadata
4. Heap Substitute 8. top -H
3
Effective Troubleshooting Strategy: 360° Artifacts
./yc –p <PROCESS_ID>
4
Key Features of yc-360 Script
1. Platform Agnostic 2. Pristine Data Capture 3. Extended Data
Capturing
4. Security and Compliance 5. Almost zero Overhead 6. Completes under 30
seconds
https://docs.ycrash.io/yc-360/en
vironment/introduction.html
https://docs.ycrash.io/yc-360/fe
atures/pristine-capture.html
https://docs.ycrash.io/yc-360/feat
ures/extended-data.html
https://docs.ycrash.io/ycrash-featur
es/secure-troubleshooting.html
https://blog.ycrash.io/ycrash-age
nt-overhead-performance/
https://blog.ycrash.io/ycrash-age
nt-overhead-performance/
5
360° Analysis Demo
https://docs.ycrash.io/ycrash-features/bundle-upload.html
6
Case Study 1: Transaction Timeout
https://blog.gceasy.io/java-gc-tuning-improved-insurance-company-throughput/
7
Case Study 2: CPU Spike
https://blog.gceasy.io/java-gc-tuning-improved-insurance-company-throughput/
8
Case Study 3: Intermittent HTTP 502 Errors
https://blog.gceasy.io/java-gc-tuning-improved-insurance-company-throughput/
EBS Architecture
9
Clue: Nginx Error
10
1. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump
6. ps
8. Disk Usage
5. top 13. iostat
11. ping
14. Kernel Params
15. App Logs
16. metadata
4. Heap Substitute
7. top -H
11
360° Data
Open-Source script:
https://github.com/ycrash/yc-360-script
12
13
Benefits of 360° Artifacts
1. Simplifies the root cause
identification process
2. Minimizes Friction with
Customer
Thank you, my friends!
Ram Lakshmanan
@ycrash_rca
This deck will be published in: https://blog.ycrash.io
https://www.linkedin.com/company/ycrash

Key Challenges in Troubleshooting Customer On-Premise Applications

  • 2.
    2 Challenges in TroubleshootingOn-Prem Applications 1. Missing Troubleshooting Artifacts 2. Reliance on Customer Staff for Timely Data Collection 3. Refusal to Share Artifacts for Security Concerns 4. Environmental Instability & Hidden Customizations 5. Misleading Communication About Problems 6. Unable to Reproduce Problem Internally
  • 3.
    Open-Source script: https://github.com/ycrash/yc-360-script 5. GCLog 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 13. ps 15. Disk Usage 7. top 6. Extended Data 11. ping 14. Kernel Params 1. App Logs 16. metadata 4. Heap Substitute 8. top -H 3 Effective Troubleshooting Strategy: 360° Artifacts ./yc –p <PROCESS_ID>
  • 4.
    4 Key Features ofyc-360 Script 1. Platform Agnostic 2. Pristine Data Capture 3. Extended Data Capturing 4. Security and Compliance 5. Almost zero Overhead 6. Completes under 30 seconds https://docs.ycrash.io/yc-360/en vironment/introduction.html https://docs.ycrash.io/yc-360/fe atures/pristine-capture.html https://docs.ycrash.io/yc-360/feat ures/extended-data.html https://docs.ycrash.io/ycrash-featur es/secure-troubleshooting.html https://blog.ycrash.io/ycrash-age nt-overhead-performance/ https://blog.ycrash.io/ycrash-age nt-overhead-performance/
  • 5.
  • 6.
    6 Case Study 1:Transaction Timeout https://blog.gceasy.io/java-gc-tuning-improved-insurance-company-throughput/
  • 7.
    7 Case Study 2:CPU Spike https://blog.gceasy.io/java-gc-tuning-improved-insurance-company-throughput/
  • 8.
    8 Case Study 3:Intermittent HTTP 502 Errors https://blog.gceasy.io/java-gc-tuning-improved-insurance-company-throughput/
  • 9.
  • 10.
  • 11.
    1. GC Log 10.netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 11 360° Data Open-Source script: https://github.com/ycrash/yc-360-script
  • 12.
  • 13.
    13 Benefits of 360°Artifacts 1. Simplifies the root cause identification process 2. Minimizes Friction with Customer
  • 14.
    Thank you, myfriends! Ram Lakshmanan @ycrash_rca This deck will be published in: https://blog.ycrash.io https://www.linkedin.com/company/ycrash

Editor's Notes

  • #2 HK: 6 Images
  • #4 HK: 6 Images
  • #6 Baseline: https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjQvMDUvMTkvamF2YV9nYy0yMDI0LTA0LTMwXzA3LTE2LTAyX24xLmxvZy0tNC01My0xMw==&channel=WEB&s=t Benchmark: https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjUvMDEvOS9qYXZhX2djLTIwMjQtMDYtMjJfMDYtMzQtMTAubG9nLS0xOC03LTQ=&channel=WEB&s=t
  • #7 https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjMvMDMvMTMvaWJtLWNvcmUtZHVtcC10b3BkYXRhLnppcC0tMjItMjItNQ==&s=t
  • #12 https://ee.ycrash.io/yc-report-kernel.jsp?ou=Testing&de=172.31.5.135&app=yc&ts=2024-09-18T11-25-22
  • #13 HK: 3 Images