• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Troubleshooting Live Java Web Applications
 

Troubleshooting Live Java Web Applications

on

  • 219 views

Speaker: Ashley Puls ...

Speaker: Ashley Puls
Operations teams tend to get nervous when debugging tools are installed on production systems. This is understandable as most people have had at least one debugging tool crash a system. However, are there tools that can be run in a production environment?
This talk will cover methods and tools that can be used to uncover and resolve performance problems arising in Java web applications that use the Spring Framework. This talk will touch on a few common problems such as slow SQL, slow web page load times, blocked threads, and exceptions in the code. It will use a variety of diagnostic tools to examine these problems including log files, jmap, jvisualvm, and New Relic. Finally, the talk will evaluate the use of these tools in a production environment, separating them into what can and cannot be realistically used in live production web applications

Statistics

Views

Total Views
219
Views on SlideShare
219
Embed Views
0

Actions

Likes
0
Downloads
10
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Troubleshooting Live Java Web Applications Troubleshooting Live Java Web Applications Presentation Transcript

    • Troubleshooting Live Java Web Applications By Ashley Puls © 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission. Sunday, September 8, 13
    • Sunday, September 8, 13
    • Troubleshooting Location and eliminate sources of trouble Determine and settle problems A systematic approach to solving problems quickly and efficiently A logical process of elimination to identify the true source of the problem Sunday, September 8, 13
    • How To Become A Good Troubleshooter Practice makes perfect Images From: http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg Sunday, September 8, 13
    • How To Become A Good Troubleshooter Practice makes perfect Have a good set of tools Images From: http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg, http://blogs.telerik.com/images/default-source/jeff-fritz/toolbox.jpg?sfvrsn=0 Sunday, September 8, 13
    • Outline examine a web application with multiple performance problems troubleshoot the problems using several tools talk about the pros and cons of each tool Sunday, September 8, 13
    • STEP 1: WE NEED AN APPLICATION Sunday, September 8, 13
    • The Application Dear Developer, I had some engineers develop “Running Central” to search for runners, races, and training teams. Can you please keep the site running? Sincerely, Max Siegel CEO of USA Track and Field Image From: http://www.sportsbusinessdaily.com/Journal/Issues/2012/06/18/Leagues-and-Governing-Bodies/~/media/8C99287D3B0E4784932EE3BF049E7DA9.ashx Sunday, September 8, 13
    • The Application Tomcat Application Server Client racerunners.war jsp Client Controller Manager Entities Servlet hibernate Client MySQL Database Sunday, September 8, 13
    • WE HAVE OUR APPLICATION SHOULD WE JUST JUMP INTO TROUBLESHOOTING? Image From: http://toosoxy.files.wordpress.com/2011/04/cliff.jpg Sunday, September 8, 13
    • What Is Trouble? Image From: http://1.bp.blogspot.com/-_LgC3yz9w4w/T4eI--IcvxI/AAAAAAAAA-o/VLdqkxyQpgg/s400/computerFire.jpg Sunday, September 8, 13
    • What Is Trouble? Site Inaccessible Sunday, September 8, 13
    • What Is Trouble? Site Inaccessible Exceptions In Logs Sunday, September 8, 13
    • What Is Trouble? Site Inaccessible Exceptions In Logs Sunday, September 8, 13 Loading Slowly
    • What Is Normal? Images From: http://espn.go.com/mlb/statistics, http://www.franklinbaseball.com/pros/american-league/central/miguel-cabrera Sunday, September 8, 13
    • What Are The Stats? Errors errors within application Thread State active thread states Sunday, September 8, 13 Application Response Time Database Response Time time per request time per request Garbage Collection Memory Allocation time spent in GC heap and PermGen
    • How To Gather The Stats Logs core JDK tools JMX Metrics Application Performance Monitoring Tools (APM) Sunday, September 8, 13 Profiler Framework/App Server Monitoring Tools
    • Basic Information Tools Overview Installation Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 Basic information about each tool Is It Free Include With JDK
    • Data In Tool Tools App Response Time DB Response Time Error Info Thread Info Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 What data does each tool provide? GC Memory
    • LETS TALK TO SOME OF OUR USERS AND START TROUBLESHOOTING Sunday, September 8, 13
    • Image From: http://resources0.news.com.au/images/2012/08/10/1226447/688548-usain-bolt.jpg Sunday, September 8, 13
    • Usain Bolt : Six Time Olympic Champion Image From: http://resources0.news.com.au/images/2012/08/10/1226447/688548-usain-bolt.jpg Sunday, September 8, 13
    • I was looking for a 5K race on your site, but it was loading like I was on a 56k Image From: http://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2012/8/9/1344533810934/f582144a-9dd2-44ed-9828-0ec1f5f768a8-460.jpeg Sunday, September 8, 13
    • LETS INVESTIGATE! Sunday, September 8, 13
    • New Relic Application Performance Monitoring (APM) tool that monitors web apps, mobile apps, servers, and databases Sign up and download agent from the New Relic Website Sunday, September 8, 13
    • Basic Information Tools Overview Installation Is It Free Include With JDK APM solution restart app with agent basic features: yes adv features: no no Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13
    • Sunday, September 8, 13
    • Application Response Time Sunday, September 8, 13
    • Application Response Time Error Information Sunday, September 8, 13
    • Application Response Time Error Information Sunday, September 8, 13
    • Sunday, September 8, 13
    • Heap Memory Sunday, September 8, 13
    • Heap Memory Non-Heap Memory Sunday, September 8, 13
    • Garbage Collection Sunday, September 8, 13
    • JVM Memory Heap Memory Class Instances Arrays Non- Heap Memory Runtime constant pool Interned Strings Field and method data Sunday, September 8, 13 Other JVM internal structures
    • Common Memory Problems Heap Memory Non- Heap Memory Keeping references to old objects Long term objects never cleaned up Sunday, September 8, 13 Leaky class loaders Interning an excessive number of Strings Other
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • JMap Prints the memory map of a Java process Commands jmap -heap JAVA_PID > /tmp/jmap.out jmap -dump:file=/tmp/heap_dump.map JAVA_PID Sunday, September 8, 13
    • Basic Information Tools Overview Installation Is It Free Include With JDK APM solution restart app with agent basic features: yes adv features: no no prints memory map of Java process no, attaches to running process yes yes Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • The File Sunday, September 8, 13
    • The File Heap Memory Sunday, September 8, 13
    • The File Heap Memory Non-Heap Memory Sunday, September 8, 13
    • The File Heap Memory Non-Heap Memory Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • The File Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • YourKit Profiler that provides in depth CPU, thread, and memory profiling Get license key and download from YourKit Website Sunday, September 8, 13
    • Basic Information Tools Overview Installation Is It Free Include With JDK New Relic APM solution restart app with agent basic features: yes adv features: no no YourKit profiler maybe no, must pay for all features no prints memory map of Java process no, attaches to running process yes yes Logs App Server Monitoring JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Thread Count Sunday, September 8, 13
    • Error Rate Thread Count Sunday, September 8, 13
    • Error Rate Thread Count Sunday, September 8, 13
    • Sunday, September 8, 13
    • Thread Information Sunday, September 8, 13
    • Thread Information Sunday, September 8, 13
    • Heap Memory Sunday, September 8, 13 Non-Heap Memory
    • Heap Memory Sunday, September 8, 13 Non-Heap Memory
    • GC Information Sunday, September 8, 13
    • Run a CPU profile Sunday, September 8, 13
    • Application Response Times Sunday, September 8, 13
    • Application Response Times Sunday, September 8, 13
    • Database Response Times Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • /** * This service records stats on the cities and states queried by users when looking for races. The stats are * printed out to the log. */ public class RaceStatsServiceImpl implements RaceStatsService { private static final Logger LOGGER = LoggingUtils.getLogger(RunnerStatsServiceImpl.class.getName()); private static final Integer ONE = Integer.valueOf(1); private static final long DELAY = 1; /** Executer service which executes the thread that compiles stats and prints them to the log. */ private final ScheduledExecutorService scheduledExecutor; /** Holds the cities queried by users in the last minute. */ private final Map<String, Integer> citiesQueried; /** Holds the states queried in the last minute. */ private final Map<String, Integer> stateQueried; public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); // Calls the processor thread’s run method every minute scheduledExecutor = Executors.newSingleThreadScheduledExecutor(); scheduledExecutor.scheduleAtFixedRate(new Processor(), DELAY, DELAY, TimeUnit.MINUTES); } /** * This should be called every time a city or state is used to search for races. * * @param city The city that was queried. * @param state The state that was queried. */ public void addLocation(String city, String state) { if (city != null && !city.isEmpty()) { addToMap(citiesQueried, city.trim()); } if (state != null && state.isEmpty()) { addToMap(stateQueried, state); } } Sunday, September 8, 13
    • /** * This service records stats on the cities and states queried by users when looking for races. The stats are * printed out to the log. */ public class RaceStatsServiceImpl implements RaceStatsService { private static final Logger LOGGER = LoggingUtils.getLogger(RunnerStatsServiceImpl.class.getName()); private static final Integer ONE = Integer.valueOf(1); private static final long DELAY = 1; /** Executer service which executes the thread that compiles stats and prints them to the log. */ private final ScheduledExecutorService scheduledExecutor; /** Holds the cities queried by users in the last minute. */ private final Map<String, Integer> citiesQueried; /** Holds the states queried in the last minute. */ private final Map<String, Integer> stateQueried; Two ConcurrentHashMaps public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); // Calls the processor thread’s run method every minute scheduledExecutor = Executors.newSingleThreadScheduledExecutor(); scheduledExecutor.scheduleAtFixedRate(new Processor(), DELAY, DELAY, TimeUnit.MINUTES); } /** * This should be called every time a city or state is used to search for races. * * @param city The city that was queried. * @param state The state that was queried. */ public void addLocation(String city, String state) { if (city != null && !city.isEmpty()) { addToMap(citiesQueried, city.trim()); } if (state != null && state.isEmpty()) { addToMap(stateQueried, state); } } Sunday, September 8, 13
    • /** * This service records stats on the cities and states queried by users when looking for races. The stats are * printed out to the log. */ public class RaceStatsServiceImpl implements RaceStatsService { private static final Logger LOGGER = LoggingUtils.getLogger(RunnerStatsServiceImpl.class.getName()); private static final Integer ONE = Integer.valueOf(1); private static final long DELAY = 1; /** Executer service which executes the thread that compiles stats and prints them to the log. */ private final ScheduledExecutorService scheduledExecutor; /** Holds the cities queried by users in the last minute. */ private final Map<String, Integer> citiesQueried; /** Holds the states queried in the last minute. */ private final Map<String, Integer> stateQueried; Two ConcurrentHashMaps public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); // Calls the processor thread’s run method every minute scheduledExecutor = Executors.newSingleThreadScheduledExecutor(); scheduledExecutor.scheduleAtFixedRate(new Processor(), DELAY, DELAY, TimeUnit.MINUTES); } /** * This should be called every time a city or state is used to search for races. * * @param city The city that was queried. * @param state The state that was queried. */ public void addLocation(String city, String state) { if (city != null && !city.isEmpty()) { addToMap(citiesQueried, city.trim()); } if (state != null && state.isEmpty()) { addToMap(stateQueried, state); } } Sunday, September 8, 13
    • /** * Thread that runs periodically and reports the frequency that cities and states are queried to the log. */ private final class Processor extends Thread { /** * Creates this Processor. */ public Processor() { super(); } /** * Prints each city and state that was queried by a user since the Processor thread was last run. */ @Override public void run() { StringBuilder sb = new StringBuilder("(Race City Query:Count: "); Iterator<Entry<String, Integer>> cityIt = citiesQueried.entrySet().iterator(); Entry<String, Integer> current; while (cityIt.hasNext()) { current = cityIt.next(); sb.append(current.getKey()).append(":").append(current.getValue()).append(" "); } sb.append("nRace State Query:Count: "); Iterator<Entry<String, Integer>> stateIt = stateQueried.entrySet().iterator(); while (stateIt.hasNext()) { current = stateIt.next(); sb.append(current.getKey()).append(current.getValue()).append(" "); } LOGGER.log(Level.INFO, sb.toString()); } } Sunday, September 8, 13
    • /** * Thread that runs periodically and reports the frequency that cities and states are queried to the log. */ private final class Processor extends Thread { /** * Creates this Processor. */ public Processor() { super(); } Processor is an inner class of RaceStatsServiceImpl /** * Prints each city and state that was queried by a user since the Processor thread was last run. */ @Override public void run() { StringBuilder sb = new StringBuilder("(Race City Query:Count: "); Iterator<Entry<String, Integer>> cityIt = citiesQueried.entrySet().iterator(); Entry<String, Integer> current; while (cityIt.hasNext()) { current = cityIt.next(); sb.append(current.getKey()).append(":").append(current.getValue()).append(" "); } sb.append("nRace State Query:Count: "); Iterator<Entry<String, Integer>> stateIt = stateQueried.entrySet().iterator(); while (stateIt.hasNext()) { current = stateIt.next(); sb.append(current.getKey()).append(current.getValue()).append(" "); } LOGGER.log(Level.INFO, sb.toString()); } } Sunday, September 8, 13
    • /** * Thread that runs periodically and reports the frequency that cities and states are queried to the log. */ private final class Processor extends Thread { /** * Creates this Processor. */ public Processor() { super(); } Processor is an inner class of RaceStatsServiceImpl /** * Prints each city and state that was queried by a user since the Processor thread was last run. */ @Override public void run() { StringBuilder sb = new StringBuilder("(Race City Query:Count: "); Iterator<Entry<String, Integer>> cityIt = citiesQueried.entrySet().iterator(); Entry<String, Integer> current; while (cityIt.hasNext()) { current = cityIt.next(); sb.append(current.getKey()).append(":").append(current.getValue()).append(" "); } sb.append("nRace State Query:Count: "); Iterator<Entry<String, Integer>> stateIt = stateQueried.entrySet().iterator(); while (stateIt.hasNext()) { current = stateIt.next(); sb.append(current.getKey()).append(current.getValue()).append(" "); } LOGGER.log(Level.INFO, sb.toString()); } } Sunday, September 8, 13 Two Concurrent HashMaps
    • private final class Processor extends Thread { /** * Creates this Processor. */ public Processor() { super(); } /** * Prints each city and state that was queried by a user since the Processor thread was last run. */ @Override public void run() { StringBuilder sb = new StringBuilder("(Race City Query:Count: "); Iterator<Entry<String, Integer>> cityIt = citiesQueried.entrySet().iterator(); Entry<String, Integer> current; while (cityIt.hasNext()) { current = cityIt.next(); sb.append(current.getKey()).append(":").append(current.getValue()).append(" "); cityIt.remove(); } sb.append("nRace State Query:Count: "); Iterator<Entry<String, Integer>> stateIt = stateQueried.entrySet().iterator(); while (stateIt.hasNext()) { current = stateIt.next(); sb.append(current.getKey()).append(current.getValue()).append(" "); stateIt.remove(); } LOGGER.log(Level.INFO, sb.toString()); } } Sunday, September 8, 13
    • SOLUTION CLEAN UP MEMORY IN LONG RUNNING OBJECTS Sunday, September 8, 13
    • AFTER THE FIX Sunday, September 8, 13
    • The File Sunday, September 8, 13
    • The File Sunday, September 8, 13
    • Image From: http://news.images.itv.com/image/file/69820/article_9e6cab04948b5c33_1343574136_9j-4aaqsk.jpeg Sunday, September 8, 13
    • Paula Radcliffe Female Marathon World Record Holder Image From: http://news.images.itv.com/image/file/69820/article_9e6cab04948b5c33_1343574136_9j-4aaqsk.jpeg Sunday, September 8, 13
    • It was taking too long to find my fellow runners on your site Image From: http://i3.manchestereveningnews.co.uk/incoming/article660410.ece/ALTERNATES/s615/C_71_article_1584722_image_list_image_list_item_0_image-660410.jpg Sunday, September 8, 13
    • LETS INVESTIGATE! Sunday, September 8, 13
    • JVisualVM Visual tool which provides heap dumps, memory leak information, garbage collection monitoring, memory profiling, and CPU profiling Attaches to local and remote running processes Sunday, September 8, 13
    • Basic Information Tools Overview Installation Is It Free Include With JDK New Relic APM solution restart app with agent basic features: yes adv features: no no YourKit profiler maybe no, must pay for all features no JVisualVm profiler no yes yes prints memory map of Java process no, attaches to running process yes yes Logs App Server Monitoring JConsole JMap/JHat JStack Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • GC Information Sunday, September 8, 13
    • GC Information Memory Information Sunday, September 8, 13
    • GC Information Memory Information Thread Information Sunday, September 8, 13
    • GC Information Memory Information Thread Information Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • Kill Dash 3 Provides the current stack trace for each thread in a local JVM (thread dump) Available on linux and macs (ctrl+break on windows) Command: kill -3 PID Sunday, September 8, 13
    • JStack Provides the current stack trace for each thread in a local or remote JVM (thread dump) Command: jstack -l JAVA_PID > /tmp/jstack.out Sunday, September 8, 13
    • Basic Information Tools Overview Installation Is It Free Include With JDK New Relic APM Solution restart app with agent basic features: yes adv features: no no YourKit profiler maybe no, must pay for all features no JVisualVm profiler no yes yes JMap/JHat prints memory map of Java process no, attaches to running process yes yes JStack prints thread dump of Java process no, attaches to running process yes yes Logs App Server Monitoring JConsole Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • The File Sunday, September 8, 13
    • The File Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • New Relic Application Performance Monitoring (APM) tool that monitors web apps, mobile apps, servers, and databases Sign up and download agent from the New Relic Website Sunday, September 8, 13
    • Sunday, September 8, 13
    • Thread Count Sunday, September 8, 13
    • Thread Count Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • /** * Stores stats on the last names of runners which have been queried. */ public class RunnerStatsServiceImpl implements RunnerStatsService { private static final Logger LOGGER = LoggingUtils.getLogger(RunnerStatsServiceImpl.class.getName()); private static final long DELAY = 1; /** Executor service which schedules the StatsProcessor thread to run every minute. */ private final ScheduledExecutorService scheduledExecutor; /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried; public RunnerStatsServiceImpl() { lastNamesQueried = new ArrayBlockingQueue<String>(100); // reports the metrics to the log every minute scheduledExecutor = Executors.newSingleThreadScheduledExecutor(); scheduledExecutor.scheduleAtFixedRate(new StatsProcessor(), DELAY, DELAY, TimeUnit.MINUTES); } /** * Adds an entry in the stats. * * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } } Sunday, September 8, 13
    • /** * Stores stats on the last names of runners which have been queried. */ public class RunnerStatsServiceImpl implements RunnerStatsService { private static final Logger LOGGER = LoggingUtils.getLogger(RunnerStatsServiceImpl.class.getName()); private static final long DELAY = 1; /** Executor service which schedules the StatsProcessor thread to run every minute. */ private final ScheduledExecutorService scheduledExecutor; /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried; public RunnerStatsServiceImpl() { lastNamesQueried = new ArrayBlockingQueue<String>(100); // reports the metrics to the log every minute scheduledExecutor = Executors.newSingleThreadScheduledExecutor(); scheduledExecutor.scheduleAtFixedRate(new StatsProcessor(), DELAY, DELAY, TimeUnit.MINUTES); } /** * Adds an entry in the stats. * * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } } Sunday, September 8, 13 Where the threads are sleeping
    • /** * Stores stats on the last names of runners which have been queried. */ public class RunnerStatsServiceImpl implements RunnerStatsService { private static final Logger LOGGER = LoggingUtils.getLogger(RunnerStatsServiceImpl.class.getName()); private static final long DELAY = 1; /** Executor service which schedules the StatsProcessor thread to run every minute. */ private final ScheduledExecutorService scheduledExecutor; /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried; queue blocking at 100 strings public RunnerStatsServiceImpl() { lastNamesQueried = new ArrayBlockingQueue<String>(100); // reports the metrics to the log every minute scheduledExecutor = Executors.newSingleThreadScheduledExecutor(); scheduledExecutor.scheduleAtFixedRate(new StatsProcessor(), DELAY, DELAY, TimeUnit.MINUTES); } /** * Adds an entry in the stats. * * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } } Sunday, September 8, 13 Where the threads are sleeping
    • /** * Stores stats on the last names of runners which have been queried. */ public class RunnerStatsServiceImpl implements RunnerStatsService { private static final Logger LOGGER = LoggingUtils.getLogger(RunnerStatsServiceImpl.class.getName()); private static final long DELAY = 1; /** Executor service which schedules the StatsProcessor thread to run every minute. */ private final ScheduledExecutorService scheduledExecutor; /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried; public RunnerStatsServiceImpl() { queue blocking at 100 strings lastNamesQueried = new ArrayBlockingQueue<String>(100); // reports the metrics to the log every minute scheduledExecutor = Executors.newSingleThreadScheduledExecutor(); scheduledExecutor.scheduleAtFixedRate(new StatsProcessor(), DELAY, DELAY, TimeUnit.MINUTES); } /** * Adds an entry in the stats. * * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } } Sunday, September 8, 13 Where the threads are sleeping
    • SOLUTION DO NOT BLOCK THREADS Sunday, September 8, 13
    • AFTER THE FIX Sunday, September 8, 13
    • The File Sunday, September 8, 13
    • The File waiting for more requests to come in Sunday, September 8, 13
    • Image From: http://www.webrun.com.br/multimidia/fotos/2011/20110926_175711_g.jpg Sunday, September 8, 13
    • Patrick Makau Musyoki Marathon World Record Holder Image From: http://www.webrun.com.br/multimidia/fotos/2011/20110926_175711_g.jpg Sunday, September 8, 13
    • I tried to look for marathons on your site but could not access it Image From: http://simonfreeman.co.uk/wp-content/uploads/2013/04/image-4.jpeg Sunday, September 8, 13
    • LETS INVESTIGATE! Sunday, September 8, 13
    • New Relic Application Performance Monitoring (APM) tool that monitors web apps, mobile apps, servers, and databases Sign up and download agent from the New Relic Website Sunday, September 8, 13
    • Sunday, September 8, 13
    • JConsole Graphical User Interface (GUI) which displays Java Management Extension (JMX) metrics for a local or remote JVMs JMX metrics should be enabled in your application server Sunday, September 8, 13
    • Basic Information Tools Overview Installation Cost Include With JDK New Relic APM solution restart app with agent basic features: yes adv features: no no YourKit profiler maybe no, must pay for all features no JVisualVm profiler no yes yes JConsole JMX metrics no yes yes JMap/JHat prints memory map of Java process no, attaches to running process yes yes JStack prints thread dump of Java process no, attaches to running process yes yes Logs App Server Monitoring Sunday, September 8, 13
    • Basic Information Tools Overview Installation Cost Include With JDK New Relic APM solution restart app with agent basic features: yes adv features: no no YourKit profiler maybe no, must pay for all features no JVisualVm profiler no yes yes JConsole JMX metrics no yes yes JMap/JHat prints memory map of Java process no, attaches to running process yes yes JStack prints thread dump of Java process no, attaches to running process yes yes Logs App Server Monitoring Sunday, September 8, 13
    • JMX Managed beans (mbeans) represent resources. They are a remotely accessible interface into a resource that provides stats and can be used to manipulate the resource. Examples include applications, JDBC drivers, garbage collection mechanisms, and thread pools. MBean Sunday, September 8, 13 MBean MBean
    • JMX The mbean server acts as the intermediary between the mbeans and the application requesting the information. Mbeans can be created, registered, and deleted through the server. MBean Server MBean Sunday, September 8, 13 MBean MBean
    • JMX JConsole Connector Adaptor MBean Server MBean Sunday, September 8, 13 MBean MBean
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Memory Information Sunday, September 8, 13
    • Thread Information Memory Information Sunday, September 8, 13
    • Thread Information Memory Information Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • JVM specific mbeans Sunday, September 8, 13
    • Tomcat Application Server mbeans JVM specific mbeans Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • CAN YOU DO THIS IN JVISUALVM? Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Tomcat Application Server mbeans JVM specific mbeans Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • The File Sunday, September 8, 13 vi tomcat/config/server.xml
    • The File Sunday, September 8, 13 vi tomcat/config/server.xml
    • The File Sunday, September 8, 13 vi tomcat/config/server.xml
    • SOLUTION THREAD POOLS NEEDED TO BE TUNED Sunday, September 8, 13
    • What Should Be The Size Of The Pool? • If the pool size is too small, then requests back up Image From: http://terrortrendsbulletin.files.wordpress.com/2012/11/airport-security-lines1.jpg Sunday, September 8, 13
    • What Should Be The Size Of The Pool? • If the pool size is too large, then system resources might become exhausted Sunday, September 8, 13
    • What Should Be The Size Of The Pool? • If the pool size is too large, then system resources might become exhausted Image From: http://static.ddmcdn.com/gif/airport-security-detector1.jpg Sunday, September 8, 13
    • What Should Be The Size Of The Pool? • If the pool size is too large, then system resources might become exhausted Images From: http://static.ddmcdn.com/gif/airport-security-detector1.jpg, http://graphics8.nytimes.com/images/2012/10/07/travel/07GETAWAY_SPAN/ 07GETAWAY_SPAN-articleLarge.jpg Sunday, September 8, 13
    • What Should Be The Size Of The Pool? • Pool size limiting factors – The amount of memory • In Java 6, the default thread stack size on Sparc is 512k for the 32-bit VM and 1024k for the 64-bit VM Hotspot Thread Stack Numbers: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html Sunday, September 8, 13
    • What Should Be The Size Of The Pool? • Pool size limiting factors – The amount of memory • In Java 6, the default thread stack size on Sparc is 512k for the 32-bit VM and 1024k for the 64-bit VM –The number of CPUs • Context switching could become an issue Hotspot Thread Stack Numbers: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html Sunday, September 8, 13
    • Little’s Law = the average response time for a request = the average arrival rate of requests = the approximate size for your pool Sunday, September 8, 13
    • AFTER THE FIX Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Image From: http://coedmagazine.files.wordpress.com/2012/08/carmelita-jeter-7.jpg Sunday, September 8, 13
    • Carmelita Jeter Fastest Woman Alive in the 100 meter sprint Image From: http://coedmagazine.files.wordpress.com/2012/08/carmelita-jeter-7.jpg Sunday, September 8, 13
    • My team’s 4x100 time is faster than your website. Get that fixed! Image From: http://www4.pictures.zimbio.com/gi/Carmelita+Jeter+Olympics+Day+14+Athletics+G48Jqgqo2nml.jpg Sunday, September 8, 13
    • LETS INVESTIGATE! Sunday, September 8, 13
    • Application Performance Monitoring Most application servers have an admin console which provides some monitoring information The quality of the admin console varies significantly between products Sunday, September 8, 13
    • Basic Information Tools Overview Installation Is It Free Include With JDK admin console, varies between app servers sometimes yes no New Relic APM solution restart app with agent basic features: yes adv features: no no YourKit profiler maybe no, must pay for all features no JVisualVm profiler no yes yes JConsole JMX metrics no yes yes JMap/JHat prints memory map of Java process no, attaches to running process yes yes JStack prints thread dump of Java process no, attaches to running process yes yes Logs App Server Monitoring Sunday, September 8, 13
    • Sunday, September 8, 13
    • Memory Information Sunday, September 8, 13
    • Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • New Relic Application Performance Monitoring (APM) tool that monitors web apps, mobile apps, servers, and databases Sign up and download agent from the New Relic Website Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • Logs Application, server, and database logs provide information developers consider important for evaluating the status of the application The quality of the logs varies significantly between products Tools like Splunk and Loggly can help analyze logs Sunday, September 8, 13
    • Basic Information Tools Overview Installation Is It Free Include With JDK Logs application, app server, framework, and db logs no yes yes App Server Monitoring admin console, varies between app servers sometimes yes no New Relic APM solution restart app with agent basic features: yes adv features: no no YourKit profiler maybe no, must pay for all features no JVisualVm profiler no yes yes JConsole m no yes yes JMap/JHat prints memory map of Java process no, attaches to running process yes yes JStack prints thread dump of Java process no, attaches to running process yes yes Sunday, September 8, 13
    • The Application Log Sunday, September 8, 13
    • The Application Log Varies between applications Sunday, September 8, 13
    • The Application Log Varies between applications Sunday, September 8, 13
    • The Application Log Spring AOP Sunday, September 8, 13
    • The Application Log Create The Aspect Sunday, September 8, 13
    • The Application Log Create The Aspect Sunday, September 8, 13
    • The Application Log Create The Aspect Sunday, September 8, 13
    • The Application Log Create The Aspect Sunday, September 8, 13
    • The Application Log Create The Aspect Sunday, September 8, 13
    • Application Server Log Sunday, September 8, 13
    • Application Server Log Information on requests host_access_log Sunday, September 8, 13
    • Application Server Log Information on requests host_access_log Sunday, September 8, 13
    • Application Server Log This is what the app server or framework developer considered important. Catalina.out Sunday, September 8, 13
    • Application Server Log This is what the app server or framework developer considered important. Catalina.out Sunday, September 8, 13
    • Application Server Log This is what the app server or framework developer considered important. applicationContext.xml Sunday, September 8, 13
    • Application Server Log This is what the app server or framework developer considered important. applicationContext.xml Sunday, September 8, 13
    • Application Server Log This is what the app server or framework developer considered important. applicationContext.xml Sunday, September 8, 13
    • Application Server Log This is what the app server or framework developer considered important. Catalina.out Sunday, September 8, 13
    • MySQL Log Sunday, September 8, 13
    • MySQL Log The only log on by default is the error log. Sunday, September 8, 13
    • MySQL Log You can turn on the slow sql log. Add the following to your /etc/my.cnf Sunday, September 8, 13
    • MySQL Log You can turn on the slow sql log. Add the following to your /etc/my.cnf Sunday, September 8, 13
    • MySQL Log You can turn on the slow sql log. Add the following to your /etc/my.cnf Query Time Sunday, September 8, 13 Rows Examined
    • MySQL Log You can turn on the slow sql log. Add the following to your /etc/my.cnf Query Time Sunday, September 8, 13 Rows Examined
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • SOLUTION USE INDEXES APPROPRIATELY Sunday, September 8, 13
    • AFTER THE FIX Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • In a Terminal Sunday, September 8, 13
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • Data In Tool Tools Logs App Server Monitoring New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, September 8, 13 App Response Time DB Response Time Error Info Thread Info GC Memory
    • Basic Information Tools Application Response Time Logs JConsole App Server Monitoring JVisualVm - No Profiler JStack New Relic Your Kit - No Profiler JMap Your Kit - With Profiler JVisualVM - With Profiler Sunday, September 8, 13 What is the impact of each tool?
    • Basic Information Tools Logs None JConsole None App Server Monitoring None JVisualVm - No Profiler None JStack Low New Relic Low Your Kit - No Profiler High JMap High Your Kit - With Profiler Very High JVisualVM - With Profiler Sunday, September 8, 13 Application Response Time Very High
    • Basic Information Tools Application Response Time Logs None JConsole None App Server Monitoring None JVisualVm - No Profiler Use In Production None JStack New Relic Low Your Kit - No Profiler High JMap High Your Kit - With Profiler Very High JVisualVM - With Profiler Sunday, September 8, 13 Low Very High
    • Basic Information Tools Application Response Time Logs None JConsole None App Server Monitoring None Use In Production JVisualVm - No Profiler None JStack Low New Relic Low Your Kit - No Profiler High JMap High Your Kit - With Profiler Use In Development JVisualVM - With Profiler Sunday, September 8, 13 Very High Very High
    • Basic Information Tools Application Response Time Logs None JConsole None App Server Monitoring None Use In Production JVisualVm - No Profiler None JStack Low New Relic Low Your Kit - No Profiler High JMap High Your Kit - With Profiler Use In Development JVisualVM - With Profiler Sunday, September 8, 13 Very High Very High
    • Basic Information Tools Overview Installation Is It Free Include With JDK Logs application, app server, framework, and db logs no yes yes yes App Server Monitoring admin console, varies between app servers sometimes yes no yes New Relic APM Solution restart app with agent no yes YourKit Profiler maybe basic features: yes advmust pay for no, features: no no no JVisualVm Profiler no yes yes basic: yes profile: no JConsole JMX Metrics no yes yes yes JMap/JHat prints memory map of Java process No, attaches to running process yes yes no JStack prints thread dump of Java process no, attaches to running process yes yes yes Sunday, September 8, 13 all features For Prod
    • Basic Information Tools Overview Installation Is It Free Include With JDK Logs application, app server, framework, and db logs no yes yes yes App Server Monitoring admin console, varies between app servers sometimes yes no yes New Relic APM Solution restart app with agent no yes YourKit Profiler maybe basic features: yes advmust pay for no, features: no no no JVisualVm Profiler no yes yes basic: yes profile: no JConsole JMX Metrics no yes yes yes JMap/JHat prints memory map of Java process No, attaches to running process yes yes no JStack prints thread dump of Java process no, attaches to running process yes yes yes Sunday, September 8, 13 all features For Prod
    • I encourage you to add a new tool to your tool box Image From: http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg Sunday, September 8, 13
    • I encourage you to add a new tool to your tool box Add the tool sooner rather than later Image From: http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg Sunday, September 8, 13
    • You should learn how to use the tools before you need them Image From: http://www.mediaemerging.com/wp-content/uploads/2010/11/Wrong-Tool-for-the-Job2.jpg Sunday, September 8, 13
    • WE WANT HAPPY USERS Images From: http://seattletimes.com/ABPub/2011/09/25/2016318932.jpg, http://static.guim.co.uk/sys-images/Sport/Pix/pictures/2007/12/08/PaulaRadcliffe_ap_SethWenig.jpg, http:// www.runnersworld.com/sites/default/files/rw/Jeter-Carmelita-Happy-300x200.jpg, http://www.thefumble.com/sites/captainwag.com/files/wp-content/uploads/2012/08/BOLTBDAY.gif Sunday, September 8, 13
    • Learn More. Stay Connected. Go try a new tool! Talk to us on Twitter: @springcentral Find session replays on YouTube: spring.io/video Sunday, September 8, 13
    • We are hiring: newrelic.com/jobs Sunday, September 8, 13
    • http://futurestack.io Sunday, September 8, 13