Download to read offline
Uploaded byJulia Kulla-Mader
Nc tech4 good_presentation_2014_up
The document provides tips for troubleshooting website problems when a site goes down. It recommends checking for obvious causes like outages at the server, ISP, or DNS level. It also suggests using tools to check for server issues like high memory usage, full disk space, or outdated security patches. Finally, it advises interviewing staff, website visitors, and other sites sharing the same IP address to uncover any recent changes or unusual traffic that may have contributed to the problem. The overall goal is to find clues that help diagnose and avert a full-blown website crisis.
Nc tech4 good_presentation_2014_up
- 1. Welcome to theWebsite Emergency Room: Find and Pinpoint Problems When Everything Falls Apart Julia Kulla-Mader Jackson River @juliakm
- 2. It’s 6am. Yourwebsite is down. People care.
- 3. You are thewebmaster and it is your job to get the site back up.
- 4. The usual solution(restarting the server) isn’t working.
- 5. You need away to diagnose and troubleshoot difficult website problems.
- 6. Look for cluesto avert a website crisis.
- 7. 1. Check forobvious causes like a service outage
- 8. Is it downfor everyone?
- 9. Is the ISPdown?
- 10.
- 11. Did someone dosomething by accident? (User error)
- 12. Are all databasesonline? (Example of deleting a DB)
- 13.
- 14. Is the site“offline”?
- 15. Can you reproducethe issue? vs
- 16.
- 17.
- 18.
- 19. 2. Check withthe beat cops (tools)
- 20. Are there anyserver red flags?
- 21.
- 22.
- 23.
- 24. Are all ofyour security patches up-to-date?
- 25. Open Source Code(Drupal/Wordpress)
- 26. What does Googlesay?
- 27. Google Webmaster toolssays the site is hacked
- 28. 10 percent dropor increase in traffic
- 29. Odd search queries– View the site as a bot
- 30. 3. Interview everyoneconnected to the event
- 31. How do staffweb administrators use your site?
- 32. Maintain a descriptionof common operations on the site. Then walk through the process.
- 33. Did you changeanything recently?
- 34. How to websitevisitors outside of your organization use the site?
- 35. Look at themost popular parts of your site
- 36. Neighbors: How dothe other sites sharing your IP address use the site?
- 37.
Editor's Notes
- #3 The scenarios described today have been slightly changed to protect the innocent. Let’s imagine that you are the IT manager for a small nonprofit. You woke up to a flood of text messages from Pingdom alerting you that your site is down and has been since about 2 am. Even worse, the organization’s annual conference kicks off in a few hours.
- #4 While it would be great to be able to call someone else to fix this, it is clearly your responsibility. Part of your job is making sure that the website stays up.
- #5 You have some tech experience. You’ve tried to log into the server and couldn’t. So, you restarted the server. The site came back up for a couple of minutes and then once again crashed. You tried restarting two more times with the same result – a couple of minutes of uptime and then a crash.
- #6 Restarting isn’t working. Your executive director will be coming online any minute. How can you troubleshoot the problem? Photo Credit: http://www.flickr.com/photos/24480842@N03/3040615449/">pablo lizardo
- #7 The goal today is to step back and look for clues like a good detective. This is often incredibly hard to do when it feels like everything is falling apart. But for today, pretend that you are Sherlock Holmes or Olivia Benson.
- #8 In any crime show, they always start with the most obvious cause first. The first suspect almost always ends up being ruled out, but we still need to start with that person. I once worked for an organization where the website suddenly went down and no one knew why. It turned out that the server was in someone’s basement and their cat had sat on the off switch. As you gather clues you will begin to deduce the root cause of the issue, by developing a working theory, or theories.
- #9 Is it down for everyone or just me? (Website example) The best case scenario is that your website is just down for you. For example, since you are at a conference, perhaps there is something off with the wireless network. A good way to check whether this is true is to visit downforeveryoneorjustme. On this site, you can enter a website to check if it is down. Sometimes a site can be down just for your local network or for your router.
- #10 Another, less positive option, is that your Internet service provider is down. Almost always, ISPs will have a website (off of their servers) that you can check to see if the site is down. Or, Twitter is also always a good option.
- #11 Another, less likely, option is that your Domain Host is down while your ISP is up. This has happened with Network Solutions and AT&T in the past few years. If this happens, you don’t have a lot of options. Your site will be accessible via IP but the actual domain will not be accessible. Source: http://media.infospesial.net/image/p/2013/08/tips-jika-dns-google-down_a9729.jpg
- #12 Another unique possibility is that someone did something by accident. For example, we recently had a problem on a client site at work where the client accidentally managed to disable the Drupal panel powering the homepage, which led to a blank page.
- #13 Sometimes your databases will go down. Sometimes they will get accidentally deleted. You can easily check and see if the database is up via phpMiyAdmin or another tool.
- #14 Another easy to spot issue is that Apache, or Nginx or another web server powering your site, is just offline. I’ve experienced this a couple of times. A server will have been automatically restarted and for one reason or another, Apache will simply have never been turned on.
- #15 Another possibility is that your site is in the equivalent of maintenance mode. For example, in Drupal, you can take your site offline so that users can’t enter content. You need to remember to bring it back afterwards, which is sometimes easy to forget in the heat of the moment. When I was IT manager, we once had site visitors calling in because they couldn’t see anything. We could see everything fine because we were logged in. Little did we know that the site was in maintenance mode.
- #16 Provided a real problem exists. The next important thing to do is to try and reproduce.
- #17 Different browsers will behave in different ways. Okay, that was polite. The real thing is that a lot of the time IE will not display in standard ways. When you are checking for browsers, check IE from the version your site supports. Then, check the other browsers.
- #18 Is this issue only for Linux? Android? You need to identify what type of computer or phone the person is using. The next step is often to figure out how many people who visit your site use this type of device or operating system.
- #19 If this is a mobile issue, it’s good to identify quickly. As before, one of the first things you should do is check how many people are using this device. For example, we had a client where someone called in reporting a major problem with the donation form. It was for a very old version of Blackberry.
- #20 We’ve checked for obvious causes, now it’s time to check with all the tools that track information about your site.
- #21 There are a number of ways that a server can tell you it is unhappy. You do not necessarily have to be a systems administrator to read these tea leaves. You can just know when something is awry.
- #22 What your server is used for will determine how much memory and storage you need. Memory determines how fast you can retrieve information on your server. So, how fast Apache can return a page or how many applications (or for example websites) you can run on one server at the same time. When memory is running out on your server, everything will slow down and applications like PHP powering your site may start to crash. Anyone who has every had a computer with subpar memory knows what it feels like when your machine runs out memory – it slows to a crawl.
- #23 Disk space or “storage” is the size of your hard drive. You want to avoid having your server fill to capacity without any warning. You can mitigate this with per-user storage limits. For example, everyone except for the super admin may be limited to 1 GB. When your server is full, maintenance services like database backups will often stop running. This is an early sign. Later on, your whole server may crash again and again, leading to hard drive corruption. This is something you want to avoid.
- #24 High CPU typically means that you have a service running at 100 percent processing capacity. Depending on how many cores your server has, this could be 100 percent of the processor power of your machine or less. For example, if you have a dual core machine, it will have 50 percent of processing capacity in use. High CPU use – over 50 percent – can point to a performance problem. For example, high httpd use can crash Apache. CPU typically also corresponds to power use. More CPUs cost more money. TODO: Verify above with Ben/Matt.
- #25 Another scenario is that your code is running fine until some time of security vulnerability is exposed. You need to make sure that everything is up to date on your server.
- #26 If you are using a CMS, you also need to make sure that your core code and contributed plugins and modules are up-to-date. The benefit and drawback of Open Source is that you have thousands of people using the same code as you. This is great because problems are found and fixed by people who don’t necessarily work for you. It can be problematic because hackers can find security holes and exploit them. To really take advantage of Open Source, you need to be frequently updating your code.
- #27 Google offers a number of great resources for debugging.
- #28 Anyone who has had Google Webmaster tools identify their site has hacked likely remembers the experience. You start getting emails from Google that indicate your site will be removed from their search engine and all of your visitors see the scary message “This site may be hacked”. This site may be hacked: This warning means that Google detected some suspicious links or pages in your site that are not malware related in a way that would infect your users, but they still should not be there. Visiting this site may harm your computer: You’ll see this message if Google believes that a site can download malicious software onto your computer
- #29 If you see a big increase or decrease in traffic, particularly to one page, it may indicate a problem. It could be that one page is inaccessible or that spammers are targeting one page in particular. We had a huge increase in traffic and a crashing site at a previous job and it was because there were links to “Justin Timberlake naked” everywhere that were hidden
- #30 In the Timberlake example, we could have also seen that there was a lot of incoming traffic for “Justin Timberlake naked” as opposed to sustainability, the focus of the site
- #31 Like a good detective, the next step is to talk to people if we haven’t solved the problem yet. Put on your detective hat and let’s get started.
- #32 First, we want to look at how anyone making changes to the site interacts with the website.
- #33 The key thing to know in advance is how administrators work with your site. Do they typically make lots of changes each day. What content are they editing?
- #34 Once we’ve gone over basic options. Next we want to know what each person on the team as done recently. Example: base64 encoding issue. We ran into a problem recently where someone was dragging and dropping images in Firefox, which was triggering a problem with the SpamSpan module
- #35 Once we’ve made sure that no one internal is the cause of the problem, it’s time to look outside at who is visiting your site and why Source: https://www.digitalgov.gov/files/2014/03/Kidsgov-Usability-test-IMG_9987a-600-x-400.jpg
- #36 Since a typical site has thousands of visitors, interviewing each one may be challenging. Let’s try looking at the most popular pages as reported by Google Analytics first. Here in GA we can see that these are the most popular parts of the site. It looks like there’s a lot of incoming traffic to this page in particular. If we retrace our steps, it could be that Huffington Post has made this blog post their headline story. That explains the increase.
- #37 If we’ve eliminated everything else, it may be time to look at even more external factors. For example, I once had an Oscar predicting blog. Each Oscars, the site would eventually crash due to traffic, taking down every other site hosted on the same server.
- #38 You can do a reverse IP lookup to see what other websites may share your IP address.
- #39 You’ll remember that we started out today by looking at what to do when your site is down, people care, and the easy solutions like restarting your server aren’t working.
- #40 You need a way to diagnose and troubleshoot difficult website problems.
- #41 Today I hope we’ve shown you that you can look for clues to systematically avoid a website crisis. In particular, you can start out by looking for obvious causes, then check with the beat cops (tools like server red flags and google), and then talk to everyone connected to the event (people internally and externally)