Downloaded 11 times
![Creating Unified IT Monitoring and Management in Your Environment Don Jones
75
“It isn’t just the servers, of course,” Lisa told me. “It’s what’s running on those servers:
databases, Web services, you name it. Someone else supports those different pieces, so
there used to be a lot of time spent arguing about whose fault something was.”
I asked her for an example of how things worked in their environment prior to
implementing a unified management system. She laughed and brought out a file that she’d
clearly held on to for some time. It looked like the text from a Help desk ticket’s notes.
Here’s the complete text, with names edited; I’ve added some [editorial] notes for items
that I had to ask Lisa to explain.
OPENED BY HelpDesk AT 2009‐06‐14 13:34
User states that BOS [an LOB application] is extremely slow. Have several e‐
r BOSDB02 responding slowly to pings. mails about this in the q also. Serve
ASSIGNED TO LHarte [this is Lisa]
NOTES BY LHarte AT 2009‐06‐14 15:26
BOSDB02 is working fine, apart from the fact that SQL is hogging 100% of the
CPU. Passing to DBA.
ASSIGNED TO DShields
NOTES BY DShields AT 2009‐06‐14 16:53
Probably the indexes again, SQL is taking longer to complete queries than it
tonight should. Will schedule indexes to be rebuilt
NOTES BY HelpDesk at 2009‐06‐15 10:44
Still getting calls on this
NOTES BY DShields AT 2009‐06‐15 11:12
Indexes rebuilt
ASSIGNED TO HelpDesk
NOTES BY HelpDesk AT 2009‐06‐15 11:34
SDB02 is still slow to ping Still getting calls that BO
ASSIGNED TO DShields
NOTES BY DShields AT 2009‐06‐15 13:12
SQL is still slow—looks like it is in disk IO. Fragmented disk? Need server
support.
ASSIGNED TO LHarte
NOTES BY LHarte AT 2009‐06‐15 13:47
Server disk shows less than 2% frag—not the problem. IO is slow because
s. Maybe your DB is fragged. I’ll call you. SQL is thrashing the disk
ASSIGNED TO DShields](https://image.slidesharecdn.com/ebook-creating-unified-it-monitoring-and-management-in-your-environment-chap-1-6-150526111558-lva1-app6891/75/E-book-creating-unified-it-monitoring-and-management-in-your-environment-chap-1-6-81-2048.jpg)
E book creating-unified-it-monitoring-and-management-in-your-environment-chap-1-6
- 1. sponsored by Creating UnifiedIT Monitoring and Management in Your Environment Don Jones
- 3. Creating Unified IT Monitoring and Management in Your Environment Don Jones i Introduction to Realtime Publishers by DonJones, Series Editor For several years now, Realtime has produced dozens and dozens of high‐quality books that just happen to be delivered in electronic format—at no cost to you, the reader. We’ve made this unique publishing model work through the generous support and cooperation of our sponsors, who agree to bear each book’s production expenses for the benefit of our readers. Although we’ve always offered our publications to you for free, don’t think for a moment that quality is anything less than our top priority. My job is to make sure that our books are as good as—and in most cases better than—any printed book that would cost you $40 or more. Our electronic publishing model offers several advantages over printed books: You receive chapters literally as fast as our authors produce them (hence the “realtime” aspect of our model), and we can update chapters to reflect the latest changes in technology. I want to point out that our books are by no means paid advertisements or white papers. We’re an independent publishing company, and an important aspect of my job is to make sure that our authors are free to voice their expertise and opinions without reservation or restriction. We maintain complete editorial control of our publications, and I’m proud that we’ve produced so many quality books over the past years. I want to extend an invitation to visit us at http://nexus.realtimepublishers.com, especially if you’ve received this publication from a friend or colleague. We have a wide variety of additional books on a range of topics, and you’re sure to find something that’s of interest to you—and it won’t cost you a thing. We hope you’ll continue to come to Realtime for your far into the future. educational needs enjoy. Until then, Don Jones
- 4. Creating Unified IT Monitoring and Management in Your Environment Don Jones ii Introduction to Realtime Publishers ................................................................................................................. i Ch apter 1: Managing Your IT Environment: Four Things You’re Doing Wrong ........................... 1 IT Management: How We Got to Where We Are Today ..................................................................... 1 Problem 1: You’re Managing IT in Silos ..................................................................................................... 3 Problem 2: You Aren’t Connecting Your Users, Service Desk, and IT Management ............... 6 Problem 3: You’re Measuring the Wrong Things ................................................................................. 8 Problem 4: You’re Losing Knowledge ..................................................................................................... 12 How Truly Unified Management Can Fix the Problems ................................................................... 13 Summary .............................................................................................................................................................. 14 Ch apter 2: Eliminating the Silos in IT Management ............................................................................... 16 Too Many Tools Means Too Few Solutions ........................................................................................... 16 Domain‐Specific Tools Don’t Facilitate Cooperation ........................................................................ 19 The Cloud Question: Unifying On‐Premise and Off‐Premise Monitoring................................. 21 Missing Pieces .................................................................................................................................................... 23 Not All of IT Is a Problem: Ordering, Routing, and Providing Services ..................................... 27 Coming Up Next… ............................................................................................................................................. 28 Ch apter 3: Connecting Everyone to the IT Management Loop ........................................................... 29 Starting the Loop: Connecting Monitoring to the Service Desk ................................................... 30 Making Changes: How to Find a Change Management Window .................................................. 35 Communicating: How to Bring Users into the Loop .......................................................................... 37 SLAs: Setting and Meeting Realistic Expectations .............................................................................. 39 Thin Tell Me What You Really k ................................................................................................................... 41 When Everyone Doesn’t Need to See Everything: A Multi‐Tenant Approach ........................ 42 Call It a Private Management Cloud: Allocating Costs ...................................................................... 43 Conclusion ........................................................................................................................................................... 44 Coming Up Next… ............................................................................................................................................. 44 Ch apter 4: Monitoring: Look Outside the Data Center .......................................................................... 45 Monitoring Technical Counters vs. the End‐User Experience ...................................................... 45
- 5. Creating Unified IT Monitoring and Management in Your Environment Don Jones iii How the EUE Drives Better SLAs ............................................................................................................... 46 How It’s Done: Synthetic Transactions, Transaction Tracking, and More ............................... 49 Top‐Down Monitoring: From the EUE to the Root Problem ......................................................... 50 Agent vs. Agentless Monitoring .................................................................................................................. 51 Monitoring What Isn’t Yours ....................................................................................................................... 54 Critical Capability: You Need to Monitor Everything ........................................................................ 57 Conclusion ........................................................................................................................................................... 59 Coming Up Next… ............................................................................................................................................. 59 Ch apter 5: Turning Problems into Solutions ............................................................................................. 60 Closing the Loop: Connecting the Service Desk to Monitoring ..................................................... 60 Re taining Knowledge Means Faster Future Resolution .................................................................. 62 Knowledge Bases ......................................................................................................................................... 63 Tickets as Knowledge Base Articles .................................................................................................... 64 Unifying the Knowledge Base ................................................................................................................. 65 Making Tickets an Asset ........................................................................................................................... 69 Pa st Performance Is an Indication of Future Results ........................................................................ 69 It’s the Performance Database ............................................................................................................... 72 Summary .............................................................................................................................................................. 73 Coming Up Next… ............................................................................................................................................. 73 Ch apter 6: Unified Management, Illustrated ............................................................................................. 74 Th e Case Studies ............................................................................................................................................... 74 Detecting and Solving Problems ........................................................................................................... 74 Fulfilling User Orders ................................................................................................................................. 79 A Shopping List for Unified IT Management ......................................................................................... 82 Ways to Buy Your Unified IT ....................................................................................................................... 84 Conclusion ........................................................................................................................................................... 85
- 6. Creating Unified IT Monitoring and Management in Your Environment Don Jones iv Copyright Statement ©2012 Realtime Publishers. All rights reserved. This site contains materials that have been created, developed, or commissioned by, and published with the permission of, Realtime Publishers (the “Materials”) and this site and any such Materials are protected by international copyright and trademark laws. THE MATERIALS ARE PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice and do not represent a commitment on the part of Realtime Publishers its web site sponsors. In no event shall Realtime Publishers or its web site sponsors be held liable for technical or editorial errors or omissions contained in the Materials, including without limitation, for any direct, indirect, incidental, special, exemplary or consequential damages whatsoever resulting from the use of any information contained in the Materials. The Materials (including but not limited to the text, images, audio, and/or video) may not be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, in whole or in part, except that one copy may be downloaded for your personal, non- commercial use on a single computer. In connection with such use, you may not modify or obscure any copyright or other proprietary notice. The Materials may contain trademarks, services marks and logos that are the property of third parties. You are not permitted to use these trademarks, services marks or logos without prior written consent of such third parties. Realtime Publishers and the Realtime Publishers logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. If you have any questions about these terms, or if you would like information about licensing materials from Realtime Publishers, please contact us via e-mail at info@realtimepublishers.com.
- 7.
- 8. Creating Unified IT Monitoring and Management in Your Environment Don Jones 2 As we started to evolve that IT car into a space shuttle, we gradually needed to allow for specialization. Individual systems became so complex in and of themselves that we needed domain‐specific experts to be able to monitor, maintain, and manage each system. Messaging systems. Databases. Infrastructure components. Directory services. The vendors who produced these systems, along with third parties, developed tools to help our experts monitor and manage each system. That’s really where things went wrong. It seemed perfectly sensible at the time, and indeed there was probably no other way to have done things, but that establishment of domain‐specific silos—each with their own tools, their own procedures, and their own expertise—was the seed for what would become a towering problem inside many IT shops. Fast forward to today, when our systems are vastly more complex, vastly interconnected, and increasingly not even hosted within our own data centers. When a user encounters a problem, they obviously can’t tell us which of our many complex systems is at fault. They simply tell us what they observe and experience about the problem, which may be the aggregate result of several systems’ interactions and interdependencies. Our users see a holistic environment: “IT.” That doesn’t correspond well to what we see on the back end: databases, servers, directories, files, networks, and more. As a result, we often spend a lot of time trying to track down the root cause of problems. Worse, we often don’t even see the problems coming, because the problems only exist when you look at the end result of the entire environment rather than at individual subsystems. Users feel completely disconnected from the process, shielded from IT by a sometimes‐helpful‐sometimes‐not “Help desk.” IT management has a difficult time wrapping their heads around things like performance, availability, and so on, simply because they’re forced to use metrics that are specific to each system on the network rather than look at the environment as a whole. The way we’ve built out our IT organizations has led to very specific business‐level issues, which have become common concerns and complaints throughout the world: •IT has difficulty defining and meeting business‐level SLAs. “The messaging server will be up 99% of the time” isn’t a business‐level SLA; it’s a technical one. “Email will flow between internal and external users 99% of the time” is a business‐level SLA, but it can be difficult to measure because that statement involves significantly more systems than just the email server. • IT has difficulty proactively predicting problems based on system health, and remains largely reactive to problems. • When problems occur, IT often spends far too much time pinpointing the root cause of the problem. • IT’s concept of performance and system health is driven by systems—database servers, directory services, network devices, and so forth—rather than by how users and the organization as a whole are experiencing the services delivered by those systems.
- 9. Creating Unified IT Monitoring and Management in Your Environment Don Jones 3 •IT has a tough time rapidly adopting new technologies that can benefit the business. Oxymoronically, IT is often the part of the organization most opposed to change, because change is usually the trigger for problems. Broken systems don’t help anyone, but an inability to quickly incorporate changes can also be a detriment to the organization’s competitiveness and flexibility. • IT has a really tough time adopting new technologies that are significantly outside the team’s experience or physical reach—most specifically the bevy of outsourced offerings commonly grouped under the term “cloud computing.” These technologies and approaches to technology are so different from what’s come before that IT doesn’t feel confident that they can monitor and manage these new systems. Thus, they resist implementing these types of systems for fear that doing so will simply damage the organization. • Even with modern self‐service Help desk systems, users feel incredibly powerless and out of touch when it comes to IT. All of these business‐level problems are the direct result of how we’ve always managed IT. Our processes for monitoring and managing IT basically have four core problems. Not every organization has every single one of these, of course, and most organizations are at least aware of some of these and work hard to correct them. Ultimately, however, organizations need to ensure that all four of these core problems are addressed. Doing so will immediately begin to resolve the business‐level issues I’ve outlined. Problem 1: You’re Managing IT in Silos Figures 1.1, 1.2, and 1.3 illustrate one of the fundamental problems in IT monitoring and management today. Figure 1.1: Windows Performance Monitor.
- 10.
- 11.
- 12.
- 13. Creating Unified IT Monitoring and Management in Your Environment Don Jones 7 Most IT teams do know the things that need to be communicated throughout the organization, for example: •SLAs • ey’re being met The current status of SLAs—whether th • Planned outages and degraded service • ices Average response times for specific serv • Known issues that are being worked on What most IT teams have a problem with is communicating these items consistently across the entire organization. Some organizations rely on email, which as I’ve already pointed out can be inefficient and not consistently effective. Some organizations will use an intranet Web site, such as a SharePoint portal, to post notices—but these sites aren’t directly integrated with the Help desk, making it an extra step to keep them updated and requiring users to remember to check them. Case Study Tom works as an inside salesperson for a midsize manufacturing company. Recently, the application that Tom uses to track prospects and create new orders started responding very slowly, and over the course of the day, stopped working completely. Tom’s initial action was to call his company’s IT Help desk. The Help desk technician sounded harried and frustrated, and told Tom, “We know, we’re working on it,” and hung up. Tom had no expectation when the system might return to normal, and was afraid to bother the Help desk by calling back for more details. Over the course of that day, the Help desk logged calls from nearly every salesperson, each of whom called on their own to find out what was going on. Eventually, the Help desk simply stopped logging the calls, telling everyone that, “A ticket is already open,” and disconnecting the call. Someone on the IT management team eventually sent out an email explaining that a server had failed and that the application wasn’t expected to be online until the next morning. Tom wished he had known earlier; although he’d originally planned to make sales calls all day, if he’d known that the application would be down for that long, he could have switched to other activities for the day or even just taken the day of f.
- 14.
- 15.
- 16. Creating Unified IT Monitoring and Management in Your Environment Don Jones 10 They see the components. Domain experts measure the performance of each component using technical metrics, such as processor utilization, response time, and so forth. When a component’s performance exceeds certain predefined thresholds, someone in IT pays attention. Figure 1.5, however, shows how a user sees this same application. Figure 1.5er’s perspective of a distributed application. The user doesn’t—often can’t—see any of the components. They simply see an application, and either it’s responding the way they expect, or it isn’t. It doesn’t matter a bit to the user if every single constituent component is running at an “acceptable level of processor utilization”—whatever that means. They simply care whether the application is working. This creates a major disconnect between the user population and IT, as Figure 1.6 illustrates. : Us
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26. Creating Unified IT Monitoring and Management in Your Environment Don Jones 20 There are two major reasons that these silos persist, and almost any IT professional can describe them to you: •“I don’t know anything about that.” Each domain expert is an expert in his technical area. The database administrator isn’t proficient at monitoring or managing routers, and doesn’t especially want to work with them anyway. There’s little real value in extensive technical cross‐training for most organizations, simply because their staff doesn’t have the time. Devoting time to secondary and tertiary disciplines reduces the amount of time available for their primary job responsibilities. • “I don’t want anyone messing with my stuff.” IT professionals want to do a good job, and they’re keenly aware that most problems come about as the result of change. Allow someone to change something, and you’re asking for trouble. If someone changes something in your part of the environment, and you don’t know about their activity, you’ll have a harder time fixing any resulting problems. Both of these reasons are completely valid, and I’m in no way suggesting that everyone on the IT team become an expert in every technology that the organization must support. minor adjHowever, the attitudes reflected in these two perspectives require some ustment. One reason I keep coming back to domain‐specific tools is because they encourage this kind of walled‐garden separation, and do nothing to encourage even the most cursory cooperation between IT specialists. Cooperation, when it exists, comes about through good human working relationships—and those relationships often struggle with the fact that each specialist is looking at a different set of data and working from a different “sheet of music,” so to speak. I’ve been in environments and seen administrators spend hours arguing about whose “fault” something was, each pointing to their own domain‐specific tools as “evidence.” Case Study Dan is an Active Directory administrator for his company, and is responsible for around two dozen domain controllers, each of which runs in a virtual machine. Peg is responsible for the organization’s virtual server infrastructure, and manages the physical hosts that run all of the virtual machines. One afternoon, Peg gets a call from Dan. Dan’s troubleshooting a performance problem on some of the domain controllers, and suspects that something is consuming resources on the virtualization host that his domain controllers need.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33. Creating Unified IT Monitoring and Management in Your Environment Don Jones 27 Discovery can also help identify components that don’t fit neatly within our technology silos, and that don’t “belong” to anyone. Infrastructure components like routers and switches are commonly‐used examples of these “orphan” components because not every organization maintains a dedicated infrastructure specialist to support these devices. However, legacy applications and servers, specialty equipment, and other components can all be overlooked when they’re not anyone’s specific area of responsibility. Discovery helps keep us from overlooking them. Not All of IT Is a Problem: Ordering, Routing, and Providing Services Most organizations tend to get into the habit of thinking of their IT department as “fire fighters.” IT exists to solve problems. That isn’t true, of course, and any organization probably (hopefully) depends more on IT to carry out day‐to‐day tasks and requests more than they rely on them to solve problems. But the day‐to‐day tasks are easy to overlook, whereas “fire fighting” gets everyone’s attention. The result of this way of thinking is that IT management tends to focus on tools that help make problem‐solving easier. Unified monitoring is exactly that kind of tool: If nothing ever went wrong, we wouldn’t need it. It’s there to make problem‐solving faster, primarily in the rformd availabilityareas of pe ance an . Right? Not quite. Truly unified management also entails making day‐to‐day IT tasks easier for everyone involved. Users, for example, need to order and receive routine services, from simple password resets and account unlocks to new hardware and software requests. I’ll make what some consider to be a bold statement and say that those routine requests should be treated in the exact same way as a problem. Look at any IT management framework, such as ITIL, and you’ll find that concept runs throughout: Routine IT requests should be part of a unified management process, which also includes problem‐solving. Consider some of these broad functional capabilities that a unified management (versus mere “monitoring”) can offer both to problem‐solving activities and to routine IT services: • Workflow—When problems arise, following a structured process, or workflow, can help make problem‐solving more consistent and efficient. Similarly, structured workflows can help make routine IT services more efficient and consistent. The workflows will be different for problem‐solving and for various routine services, but having the ability to manage and monitor workflows can be a real benefit. • Approvals—Workflows should include approvals. This capability is most obvious for routine services like hardware and software requests, security requests, and so on—but it can be just as important for problem solving. Not every problem can be fixed by changing a setting or rebooting a device; sometimes you’ll need to make a more significant change, and having the ability to formally route approval to make that change is a benefit.
- 34. Creating Unified IT Monitoring and Management in Your Environment Don Jones 28 •Routing. The specialist who fixes a problem is usually the last one to hear about it. Front‐line resources, such as your Help desk and your end users, are the first “responders.” Being able to select a problem category and have a ticket routed to the right individual helps speed problem resolution. The same is true for routine services: Things get done quicker when the right person has the request. Automated routing capabilities can help get the right person on the job more quickly and more accurately. • Self‐service. Reducing phone calls and manual email juggling is crucial to achieving better efficiency. Self‐service can help do that for both problems and routine requests. When users experience a problem, self‐service can allow them to submit tickets as well as help them solve the problem on their own, through a knowledge base. When users need routine service, self‐service helps them submit that request without having to engage additional IT services. • Service catalog. Part of self‐service is the ability to create an “online store” for services that users can request. There are more capabilities, of course, but we’ll cover them in upcoming chapters. These are simply some of the basic capabilities that we need in order to make both routine IT requests and problem‐solving more consistent and efficient. Coming Up Next… This chapter has been about breaking down the silos between technology specialties, or at least building doorways between them. That helps to solve one of the major problems in modern IT monitoring and management. The next chapter will tackle a somewhat more complicated problem: Keeping everyone in the management loop. It’s about improving communications. Unfortunately, communications are too often a voluntary, secondary exercise—we have to make an effort to communicate, and when we’re really feeling the pressure, it’s easy to want to put that effort elsewhere. So we need to adopt processes and tools that make communications more automatic, helping keep everyone in the loop without requiring a massive secondary effort to do so.
- 35.
- 36.
- 37.
- 38. Creating Unified IT Monitoring and Management in Your Environment Don Jones 32 Figure 3.2: Relating multiple incidents to a single problem. I’ve used a couple of keywords in the forgoing discussion and want to take a moment to specific define allythem in the context of this book: • An incident is something that happens in the environment, such as a failed server or ion. a slow applicat • IT staff create problem records to help manage the incident. Problems may in fact be associated with multiple incidents, as in the case of that overheating router, which caused multiple disparate failures throughout the environment. I’m going to start using those two terms more consistently from here on. Hopefully, some of the benefits of combining monitoring with problem solving will become clear. For example, more simplistic Help desk solutions allow multiple tickets to be opened against what is essentially the exact same issue. That can result in a lot of duplicated effort, as multiple IT team members attempt to work the issues on their own. It can also result in a lot of paperwork because solving the root cause then requires technicians to spend time laboriously closing each ticket. With a more sophisticated system in place, everything can be consolidated into a single, managed problem record. Doing so creates additional benefits, such as identifying solutions or workarounds, which I’ll discuss in upcoming chapters.
- 39. Creating Unified IT Monitoring and Management in Your Environment Don Jones 33 Problems and incidents, however, aren’t the only reason that users interact with IT. Hopefully, they’re not even the major reason your users interact with IT! Aside from reporting incidents, users also need to request routine services: advice, new hardware requests, routine change requests, access requests, and so forth. These interactions should be managed through a more formal workflow in which users submit their request, have it assigned to the appropriate technician after being approved, and be able to track the status st. of their reque Fora ex mple: 1. A user might visit a Web site to browse a “catalog” of items they can request, such as access to systems, changes to hardware, and so forth. 2. A user selects an item from the catalog, and provides whatever details are necessary to complete the request. ending proval. 3. A ticket is created in the service desk that represents the user’s request. Dep upon the request, the ticket might first be routed to the user’s manager for ap 4. Once approved, the ticket would be automatically routed to the appropriate technician or IT team for completion. 5. The user would receive status updates, perhaps via email, throughout this process, keeping them informed of its progress. The status updates would include a “completed” update once the request was finished. By using the same ticket‐based system employed for problem‐solving to address routine requests, IT technicians can rely on a single interface to manage their workload. Figure 3.3 shows what a routine request ticket might look like.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59.
- 60. Creating Unified IT Monitoring and Management in Your Environment Don Jones 54 Monitoring What Isn’t Yours The off‐premise stuff is where our traditional monitoring falls part. It’s unlikely that Amazon is going to give you detailed performance statistics into their Elastic Compute Cloud (EC2), and it’s unlikely that Microsoft would do so for Windows Azure. SalesForce.com isn’t going to send you database query response times or Web server CPU utilization. Even the hosting company where you’ve collocated your own servers isn’t going to be sending you detailed data about their routers’ dropped packet percentages, or any other infrastructure‐level statistic. Yet those numbers matter to you. If you have an application that relies on cloud computing components, collocated servers, Software as a Service (SaaS) solutions, or any other outsourced component, then the performance of those components affects your application performance your EUE. In short, when Amazon feels performance issues, so do your users. That’s where hybrid monitoring enters the picture. As Figure 4.7 shows, it usually takes the form of some external monitoring service, which collects key external performance information from major outsource providers—the “cloud components,” if you will, with the data collection shown as red lines—and reports back to your central monitoring console, as indicated by the green line. This is where a lot of what I’ve been outlining in the previous chapters really starts to come together: • You need both your internal systems and any external components monitored in the same view. There’s no way you can treat your systems as systems if you can’t get all of the constituent components into a single monitoring space. •The key competitive point for these hybrid monitoring services is the breadth of external components that they can monitor. Make sure you’re choosing one that can . monitor everything you’ve got—including all the outsourced dependencies • Monitoring the EUE becomes important because there may well be a lot of fluctuation with your external services and your use of them. For example, you won’t care that Azure is experiencing slow response times during periods when your users aren’t relying on that service. You only need to pay attention—and be alerted—when your users are experiencing a problem.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73. Creating Unified IT Monitoring and Management in Your Environment Don Jones 67 In fact, for tickets generated automatically from an alert, the ticketing system can potentially do a much better job of finding older tickets that actually relate to the problem. Because the system doesn’t mind taking the extra steps, it can include more detailed search criteria, such as the nature of the problem, the device or service affected, and so forth. A technician might not think to include all the detail, which would net them a large set of search results, which is often what discourages them from searching in the first place. By getting a narrow result set to begin with, the automatically‐referenced tickets are more likely to relate to the problem at hand. Such a system can be made even better if the Help desk system includes a couple of check boxes in its tickets. When closing a ticket, a technician should be able to independently indicate: •Whether this ticket contains a bona fide resolution to the problem. For example, sometimes the technician might solve the problem by looking at an older ticket, meaning the current ticket might not contain a lot of detail on how the problem was fixed. But if the technician has fixed the problem, and has filled in the details of what was done, then the current ticket can be marked as a “solution” ticket, making it bubble to the top of future search results. • Whether this ticket contains an end‐user consumable, self‐service solution. Many ticket‐tracking systems these days include both “public” and “private” notes fields, helping to ensure that end users don’t see anything that they might find upsetting or insulting, while accepting the fact that technicians will put that stuff into a ticket sometimes. By having a specific indication of which tickets are consumable outside the IT team, and which ones specifically contain user‐implementable solutions, you can build a self‐service knowledge base that actually works. Figure 5.6 shows how a system might implement this—in this case, rather than a checkbox, the system uses a “visibility” drop‐down to change a ticket from being held to being published.
- 74.
- 75. Creating Unified IT Monitoring and Management in Your Environment Don Jones 69 Making Tickets an Asset The overall idea is to take your tickets from being a way of tracking problems and work to being a complete life cycle for problem‐solving. In order to be effective, tickets‐as‐a‐ solution has to overcome some of the common human behaviors and implementation issues that have often been hurdles in the past: • Technicians don’t always search the ticket database—so that should happen automatically to some degree, with tickets being offered as potential solutions. •Technicians’ search skills aren’t always that great—so a unified system should, using the information it already has at its disposal, make a first attempt at finding relevant tickets. • Technicians’ writing skills aren’t always a priority—so the system should emphasize the need for complete solutions, management should focus on that as a metric, and technicians should be able to offer both “internal” and “externally‐consumable” versions of a solution, when appropriate. With the right system—particularly one connected to your monitoring system, making a truly unified environment—solutions to problems can be just a click away. Past Performance Is an Indication of Future Results Another way to use historical data is in developing service level expectations. I’m deliberately avoiding the phrase “service level agreement” because an SLA is a formal document that often includes some element of an organization’s politics. A service level expectation, however, is the level of service that, based upon past performance, you can realistically expect to achieve in the future. An SLA will ideally be based upon those real‐ world expectations—if you can provide them. One issue with many organizations’ SLAs is that they’re not actually based in reality. Someone will either make up an ambitious goal to “look good,” like promising 99.999% availability and then boldly stating that they will just “manage to that number.” Other times, someone will take an overly‐cautious approach when establishing service levels, forcing the organization to expect a lesser level of service than they realistically could. At fault are the tools we use. This gets all the way back to the first chapter of this book, when I wrote about the silos that IT tends to work in, and the varying domain‐specific tools we rely on to troubleshoot and solve problems. Those same domain‐specific tools are what we use to measure our existing performance levels. Because those tools don’t all speak a common language or use a common set of metrics, it’s actually really tough to figure out what our actual service levels are.
- 76.
- 77.
- 78.
- 79. Creating Unified IT Monitoring and Management in Your Environment Don Jones 73 The point of this figure is simply that you need to get performance data from every component—even the outsourced ones—into that database. Why? Two reasons: •This database is what’s going to show you what your performance really looks like on what you consider to be a normal day. This is where your performance expectations will come from, and it’s hopefully what you’ll use to derive more realistic and meaningful SLAs. • This database is what’s going to tell you when your performance is trending away from previously‐established norms. I’m not referring here to a situation where one component’s performance goes wonky due to a problem—live monitoring and alerting will take care of that. The database is there to spot the long‐term trends: “Hey, did you know that performance is down 1% from last month, which was down .75% from the month before? At this rate, you’ll be unable to meet your SLAs in 6 months.” And frankly, a good monitoring solution shouldn’t even show you the prototypical “trend line” in performance as its first step. The first step should be a simple dashboard: “You’re meeting your SLA, and based on current trends, will continue to do so for the foreseeable future.” Or, “You’re meeting your SLA—but barely, and based on trends, you aren’t going to be able to meet your SLA for more than a month or two.” From there, you can drill down into graphs and charts that give you more detail so that you can find the component or components that look like the current bottleneck, and start making plans to get more capacity in place before you miss your SLA. Summary Embracing the past to make a better future—that’s been the theme of this chapter. Whether you’re gathering ticket‐resolution information so that it can be used to solve future problems more quickly, or gathering performance information so that you can establish expectations and predict capacity, it’s all about keeping historical data and leveraging it to put the organization on a better footing for tomorrow. Coming Up Next… In the last chapter of this book, we’re going to step all the way back to the beginning and look at unified management from a case study perspective. I’ll use my consulting and field experience to construct a composite case study, drawing elements of unified management together to show you what a modern, truly‐unified environment can look like. I’ll share specific problems from each environment, and explain how unified management helped solve those problems more quickly and effectively.
- 80.
- 81.
- 82.
- 83.
- 84.
- 85. Creating Unified IT Monitoring and Management in Your Environment Don Jones 79 Posting a message, Kevin says, has helped the Help desk tremendously. “We have this Web portal where users can log tickets, and current system status is shown right there. So before they even open a ticket, they can see that we know there’s a problem. Once we trained them to trust us on that, they stopped logging duplicate tickets.” He admits that the training was a big step. “We didn’t do it initially,” he said, “but once users realized we were being pretty honest and consistent about posting problems, they started to trust us more. We had a big communications effort, and now there’s even a mailing list users can add themselves to so that they get a message whenever a system they use is affected. Being proactive cuts back on the Help desk volume a ton.” The benefits of a unified management system were pretty clear for this team: faster time to resolution, less passing the buck, and more proactive communications with their end users. The biggest challenge they faced? “A trust thing,” Lisa told me. “We had to learn to trust this new system to monitor everything as well as we could with the tools we were familiar with. So the first few times things went wrong, we went right back to what we knew to troubleshoot the problem. Once we realized that we were seeing the same data, we started trusting the new system more, and just started relying on it. We’ll still dig out the old tools if we have to dive deep into an affected system, but by the time we do that we know the problem is in that system, so we’re not wasting time. You don’t pass the buck to someone else at that point, you stay in that problem area until you spot the problem.” Fulfilling User Orders Kevin provides the link to the other side of the unified management story. “We’re not just responsible for opening tickets for problems. We also open tickets when routine changes need to be made.” I asked him to give me an example of how this was handled prior to the implem, and he pulled out an archived ticket. entation of their unified management system OPENED BY HelpDesk AT 2010‐08‐12 15:50 User BDOUDS needs a new SharePoint site deployed as iversitybid. User will be site admin. intranet/projects/un ASSIGNED TO JHoltz NOTES BY JHoltz AT 2010‐08‐13 08:27 Sent e‐mail to Bill’s manager confirming. Also sent e‐mail to Special Projects confirming. NOTES BY JHoltz AT 2010‐08‐16 11:12 waiting to hear from Special Projects. Bill’s manager, KHICKEY, confirms. Still NOTES BY JHoltz AT 2010‐08‐18 11:05 eft VM. Still waiting to hear from Special Projects. L NOTES BY HelpDesk AT 2010‐08‐20 10:34 User is asking for status.
- 86.
- 87.
- 88. Creating Unified IT Monitoring and Management in Your Environment Don Jones 82 We were discussing access permissions as an example, so I asked what happened when those needed to change. “They never did,” Kevin admitted. “Once you had access, you usually kept it until you left the company. We just didn’t keep track of it. Now, the catalog keeps track of it. If you don’t need something, you ‘return’ it to the store, it goes through whatever approvals, and we get a ticket to remove your access. Different managers also have to occasionally complete an attestation, which is where they review who has access to their resources and let us know if anyone needs to be removed, or if everyone can stay. We’re not the gatekeepers anymore.” I noted that an automated workflow wouldn’t necessarily guarantee a speedy response time. “Oh, users still have to wait two weeks for approvals, sometimes. But when they submit their request through the catalog, they can check the status of that request on their own. They can see that it hasn’t made it to us, and they can take it on themselves to bug their manager or whatever. We’re totally out of the loop until it’s approved, and they know that, because the request shows it hasn’t even made it to us yet.” Such a system does a better job of keeping users informed, and helping them to understand what’s really holding things up. A Shopping List for Unified IT Management I want to use this section to present a list of what I believe are the must‐have features of a true unified management system. As you’re evaluating solutions, make sure they offer these features—and make sure the features operate in a way that makes sense for your environment’s needs. •Workflow. Unified management solutions should offer workflows that can help automate responses and service management. Workflow construction should be as drag‐and‐drop as possible, involving as little programming as possible. • Agents. I know there’s a huge divide between people who are fine with deploying agents, and those who hate the idea; I’d suggest looking for a solution that supports both models. Agentless data collection is fine in some instances, although it can offer less performance and coverage than an installed agent. I think a hybrid approach is probably best for most organizations, and unified monitoring solutions ought to support that. • Alarm integration. When a problem arises, a unified management solution should obviously tell the designated individuals; it should also open a Help desk ticket and automatically search for related alarms from the past. Doing so will help speed up the time to resolution. This kind of “knowledge automation” is really crucial. • Approvals. As I’ve pointed out, “tickets” aren’t always for problems—sometimes they’re for new work, like change requests. A unified management system should support a review/approval workflow for these requests so that IT can be taken out of its traditional “gatekeeper” role and instead simply work those tickets that have been approved for implementation by the business.
- 89. Creating Unified IT Monitoring and Management in Your Environment Don Jones 83 •Discovery and deployment. A unified management solution should help you discover manageable nodes and services and deploy any necessary agents to monitor them. This discovery should happen more or less continuously, or at least be able to be run regularly, so that changes to your environment can be captured. • Routing. Tickets—whether for problems or for requests—should be automatically routed based on custom business rules that you can define. In other words, tickets should head straight to the correct implementer as quickly as possible. • Scheduling. A unified management system should have some kind of internal calendar that lets you schedule maintenance tasks. This functionality helps to resolve maintenance window conflicts and schedule work to happen at the right time. • Catalog. This is a key part of making a unified management solution part of a self‐ service, managed system. In addition, a catalog helps work toward bringing process compliance—such as ITIL compliance—into your environment. A catalog provides users with a list of “orderable products,” not unlike shopping at an online Web host. Users’ “purchases” translate into tickets, which go through review/approval prior to being passed to IT for implementation. • Communications. Users need to be able to submit requests, and users and your team must be able to review them from a familiar place. A Web portal is the traditional way to enable this communication, but systems that can integrate via users’ inboxes—which they’re in all the time, anyway—is even better. • Interface. You can’t have too many interfaces into a unified management system, and whatever solution you pick should offer both Web‐based and mobile‐friendly versions of its UI. • Metering. If you’re monitoring actual paying customers, you’ll need the ability to charge them for what they use. Even if you’re just dealing with internal “customers,” being able to perform “charge backs” for their IT resource consumption is going to be critical as business managers advance their management strategies. There’s no reason for IT to be seen purely as overhead when resources can—and should—be tracked back to the business components that are consuming them. • SLAs. A unified management system should assist you in both defining and monitoring service level agreements (SLAs) based on actual historic trends. • Trends. A unified management solution should include a performance database that lets you track historical performance trends. This database can be used to help define and report on SLAs as well as perform capacity planning. • Surveys. Closing the loop with your end users is crucial because technical SLAs aren’t the only way your success is being measured, whether you know it or not. Being able to poll users helps you define SLAs in their terms, creating more appropriate expectations.
- 90. Creating Unified IT Monitoring and Management in Your Environment Don Jones 84 •Reports. Look for reports and dashboards that provide managerial‐ and executive‐ level views of items such as workload, SLA compliance, and so forth. Heck, even dashboards that can be exposed to end users, helping them see that the environment is performing as it should, can go a long way toward helping IT be seen as more responsive and engaged with the business. • Visualization. Being able to visualize your environment can help make root cause analysis and problem resolution faster and easier. • Everything in one place. As I’ve written several times in this guide, a unified management system’s primary value is unity, or the ability to get all your performance concerns into a single place, using a single set of metrics, alarms, identifiers, and so forth. This singular view helps to break down the traditional domain‐based “silos” that IT is built around, and gets everyone focused on the root cause of a problem more quickly. • Knowledge retention. A unified management system should help your organization retain critical knowledge by turning Help desk tickets into an automated, searchable knowledge base. • Pre‐loading information. When an alarm generates a ticket, that ticket should include whatever details the unified management system can provide: IP addresses, response times, and so forth. The more information included in the ticket, the less the responder has to go look up, and the sooner they can start working on resolving the problem. This list obviously isn’t comprehensive but provides a starting point. If a potential solution offers these features and meets your organization’s specific needs, that solution is probably worth looking at in detail during an evaluation. Make sure you gain not only a “check mark” on these features but also a detailed explanation of how they’re implemented. Also, ensure that the implementation is one that will work within your organization’s requirements. Ways to Buy Your Unified IT I want to briefly outline different approaches that vendors take for delivering unified management solutions. Let me emphasize up front that I don’t regard any of these as “right” or “wrong;” there’s merely “what’s right for you,” which you’ll need to decide on your own. Typically, you’ll find that solutions of this kind are priced based on the number of nodes you need to manage, possibly also incorporating the number of users in your organization. A “node” is typically defined as any manageable device: a router, a server, and so forth. Some vendors are more creative than others with this portion of their licensing model; don’t let a complex model scare you off. In some cases, more complex license models are actually to your benefit because vendors are trying to precisely accommodate a wide range of scenarios. You should be more concerned about what you’re licensing.
- 91. Creating Unified IT Monitoring and Management in Your Environment Don Jones 85 For example, at one end of the spectrum, you’ll find what I call monolithic solutions. With these, you get—and pay for—every feature that the vendor offers, regardless of which ones you’ll need right away. I think it’s hugely important to make sure you’re acquiring a solution that can do everything you want, although I’m not sure you necessarily want to pay for all of that up front. In some cases, you may want to implement a solution in a phased approach, licensing just the functionality you need for each phase, thus allowing yourself to kind of “ramp up” into the full licensing and functionality of a product. The nice thing about monolithic solutions is that they’re often well‐integrated because everything is u in a single piece. delivered to yo There are also pluggable frameworks. I tend to view big frameworks like HP OpenView as fitting into this kind of model. With these solutions, you buy a base product, then add in the various bits and pieces you need to speak to your environment. These models offer a ton of flexibility, of course, and if you’re going with a big enough vendor, you should be able to find plug‐ins for every bit of functionality you need. These solutions run the risk of becoming a massive do‐it‐yourself project, though, and the plug‐ins aren’t always as well‐ integrated as you might like. Licensing can also be really, really complex because you’re g‐ins separatoften licensing the pluely from the base framework. Another model is the pay as you go approach. With this model, the solution offers all the functionality you might ever need, but you don’t “switch it all on” right away. Instead, you turn on the modules, or functionality, that you need immediately, and you just pay for that. As you add more responsibility to the solution, you pay a bit more. This setup is a bit more like a “cloud” model, where you can grow as large as you like but only pay for what you need right now. You’re not typically dealing with plug‐ins, or if you are, they’re usually all . delivered by the same solution vendor. I’m seeing more clients considering this approach The last thing you’ll need to think about is where the solution will live. In this age of “the cloud,” you actually have a choice of hosting your monitoring and management solution in your own data center or simply purchasing it as a hosted service that lives in the vendor’s data center. Either way, the vendor’s agents get installed into your environment. I won’t dig into the “on‐premise versus hosted” debate; you probably know what’s right for you, and you can certainly discuss that option with whatever solution vendors you’re investigating. Regardless which side of that debate you’re on, I think it’s nice to have a solution that offers both options. Conclusion Where, there you have it: unified management. The overall idea behind this book was simple: really focusing on the straightforward theme of “get everything in one place, and get everyone on one page.” It’s only “revolutionary” compared with the disjointed approach that our existing technology tools have more or less forced us into.
- 92.