what do_they_really_want_to_know


Published on

Many people in the IT industry need to produce reports and make presentations, sometimes to their technical colleagues and sometimes to more senior people. This paper discusses the best ways to present technical information to higher levels of management. The presenter needs to take account of the intended recipient's knowledge, interest and preconceptions. When it is necessary to present results that have significant consequences for the organisation, it is good psychology to prepare the ground in advance, perhaps by "leaking" the more contentious items so that they become part of the accepted pool of knowledge.
Depending on the circumstances, there is much to be gained by regular reporting in a known and accepted format. The paper therefore goes on to discuss the extent to which automation of report publishing is both desirable and practical, and shows the results of some recent work on the automatic interpretation of computer performance data.

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

what do_they_really_want_to_know

  1. 1. What do they Really Want to Know? Presenting technical reports to management Andy Mardo Principal Consultant Metron Technology Ltd AbstractMany people in the IT industry need to produce reports and make presentations, sometimes to their technicalcolleagues and sometimes to more senior people. This paper discusses the best ways to present technicalinformation to higher levels of management. The presenter needs to take account of the intended recipientsknowledge, interest and preconceptions. When it is necessary to present results that have significantconsequences for the organisation, it is good psychology to prepare the ground in advance, perhaps by"leaking" the more contentious items so that they become part of the accepted pool of knowledge.Depending on the circumstances, there is much to be gained by regular reporting in a known and acceptedformat. The paper therefore goes on to discuss the extent to which automation of report publishing is bothdesirable and practical, and shows the results of some recent work on the automatic interpretation of computerperformance data. _____________________________________________________________________________1. Introduction In order to make a successful presentation, especially one with financial implications, you should learn as much as you can about the targetThis paper is motivated by many years experience audience. You should try to understand:of delivering technical presentations, and of beingon the receiving end of other peoples efforts. As in • The pressures they are underall fields of endeavour, there have been both • Their prioritiessuccesses and failures. The first objective of this • Their technical awareness.paper is to share some of the authors understandingof what makes for a successful technicalpresentation. The second objective is to look at the 2.1 Pressures and Prioritiestechnology of report production. Most of the pressures on managers originate from2. Knowing the Target Audience their own superiors, who are possibly even more pressed for time and even less technically literate than they are. In order to keep his superiors happy,These days, everyone works under pressure. We your boss probably has to juggle a conflicting set ofall have to face deadlines, handle difficult clients, priorities. He will probably not react well to youand cover for sick colleagues. We wrestle with trying to redefine them. Dont, for example, spendunfamiliar software, fix bugs to keep the critical a lot of time explaining how important theapplications running, and risk our lives travelling to conclusions of your report are. What is deeplyand from the office every day. Sometimes, when important to you may be of only marginal interestthe stars and the dice are right, we find some time to him.to do our real jobs.The pressures faced by senior management may bedifferent in kind but are certainly not different in 2.2 Technical Awarenessdegree - indeed, they are likely to be fiercer. Thesepressures will affect the way in which your bossreacts to information from "down below". Most of us have what I will refer to as a "domain of competent understanding". For example, you may 1
  2. 2. have detailed knowledge of how a particular • Be precise. Ensure that the conclusions can beapplication works, or how to configure a Windows logically derived from the supporting material.network, or how to tune an Oracle database. Only Phrases like "tend to imply" or "may have avery rare individuals will have all these skills (or bearing on" might be suitable in a government-more). Your bosss domain of competent sponsored report on public transport, but haveunderstanding is unlikely to include your own no place in a technical presentation.domain as a subset. This means that unless you arecareful, you could overwhelm your target audiencewith information that they are not capable of 4. Presenting contentious or unpleasant resultsrelating to.The defence reaction of someone who is The nature of a surprise is that people reactoverwhelmed by the contents of a report, is usually unpredictably when confronted with one. Whatto latch on to some relatively minor ambiguity or happens next depends on whether the surprise isinconsistency in an effort to discredit the whole. pleasant or unpleasant.Dont fall into the trap of making this possible. You may find yourself in the fortunate position of being the bearer of good tidings. For example:2.3 Establishing a bond of trust • "Our website has taken far more hits than expected and the server is performing really well".Emotionally, your audience needs to trust what youare telling them. They will do this much more • "Last weeks memory upgrade solved all ourreadily if you make your presentation in terms that performance problems and the users are verythey can understand. They need to trust you happy".because they may subsequently need to pass the • "IBM has halved the price of their disk drivesrelevant information up to a higher level, and only and were going to be able to install twice asthe very brave, or the foolhardy, will stake their many as we budgeted for".reputation or credibility on anything that they dontfully understand. This is especially true if the But how often is the converse true?information that you are trying to convey has • "Our web server is overloaded and oursignificant financial implications. customers are deserting us in droves".Therefore, you need to organise your material in • "The memory upgrade made no difference.such a way that the audience is psychologically Our clients are threatening to sue".capable of trusting it. • "The lead time on hardware delivery has slipped to six months and were already nearly3. Guidelines for a trustworthy out of disk space". presentation 4.1 DenialThe author recommends the following guidelineswhen you are constructing or delivering a technical No one wants to hear any kind of bad news for thepresentation. first time during a formal presentation, especially if• Start at the beginning. Before proceeding to other people are present. If the news is bad the unknown, take time to describe what is enough, the well-known psychological already understood. This immediately phenomenon of Denial comes into play. "Denial" establishes your credibility with the audience - involves saying to oneself "This is so bad that I in their eyes, you are demonstrating that you dont want to believe it, so I will make up evidence know what you are talking about. to convince myself that it cant be true". The worse• Keep focussed. Eliminate any unnecessary the pain, the more compelling the imaginary details that do not have a direct impact on the counter-evidence becomes. If your information information being presented. Dont run the risk comes as no surprise, the "denial" reaction is of "glazed eye" syndrome, especially if you are unlikely to occur. Provided all other attributes of a presenting to a group. At the end of the trustworthy presentation are in place, your report presentation, there will probably be a will be, however reluctantly, accepted. discussion. Peer-group pressure makes it likely that everyone will feel the need to participate in this discussion. If anyone lost his way during the presentation, that persons contribution is likely to be negative. 2
  3. 3. 4.2 Preparing the ground reports are rarely or never printed out as hard copy.Recall that a good and trustworthy presentation • Daily reports. Data is typically presented atstarts from what is known, and proceeds towards its relatively high resolution, with data samplesconclusion in logical steps. So, if possible, you every two to five minutes. These reports areshould ensure that the bad situation you are intended to convey detailed information aboutreporting on is already part of the common fund of recent events. This type of report, and theaccepted knowledge before you start discussing it. following ones, may be distributed as hardThere is no pre-ordained way of doing this. copy, or (more likely) as an e-mail attachment.However, you may be able to take advantage of the • Weekly reports. These are presented at ainformal hierarchy that exists in any medium to lower resolution, perhaps with one data pointlarge organisation. This informal hierarchy exists for each aggregated hour, or possibly with justalongside, and largely independently of, the formal one aggregated point per day.reporting structure. By planting a word in the right • Monthly reports. The graphs in a monthlyear, you can ensure that everyone from the report will typically show one aggregated pointmanaging director to the car park attendant will per working day. A months worth of data issoon know what you want them to know. usually the least amount that can be used for trending purposes.5. Automatic Reporting • Year-to-date reports. The graphs in a year- to-date report will be at the lowest level ofNot all presentations are face-to-face. The act of e- resolution, certainly with no more than one point per day and more likely with one pointmailing a report to your boss is still a"presentation", even though you are not there in per week. Depending on the particular measurement(s) being reported on, the primaryperson when they read it. In many ways the objective of such a report is to display trendrequirements for clarity, appropriateness etc. ofsuch a report are much higher than if you present it information.in person, because the e-mailed report has to speakfor itself in every way - it cannot get any more 5.2 Report Management Techniquesassistance from you, the author.Automatic reporting is a practical technology thatis appropriate for many routine presentations. Automatic reporting has its dangers as well as itsFurther value can be added to automatic reports if benefits. How many reports are produced, pinnedthe information on the graphs can be summarised to the wall or posted on the intranet, and neverand interpreted in plain English. Even more benefit looked at again? In order to make reports relevant,can be obtained if the interpretations are used to interesting and useful, think about the following.trigger exception events, for example to warn • Top N reporting. Concentrate on the fewautomatically that a critical system will become busiest nodes, or devices, or users. Ensure thatoverloaded in some number of months time unless your automatic reporting application (if youcorrective action is taken soon. use one) can identify the Top N instances itself, even if they are different each time, without any intervention from you.5.1 Report types and purposes • Filtering. Ensure that reports are produced for periods of time that are important to theMany performance analysts are tasked with business. If you are in a 24*7 environment,producing regular reports on the performance of then all times are important. If yourone or several servers (possibly including organisation works a 5-day week and even letsmainframes) for which they are responsible. These you off for public holidays, then the non-reports are often, but not necessarily, in the form of working days should be filtered out of theWord or HTML documents containing annotated reports.graphs. They fall into a number of categories: • Correlation. What are the key resource• Near-real-time reports. Examples of these drivers? Which particular activities have the kinds of reports include continually updated biggest effect on the total pattern of system graphs or charts that are posted to an intranet loading? In many cases, correlation analysis web site, accessible through a standard lets you predict large-scale performance browser. Each new data sample will cause a changes caused by relatively small changes in new point to be displayed on the chart. These the nature of the workload or in user behaviour. 3
  4. 4. • Exception reporting. If you have 100 nodes - Print. in your installation, do you really want to • A common format or house style. Most report on all of them? Much more likely, you people react well to having the same kind of only want a report to be produced if some kind information presented in the same way each of exception condition is detected. time. This applies to:A good Automatic Reporting and Automatic - The sequencing of the report contentsAdvisor regime will incorporate all these facilities. - The appearance of the graphs and tables - The means of transmission (via e-mail, as5.3 Outline of Automatic Reporting hard copy, etc).technology If suitable automation is available, it means that in order to produce a regular report, the performanceIn order to produce a report manually, the analyst only needs to carry out the followingperformance analyst has to carry out the following actions once:sequence of actions: • Write an outline of the report. Ideally the• Write the outline of the report outline should contain mostly "boilerplate" text• Obtain the relevant data from whatever sources that is not going to change from one issue of are available the report to the next, though of course the analyst may want to edit the text to match the• Create graphs and tables of the data for the graphs that are actually generated on any required period of time particular occasion.• Insert these graphs and tables into the report • Create "sample" graphs and tables, from document existing data, to illustrate the report. The• Dispatch the finished report to its intended graphs should tell the story in the most recipient. understandable way. This point is expanded upon next.These activities can be time-consuming andtedious, especially when the only significant • Specify a schedule of when the report is to bedifference between one report and the next is the updated, for what period of time, and who is toname of the server that it relates to, or the period of receive it by what means of transmission.time that it covers.However, notice that all those different kinds of 5.4 Presenting information graphicallyreports have a number of common features thatmake them ideal candidates for automation. Theyare: Over the years, a great deal of work has been carried out to determine the kinds of graphical• A regular production date. For example, a presentation that make data most easily understood. daily report will be produced at 9 am each day Here are some practical examples. to display the previous days data. A weekly report will be produced every Monday. A monthly report will be produced on the first of 5.4.1 Numerical proportions every month, and so on.• A consistent analysis period. The analysis Figure 1 shows three ways of displaying the fact period is the period of time that the graphs in a that certain workloads are the largest contributors given report cover - a day, a week, a month, to the total loading on a system. The table of the year to date and so on. A common numbers, while true and accurate, is difficult to requirement for an analysis period is to go assimilate quickly. The horizontal bar chart shows back some number of days, weeks or months the relative magnitudes at a glance, but does not from the date of report production. convey the additional information that the elements• A known, stable recipient list. Each report add up to a particular total. The pie chart shows will be sent to a named individual or team, or the relative magnitudes and also conveys the is intended for saving in a "well-known information that the elements account for location", for example a particular folder on a "everything". Web server. Any of the following techniques could be used for distributing a report: - Via e-mail (as an attachment) - File Save or FTP (for Web-based reporting) 4
  5. 5. Figure 2 - Stacked area chart contrasted with aFigure 1 - Ways of representing proportions of a non-stacked bar charttotal 5.4.3 Magnitude or Variability?5.4.2 Areas and scaling You may be using the same set of data toSome variables, for example different categories of emphasise two (or more) different attributes of theCPU utilisation, can logically be summed to measurement in question. For example, you mightpresent a total value. If plotting two (or more) such want to display a graph of CPU utilisation for atvariables over time, it is good practice to stack the least two different reasons:individual values so that this total value is clearlydisplayed. In the example shown, the measured • To show how large (or small) the utilisation isvalues are hourly aggregations of what is in fact a on averagecontinuous variable, namely the CPU utilisation • To show how the utilisation varies over time.over time. A good rule of thumb is:The fact that the variable is continuous is mostclearly brought out by displaying the results as an • To emphasise magnitude, use an area chartarea graph rather than as stacked bars. Use stacked • To emphasise variability, use a line chart.bars when the values are snapshots made at specific This is illustrated in Figure 3.times, for example the number of users logged onat particular times of the day.If possible, show the results against a fixed verticalscale, rather than accepting whatever automaticdefault your graphics package determines for you.There are two reasons for this:• The viewer can see at a glance how much scope there is for a potential increase in the value of whatever is being presented• If the graph is going to be updated by the use of Automatic Reporting technology, fixed scaling contributes to consistency between one version of a report and the next. This makes it much easier to compare different versions of the same report that have been produced at different times. Figure 3 - Area chart compared with line chartThese points are illustrated in Figure 2. 6. Adding value to reports - Automatic Trending 6.1 Basic trends Daily or weekly performance reports are useful aids to performance management. They let you 5
  6. 6. and your colleagues understand what is happening - it will not happen until 20th August. Clearly theover a relatively short timescale, so that immediate trend has changed.or recent problems can be addressed and rectified.As mentioned earlier, monthly (or less frequent)reports are most useful if they include an elementof trending. This gives the significant additionalbenefit of identifying likely dates in the futurebefore which action will have to be taken in orderto avoid potential problems.Figure 4 shows two months worth of data about thetotal CPU utilisation of a particular server, with atrend line applied. As shown, the analysis windowinforms us that the trend line will reach a value of70% on 17th June. There may be a good reason forwanting to know when the trended daily averageCPU utilisation will reach 70%. This is a popularvalue among performance analysts. It is the typicalmaximum value beyond which you would not want Figure 5 - Updated graph and trend, withto run a server processing a critical workload. additional data for MarchAlthough it appears a relatively low value,remember that this trend is based on a dailyaverage, so it allows for some normal variation 6.2 Discontinuous trendsduring the day. Normally, the best way ofdetermining whether 70% (or any other particularvalue) is an appropriate cut-off point is to carry out Linear trends are extremely useful. However, theyanalytical modelling of the system being studied. have a particular shortcoming. Suppose you knowThis will show the performance impact of running that conditions are going to change in a few weeksat that loading level. or months time, in such a way as to affect the direction of the trend in a predictable fashion. Examples of this kind of situation are when you know that a particular new project is going to be rolled out onto the server in question, or that the number of users is going to increase suddenly, or that the server is going to be upgraded. In these circumstances, it is extremely useful to be able to apply a specific change to the nature of the trend at a future date. For example, you may want to specify that the slope of the trend is going to increase by a known proportion of its value at that time. This kind of change can be expressed as a "What-If". In the case where an automatic trend line has had a What-If change specified, you want this change to be honoured the next time the chart and its trend is automatically updated.Figure 4 - Trended graph with analysis window, Figure 6 shows an example of a What-If changewith underlying data from January and February specified to the trend on a particular chart. It shows the trend suddenly turning upwards at aAt this point, we are assuming an environment particular date, perhaps because the implementationwhere a chart and its associated trend can be of a new project will cause a greater rate ofupdated automatically. The previous chart covers increase in the server workload. As a result, theJanuary and February. Suppose it was generated value of 70% is predicted to occur earlier thanby an automatic reporting mechanism that produces originally thought, in fact on 15th May rather thana new year-to-date report on the first of every 17th June, as was the case for the trend without themonth. Figure 5 shows the chart that was specified change (figure 4).generated on April 1st, with the trend automaticallyre-calculated to fit the new March data as well asthe previous data. The analysis window nowshows a revised projection for 70% total utilisation 6
  7. 7. performance problem or to alleviate an existing problem. The following list gives examples of types of automatic analysis. • Top N analysis. This analysis can determine the few busiest or most resource-hungry users, devices, Oracle sessions or similar. Simply identifying them is a good start. Better, however, is to see their pattern of activity over time. • Mean value versus thresholds. This is a simple and straightforward check that the mean value of a measured data item is not too high orFigure 6 - Automatic trend with What-If change too low. Failure to stay within thresholdapplied bounds can be made to generate an exception event.Again, if the chart is automatically updated with • Proportion of time within threshold ranges.data for another month, the trend is recalculated, Typically the performance analyst will want toand the existing What-If is automatically applied to set two threshold levels for the value of certainthe revised trend. This gives a new projected date critical data items - a lower, warning thresholdfor the 70% utilisation level. and a higher, alarm threshold. It is straightforward to report automatically on theClearly, the ability to create and modify trends proportion of the measurements that fell intoautomatically gives significant added value to the each of the three ranges - below the warningcharts, and to the reports that they will be value (and therefore satisfactory), between theembedded in. warning and the alarm level, and above the alarm level. This gives valuable information7. Adding value to reports - Automatic about the relationship between peaks andInterpretation averages. • Variability around the mean value. A given set of measurements will have a mean value,Probably the best way of adding value to reports is and each individual measurement will typicallyto generate automatically an interpretation of the be some amount higher or lower than the meandata that is being presented. This relieves the value. It is often useful to categorise theanalyst from the task of modifying the report text measured value as "fairly constant", "ratherso that it matches the information in the charts. variable" or "very variable" based on theThe final sections of this paper present the outline proportion of time when the measured valuesof an Automatic Advisor system, intended to are close to or far away from the mean value.facilitate web-based publication of complete Again, if variability is a concern, this analysisperformance reports with minimal user can be made to generate an exception event.intervention. • Trended value versus thresholds. A very useful automatic analysis is to determine the7.1 Interpretation Techniques date at which the value of a particular metric is projected to exceed a certain threshold, or to reach some other predetermined boundaryGiven a chart with its underlying data, it is value (e.g. zero, 100% etc.) An exception canpractical to apply a number of analyses be generated on several different attributes ofautomatically. In most cases, the analysis can the trend, for example the fact that it will reachresult in the automatic generation of an "exception a boundary value or will cross a threshold valueincident", which will be e-mailed to a responsible on or before a predefined date.person or team. Additionally, the performanceanalyst can specify that reports be generated and • Correlation analysis. Used carefully and withpublished only if certain exception conditions in a sensible selection of metrics, Correlationfact occur. Depending on the circumstances, the Analysis can identify causal as well asresults of an automatic analysis can be turned into statistical relationships between data values.automatic advice, which gives guidance on actions For example, it is easy to identify UNIX usersthat should be taken to avoid a potential or Windows processes whose activity has a large effect on total CPU utilisation. Similarly, 7
  8. 8. the analysis can identify particular I/O devices that are associated with important warning metrics such as CPU Wait for I/O Completion.7.2 Automatic AdviceIn order for an Automatic Advisors reports to beaccepted, they must be:• Trustworthy - i.e. the conclusions are recognisably correct and are based on firm Figure 7 - System Summary Status report from an evidence Automatic Advisor application• Specific - i.e. the recommendations are specific enough to be acted on without the need for further detailed analysis Each of the underlying reports will contain detailed information about the selected aspect of the• Understandable - many advice systems in the selected system, including all the interpretation and past have proved more difficult to understand advice described previously. For any item that is than reading the relevant technical not shown as "happy", these drill-down reports will documentation itself. show trustworthy and specific advice for making itBased on the types of interpretation outlined in so.section 7.1, it is possible to offer trustworthy, Depending on the size of the installation and thespecific and understandable advice about such number of systems being reported on, thisthings as: Summary Status report could be produced at• CPU upgrades, for example if utilisation regular short intervals, so giving an effectively thresholds are currently being exceeded, or if continuous summary of the installations health. trend analysis shows that they will be exceeded soon 8. Conclusion• Memory upgrades, for example if paging and swapping rates are (or will soon be) high, or if cache hit rates are low Producing a good report manually takes a lot of effort. There are a number of psychological factors• Upgrades or tuning of the I/O subsystem, for to consider, in addition to the purely technical ones: example if particular devices are becoming hotspots, or if queuing is becoming a high • What are the needs and interests of the intended proportion if I/O service time. recipient? • How can the report be made credible and trustworthy?7.3 System-wide Exception Reporting - Traffic Lights A regime of automatic reports with intelligent interpretation can add significant value to the work of a system performance analyst. The reports canEven if reports are produced and distributed be interesting, credible, trustworthy - and perhapsautomatically, the recipient needs to be directed to most important, timely. The analyst is now free toany problems, rather than having to read through concentrate on the serious business of maintainingall the reports in order to find them. and enhancing the performance that is provided to"Traffic Lights" are a common and convenient way the people who really matter - the organisationsof showing the health (or otherwise) of a large customersnumber of systems at once. Figure 7 shows anexample of a status report covering a number ofsystems. By clicking on any of the icons, one cansee the full details of all the underlying reports .from which the colour of the icon (green, yellow orred) was generated.AuthorAndy MardoPrincipal Consultant 8
  9. 9. Metron Technology LimitedOsborne HouseTrull RoadTAUNTONTA1 4PXTel: +44 1823 259231Fax: +44 1823 334502Email: andy.mardo@metron.co.ukwww: http://www.metron.co.uk 9