Case of the Unexplained 3<br />Mark Russinovich<br />Technical Fellow<br />Microsoft Corporation<br />Session Code: WCL303...
About Me<br />Technical Fellow, Microsoft<br />Co-founder and chief software architect of Winternals Software <br />Co-aut...
Outline<br />Introduction<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br...
Case of the Unexplained…<br />This is the 2009 version of the “case of the unexplained” talk series<br />2007 & 2008 versi...
Troubleshooting<br />Most applications do a poor job of reporting unexpected errors<br />Locked, missing or corrupt files<...
Purpose of Talk<br />Show you how to solve these classes of problems by peering beneath the surface<br />Interpreting file...
Tools We’ll Use<br />Sysinternals: www.microsoft.com/technet/sysinternals <br />Process Explorer – process/thread viewer<b...
Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
The Case of the Slow Outlook Attachment<br />User would see CPU burst and Outlook would hang for 15+ seconds whenever they...
Process Monitor<br />Process Monitor is a real-time file, registry, process and thread monitor<br />It requires Windows 20...
The Case of the Slow Outlook Attachment (Continued)<br />Process Monitor trace of next received attachment implicated anti...
The Case of the Slow Outlook Attachment: Solved<br />Searched web for confirmation:<br />Checked AV settings found problem...
Process Explorer<br />Process Explorer is a Task Manager replacement<br />You can literally replace Task Manager with Opti...
The Case of the Periodic VMWare Freezes<br />Noticed CPU peg every 10 seconds and the desktop freeze when running VMWare <...
Processes and Threads<br />A process represents an instance of a running program<br />Address space<br />Resources (e.g., ...
Viewing Threads<br />Task Manager doesn’t show thread details within a process<br />Process Explorer does on “Threads” tab...
Thread Start Functions and Symbol Information<br />Process Explorer can map the addresses within a module to the names of ...
The Case of the Periodic VMWare Freezes: Solved<br />Opened Threads tab for System process and paused after a spike:<br />...
The Case of the Runaway Internet Explorer <br />Noticed a CPU spike and hovered over Process Explorer to see culprit:<br /...
The Case of the Runaway Internet Explorer: Investigation<br />The thread had a generic start address:<br />Required deeper...
Call Stacks<br />Sometimes a thread start address doesn’t tell you what a thread is doing<br />The stack might provide a h...
Viewing Call Stacks<br />Click Stack on the Threads tab to view a thread’s call stack<br />Lists functions in reverse chro...
The Case of the Runaway Internet Explorer: Stack Investigation<br />I double-clicked on the thread to see its stack:<br />
The Case of the Runaway Internet Explorer: What is GP.OCX?<br />Opened DLL view to see DLL’s version information:<br />DLL...
The Case of the Runaway Internet Explorer: Solved<br />Searched for NOS Microsystems:<br />Conclusion: Adobe uses gp.ocx, ...
Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
The Case of the Logon Script Hangs<br />Multiple users complained that logon would take three minutes<br />Investigation r...
The Case of the Logon Script Hangs (Continued)<br />User captured a Process Monitor trace after manually running Lisa Clie...
The Case of the Logon Script Hangs: Solved<br />Device error lead user to look at disks:<br />Working systems had Fujitsu ...
Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br...
The Case of the MMC Startup Failure<br />User would get an error every time they started an MMC snapin:<br />
The Case of the MMC Startup Failure: Solved<br />Ran Process Monitor and saw an Access Denied error on an IE registry key:...
The Case of the Favorite that Wouldn’t Save<br />User tried to change the URL for one of his IE favorites:<br />Trying to ...
The Case of the Favorite that Wouldn’t Save: Solved<br />Captured a Process Monitor trace:<br />AccessChk showed that fold...
The Case of the Persistent Executable<br />Noticed that opening volumes in Explorer was really slow<br />Volume context me...
The Case of the Persistent Executable (Continued)<br />Files reappeared after deleting, so monitored activity with Process...
Viewing Autostarts<br />Use Autoruns to see what’s configured to start when the system boots and you login<br />Windows Ms...
The Case of the Persistent Executable (Solved)<br />Process Explorer DLL search showed that amvo.dll loaded into Explorer ...
Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
Application Crashes<br />In most cases, there’s nothing you can do about application crashes <br />They are caused by a bu...
Finding the Crash Dump<br />On pre-Vista systems, finding the dump file is easy:<br />
Attaching to the Dying Process<br />Vista doesn’t save crash dumps for most crashes<br />Only if Microsoft requests a dump...
Identifying the Crashed Process<br />On Vista, the process name might not be enough to identify the instance that’s crashe...
Enabling Dump Archiving on Vista and Windows Server 2008<br />Or you can configure Vista SP1 and Windows Server 2008 to al...
Analyzing a Crash<br />Basic crash dump analysis is easy and it might tell you the cause<br />Requires Windbg and symbol c...
The Case of the Explorer Context Menu Crash<br />Explorer would randomly crash when the user right-clicked on a file<br />...
The Case of the Explorer Context Menu Crash (Cont)<br />Ran Process Explorer and looked at Explorer DLL view to find muang...
The Case of the Explorer Context Menu Crash: Solved<br />Was part of Icon editing software, which developer relied upon<br...
Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
Crashes and Hangs<br />Windows has various components that run in Kernel Mode, the highest privilege mode of the OS<br />O...
Online Crash Analysis<br />When you reboot after a crash, Windows offers to upload it to Microsoft Online Crash Analysis (...
Basic Crash Dump Analysis<br />Many times OCA doesn’t know the cause:<br />Basic crash dump analysis is easy and it might ...
The Case of the Crashed Phone Call<br />Laptop crashed during a Skype VOIP call<br />User reconnected and system crashed a...
The Case of the Crashed Phone Call (Cont)<br />Looked at file properties to determine what device the driver was for:<br /...
The Case of the Crashed Phone Call (Cont)<br />Right-clicked and checked Windows Update for newer driver:<br />Need to che...
The Case of the Crashed Phone Call: Solved<br />OEM site had older version:<br />Intel site had newer one:<br />Installed ...
Summary and More Information<br />A few basic tools and techniques can solve seemingly impossible problems<br />I learn by...
www.microsoft.com/teched<br />Sessions On-Demand & Community<br />www.microsoft.com/learning<br />Microsoft Certification ...
Track Resources<br /><ul><li>Want to find out which Windows Client sessions are best suited to help you in your deployment...
Want to talk face-to-face with folks from </li></ul>the Windows Product Team? <br />Meet us today at the <br />Springboard...
Required Slide<br />Complete an evaluation on CommNet and enter to win!<br />
Upcoming SlideShare
Loading in …5
×

Wcl303 russinovich

1,133 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,133
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Procexp\\vmware hang
  • procexp\\iexplorercpu spike
  • \\bsod\\Intel wireless&gt;
  • Wcl303 russinovich

    1. 1.
    2. 2. Case of the Unexplained 3<br />Mark Russinovich<br />Technical Fellow<br />Microsoft Corporation<br />Session Code: WCL303<br />
    3. 3. About Me<br />Technical Fellow, Microsoft<br />Co-founder and chief software architect of Winternals Software <br />Co-author of Windows Internals 4th and 5th edition and Inside Windows 2000 3rd edition with David Solomon<br />Author of TechNet Sysinternals<br />Home of blog and forums<br />Contributing Editor TechNet Magazine, Windows IT Pro Magazine<br />Ph.D. in Computer Engineering<br />
    4. 4. Outline<br />Introduction<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
    5. 5. Case of the Unexplained…<br />This is the 2009 version of the “case of the unexplained” talk series<br />2007 & 2008 versions covered different cases<br />Can view webcast on Sysinternals->Mark’s webcasts<br />Based on real case studies<br />Some of these have been written up on my blog<br />
    6. 6. Troubleshooting<br />Most applications do a poor job of reporting unexpected errors<br />Locked, missing or corrupt files<br />Missing or corrupt registry data<br />Permissions problems<br />Errors manifest in several different ways<br />Misleading error messages<br />Crashes or hangs<br />
    7. 7. Purpose of Talk<br />Show you how to solve these classes of problems by peering beneath the surface<br />Interpreting file and registry activity<br />Interpreting call stacks<br />You’ll learn tools and techniques to help you solve seemingly unsolvable problems<br />
    8. 8. Tools We’ll Use<br />Sysinternals: www.microsoft.com/technet/sysinternals <br />Process Explorer – process/thread viewer<br />Process Monitor – file/registry/process/thread tracing<br />Autoruns – displays all autostart locations<br />SigCheck – shows file version information <br />PsExec – execute processes remotely or in the system account<br />Pslist – list process information <br />Strings – dumps printable strings in any file<br />ADInsight – real time LDAP (Active Directory) monitor<br />Zoomit – presentation tool I’m using<br />Microsoft downloads:<br />Kernrate – sample-based system profiler<br />Visual Studio: Spy++ - Window analysis utility <br />Debugging Tools for Windows: Windbg application and kernel debugger: www.microsoft.com/whdc/devtools/debugging/Windbg<br />
    9. 9. Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
    10. 10. The Case of the Slow Outlook Attachment<br />User would see CPU burst and Outlook would hang for 15+ seconds whenever they received an attachment:<br />
    11. 11. Process Monitor<br />Process Monitor is a real-time file, registry, process and thread monitor<br />It requires Windows 2000 SP4 w/Update Rollup 1, XP SP2 or higher, Server 2003 SP1 or higher, Vista, or Server 2008 (including 64-bit versions of Windows)<br />It replaces Filemon and Regmon, but you can use Filemon and Regmon on older operating systems<br />Enhancements over Filemon/Regmon include:<br />More advanced filtering<br />Operation call stacks<br />Boot-time logging<br />Data mining views<br />Process tree to see short-lived processes<br />When in doubt, run Process Monitor!<br />It will often show you the cause for error messages<br />It many times tells you what is causing sluggish performance<br />
    12. 12. The Case of the Slow Outlook Attachment (Continued)<br />Process Monitor trace of next received attachment implicated antivirus:<br />
    13. 13. The Case of the Slow Outlook Attachment: Solved<br />Searched web for confirmation:<br />Checked AV settings found problematic option and disabled scanning:<br />
    14. 14. Process Explorer<br />Process Explorer is a Task Manager replacement<br />You can literally replace Task Manager with Options->Replace Task Manager<br />Hide-when-minimize to always have it handy<br />Hover the mouse to see a tooltip showing the process consuming the most CPU<br />Open System Information graph to see CPU usage history<br />Graphs are time stamped with hover showing biggest consumer at point in time<br />Also includes other activity such as I/O, kernel memory limits<br />
    15. 15. The Case of the Periodic VMWare Freezes<br />Noticed CPU peg every 10 seconds and the desktop freeze when running VMWare <br />Saw in the Process Explorer System Information graph that it was the System process:<br />
    16. 16. Processes and Threads<br />A process represents an instance of a running program<br />Address space<br />Resources (e.g., open handles)<br />Security profile (token)<br />A thread is an execution context within a process<br />Unit of scheduling (threads run, processes don’t run)<br />All threads in a process share the same per-process address space<br />The System process is the default home for kernel mode system threads<br />Functions in OS and some drivers that need to run as real threads<br />E.g., need to run concurrently with other system activity, wait on timers, perform background “housekeeping” work<br />Other host processes: svchost, Iexplore, mmc, dllhost<br />
    17. 17. Viewing Threads<br />Task Manager doesn’t show thread details within a process<br />Process Explorer does on “Threads” tab<br />Displays thread details such as ID, CPU usage, start time, state, priority<br />Start address is where the thread began running (not where it is now)<br />Click Module to get details on module containing thread start address<br />
    18. 18. Thread Start Functions and Symbol Information<br />Process Explorer can map the addresses within a module to the names of functions<br />This can help identify which component within a process is responsible for CPU usage<br />Requires symbol information:<br />Download the latest Debugging Tools for Windows from Microsoft (free)<br />Configure Process Monitor’s symbol engine:<br />Use dbghelp.dll from the Debugging Tools<br />Point at the Microsoft public symbol server (or internal symbol server if you have access)<br />Can configure multiple symbol paths separated by “;”<br />
    19. 19. The Case of the Periodic VMWare Freezes: Solved<br />Opened Threads tab for System process and paused after a spike:<br />Ftser2k was XM Radio USB/Serial driver<br />Stopping it didn’t remove spikes<br />Http.sys is IIS kernel-mode cache driver<br />Went to device manager and showed hidden devices<br />Stopped http.sys and hangs went away<br />Didn’t care about dependent services<br />
    20. 20. The Case of the Runaway Internet Explorer <br />Noticed a CPU spike and hovered over Process Explorer to see culprit:<br />That was unexpected, because had just installed Adobe Acrobat Reader and exited Internet Explorer<br />IE’s window wasn’t visible, but it was still in the process list<br />
    21. 21. The Case of the Runaway Internet Explorer: Investigation<br />The thread had a generic start address:<br />Required deeper investigation…<br />
    22. 22. Call Stacks<br />Sometimes a thread start address doesn’t tell you what a thread is doing<br />The stack might provide a hint:<br />The stack is a per-thread region of memory that records a history of function nesting<br />The bottom from (Function 3) is where the thread will continue executing<br />Function 1<br />Function 2<br />Function 3<br />
    23. 23. Viewing Call Stacks<br />Click Stack on the Threads tab to view a thread’s call stack<br />Lists functions in reverse chronological order<br />Note that start address on Threads tab is different than first function shown in stack<br />This is because all threads created by Windows programs start in a library function in Kernel32.dll which calls the programmed start address<br />
    24. 24. The Case of the Runaway Internet Explorer: Stack Investigation<br />I double-clicked on the thread to see its stack:<br />
    25. 25. The Case of the Runaway Internet Explorer: What is GP.OCX?<br />Opened DLL view to see DLL’s version information:<br />DLL Search Online didn’t return any useful results<br />
    26. 26. The Case of the Runaway Internet Explorer: Solved<br />Searched for NOS Microsystems:<br />Conclusion: Adobe uses gp.ocx, which had hit an infinite-loop bug<br />Terminated IE process to stop CPU usage<br />
    27. 27. Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
    28. 28. The Case of the Logon Script Hangs<br />Multiple users complained that logon would take three minutes<br />Investigation revealed that all complaints were from Dell Precision 670 workstations<br />But only some of the 670 workstations were affected<br />User configured Process Explorer to run during logon and saw Lisa Client consuming CPU:<br />Lisa Client was custom logon application that checked system for installed applications<br />Lisa Client CPU then went idle for several minutes, then exited and system would start acting normally<br />
    29. 29. The Case of the Logon Script Hangs (Continued)<br />User captured a Process Monitor trace after manually running Lisa Client <br />Saw three-minute delay correspond to device error:<br />Details column showed IOCTL_SCSI_PASS_THROUGH <br />Captured trace on working system and looked for IOCTL_SCSI_PASS_THROUGH operation<br />No device error and no delay:<br />
    30. 30. The Case of the Logon Script Hangs: Solved<br />Device error lead user to look at disks:<br />Working systems had Fujitsu disks <br />Systems with hangs had Seagate<br />Solution: <br />Temporary: wrote WMI script that queried disk type and would not launch Lisa Client on Seagate systems<br />Final: Application developers changed Lisa Client to avoid performing problematic command<br />
    31. 31. Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />Undocumented Settings<br />
    32. 32. The Case of the MMC Startup Failure<br />User would get an error every time they started an MMC snapin:<br />
    33. 33. The Case of the MMC Startup Failure: Solved<br />Ran Process Monitor and saw an Access Denied error on an IE registry key:<br />Checked permissions and Administrators had no access<br />Solution: added full-access for Administrators and MMC started successfully<br />
    34. 34. The Case of the Favorite that Wouldn’t Save<br />User tried to change the URL for one of his IE favorites:<br />Trying to save a new favorite resulted in a similar error:<br />
    35. 35. The Case of the Favorite that Wouldn’t Save: Solved<br />Captured a Process Monitor trace:<br />AccessChk showed that folder was Medium Integrity (IE requires Low):<br />Fixed integrity with Icacls and problem solved<br />
    36. 36. The Case of the Persistent Executable<br />Noticed that opening volumes in Explorer was really slow<br />Volume context menu indicated presence of Autorun.inf<br />
    37. 37. The Case of the Persistent Executable (Continued)<br />Files reappeared after deleting, so monitored activity with Process Monitor<br />File was recreated by Explorer, so looked at stack<br />
    38. 38. Viewing Autostarts<br />Use Autoruns to see what’s configured to start when the system boots and you login<br />Windows MsConfig shows a subset defined autostart locations<br />MsConfig doesn’t show as much information<br />
    39. 39. The Case of the Persistent Executable (Solved)<br />Process Explorer DLL search showed that amvo.dll loaded into Explorer and all its children<br />Found amv0.exe and used Autoruns to delete it from the system Run key<br />
    40. 40. Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
    41. 41. Application Crashes<br />In most cases, there’s nothing you can do about application crashes <br />They are caused by a bug in in the program<br />Only the developer can fix a bug<br />However, the crash may be caused by misconfiguration or an extension (a plugin)<br />Monitor the application’s crash with Process Monitor if it’s reproducible<br />Look for extensions in the crash file with Windbg<br />
    42. 42. Finding the Crash Dump<br />On pre-Vista systems, finding the dump file is easy:<br />
    43. 43. Attaching to the Dying Process<br />Vista doesn’t save crash dumps for most crashes<br />Only if Microsoft requests a dump for study and you send it in<br />When a crash occurs, don’t dismiss the crash dialog:<br />Launch Windbg and attach to the process<br />You can save a dump with the .dumpcommand<br />
    44. 44. Identifying the Crashed Process<br />On Vista, the process name might not be enough to identify the instance that’s crashed:<br />To determine the PID of the crashed instance, look at WerFault’s command line:<br />
    45. 45. Enabling Dump Archiving on Vista and Windows Server 2008<br />Or you can configure Vista SP1 and Windows Server 2008 to always generate and save a dump file<br />Create a key named:HKLMSoftwareMicrosoftWindowsWindows Error ReportingLocalDumps<br />Dumps go to %LOCALAPPDATA%CrashDumps<br />Override with a DumpFolder value (REG_EXPAND_SZ)<br />Limit dump history with a DumpCount value (DWORD)<br />
    46. 46. Analyzing a Crash<br />Basic crash dump analysis is easy and it might tell you the cause<br />Requires Windbg and symbol configuration<br />Once the dump is loaded, find the faulting thread<br />The debugger might identify it<br />If the debugger doesn’t, examine each thread stack looking for “fault”, “exception”, or “error” names<br />Examine the stack of the faulting thread to look for third-party plugins<br />If you suspect an extension:<br />Check for a new version <br />Uninstall it if the problem persists<br />
    47. 47. The Case of the Explorer Context Menu Crash<br />Explorer would randomly crash when the user right-clicked on a file<br />Attached to process and executed !analyze -v:<br />Didn’t know what muangys.dll was and because module was unloaded, Windbg provided no information<br />
    48. 48. The Case of the Explorer Context Menu Crash (Cont)<br />Ran Process Explorer and looked at Explorer DLL view to find muangys.dll:<br />File had no version information, but Strings identified the company and application:<br />
    49. 49. The Case of the Explorer Context Menu Crash: Solved<br />Was part of Icon editing software, which developer relied upon<br />No newer version<br />Solution: disable shell extension with Autoruns<br />
    50. 50. Outline<br />Sluggish Performance<br />Application Hangs<br />Error Messages<br />Application Crashes<br />Blue Screens<br />
    51. 51. Crashes and Hangs<br />Windows has various components that run in Kernel Mode, the highest privilege mode of the OS<br />OS components: Ntoskrnl.exe, Hal.dll<br />Drivers: Ntfs.sys, Tcpip.sys, device drivers<br />Kernel-mode components are privileged extensions to the OS have to adhere to various rules<br />Not accessing invalid memory<br />Accessing memory at the right “Interrupt Request Level”<br />Not causing resource deadlocks<br />When a kernel-mode component performs an illegal operation, Windows crashes (blue screens)<br />Crashing helps preserve the integrity of user data<br />A resource deadlock can hang the system<br />
    52. 52. Online Crash Analysis<br />When you reboot after a crash, Windows offers to upload it to Microsoft Online Crash Analysis (OCA)<br />Automated server generates a thumbprint of the crash and uses it as a key in a database<br />If the database has an entry, the user is told the cause and directed at a fix<br />
    53. 53. Basic Crash Dump Analysis<br />Many times OCA doesn’t know the cause:<br />Basic crash dump analysis is easy and it might tell you the cause<br />Requires Windbg and symbol configuration<br />Dump files are in either:<br /> WindowsMemory.dmp: Vista and servers<br />WindowsMinidump: Windows 2000 Pro and Windows XP<br />
    54. 54. The Case of the Crashed Phone Call<br />Laptop crashed during a Skype VOIP call<br />User reconnected and system crashed again<br />Minidump file pointed at Intel wireless driver:<br />
    55. 55. The Case of the Crashed Phone Call (Cont)<br />Looked at file properties to determine what device the driver was for:<br />Found device in Device Manager:<br />
    56. 56. The Case of the Crashed Phone Call (Cont)<br />Right-clicked and checked Windows Update for newer driver:<br />Need to check OEM site, so had to find version number<br />
    57. 57. The Case of the Crashed Phone Call: Solved<br />OEM site had older version:<br />Intel site had newer one:<br />Installed and crashes stopped<br />
    58. 58. Summary and More Information<br />A few basic tools and techniques can solve seemingly impossible problems<br />I learn by always trying to determine the root cause<br />Resources:<br />Webcasts of two previous “Case of the Unexplained “ talked<br />Sysinternals->Mark’s Webcasts<br />Sysinternals Video Library: in-depth dive on tools and troubleshooting<br />My blog<br />Windows Internals: understand the way the OS works<br />If you’ve solved one, send me a description, screenshots and log files!<br />I’ll send you a signed copy of Windows Internals<br />
    59. 59. www.microsoft.com/teched<br />Sessions On-Demand & Community<br />www.microsoft.com/learning<br />Microsoft Certification & Training Resources<br />http://microsoft.com/technet<br />Resources for IT Professionals<br />http://microsoft.com/msdn<br />Resources for Developers<br />www.microsoft.com/learning<br />Microsoft Certification and Training Resources<br />Resources<br />
    60. 60. Track Resources<br /><ul><li>Want to find out which Windows Client sessions are best suited to help you in your deployment lifecycle?
    61. 61. Want to talk face-to-face with folks from </li></ul>the Windows Product Team? <br />Meet us today at the <br />Springboard Series Lounge, or visit us at www.microsoft.com/springboard<br />Springboard Series<br />The Springboard Series empowers you to select the right resources, at the right technical level, at the right point in your Windows® Client adoption and management process. Come see why Springboard Series is yourdestination for Windows 7. <br />
    62. 62. Required Slide<br />Complete an evaluation on CommNet and enter to win!<br />
    63. 63. Required Slide<br />© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.<br />The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.<br />

    ×