Automatic Remediation &Superfluous Ticket Elimination
Presentation OutlineWho’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActi...
Who’s That Guy?• Brian Dagan from• Based in Chantilly, VA• Approximately 5,400 endpoints
Presentation OutlineWho’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActi...
Superfluous Tickets/so͞oˈpər-fləəs/• More than is sufficient or required• Unnecessary or needless
Source: http://goo.gl/SYoDK
Superfluous Tickets Eliminated2011-06-10 through 2013-04-01
$45,000/year(Source: http://www.indeed.com)50 workweeks/year40 hours/week2,000 hours/year$22.50/hour75% efficiency (25% Re...
Eliminating Superfluous Tickets• Rebuilt Event Sets, Monitor Sets• Implemented Policy Management• Wrote Automatic Remediat...
Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• Name your Event Sets for the severity– Systems Mana...
Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• Use EID, Source & Description if possible– Systems ...
Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• Use EID, Source & Description if possible– Systems ...
Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• What’s worth waking up an Engineer?– “Critical” pri...
Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• Removing superfluous Events:– Which ticket queue wa...
Eliminating Superfluous TicketsAre you pondering what I’m pondering, Pinky?• Before you add an Event… think!Can I do somet...
Presentation OutlineWho’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActi...
Automatic Remediation Case #1Are hard disk errors actually legitimate problems?Use the “Run Script” option first:Note: For...
Don’t get SMART with me!I will pull this car over right now!S.M.A.R.T. = Self-Monitoring, Analysis and ReportingTechnology...
Automatic Remediation Case #1Are hard disk errors actually legitimate problems?Is the disk fixed or removable?• We don’t c...
Use The Variables, Luke!A true Jedi will always RTFM…Event Log Alerts populate variables when aparticular Event is encount...
Using Event Log Alert VariablesThe data is there… let’s use it!Within an Email Within a Procedure Description<at> #at# ale...
Using Monitor Alarm VariablesThe data is there… let’s use it!Within an Email Within a Procedure Description<ad> #ad# alarm...
Automatic Remediation Case #1Are hard disk errors actually legitimate problems?• Make an Event Set that catches diskevents...
Phase Phase Phase1 2 3RebuildEventSetsProfit
Automatic Remediation Case #1Are hard disk errors actually legitimate problems?
Automatic Remediation Case #1Are hard disk errors actually legitimate problems?Let’s make some Agent Procedures!• Use a “p...
Automatic Remediation Case #1Assembling your utility belt• SMART Health Checker:– Checks SMART health of a fixed disk– htt...
Automatic Remediation Case #1Best Practices - Re-using “generic” proceduresInitialization script:
Automatic Remediation Case #1Are hard disk errors actually legitimate problems?What can we “key off of” in these events?
Automatic Remediation Case #1The ciiiiircle of liiiife…Errors inconsistently identify the drive, sowe’ll check all fixed d...
Automatic Remediation Case #1Oppan Agent Procedure Style!
Automatic Remediation Case #11) Turn local Event Log Alert variables intoGlobal variables:2) Fire the “Initialization” Age...
Automatic Remediation Case #1Best Practices – Re-Using Common Agent ProceduresThe “Initialization” Agent Procedure:• Defin...
Automatic Remediation Case #11) Defines abbreviated paths to theDiskPart script files:2) Copies the DiskPart script files,...
Automatic Remediation Case #11) Uses DiskPart.exe to request a list ofphysical disks on the machine:2) Starts the Loop tha...
Automatic Remediation Case #11) If there are 10+ physicaldisks, procedure will alert theglobal:monitoringAlertEMailAddress...
Automatic Remediation Case #11) Sets the disk count,smartctl.exe query letter (uses lettersinstead of disk numbers, no ide...
Automatic Remediation Case #11) Checks the disk typeand proceeds with the next AgentProcedure only if it’s not a USB disk2...
Automatic Remediation Case #1
Automatic Remediation Case #11) Checks if SMART is enableda. Attempts to enable it if not (next slide)2) Formulates the SM...
Automatic Remediation Case #11) Attempts to enable SMART usingsmartctl.exe:2) Failure to enable SMART is accounted inthe n...
Automatic Remediation Case #11) Interprets the output ofsmartctl.exe and:a. …alerts if SMART can’t be enabled on a diskwit...
Automatic Remediation Case #1
Automatic Remediation Case #1
Automatic Remediation Case #1Are hard disk errors actually legitimate problems?• If SMART can be enabled, is the diskhealt...
Are We There Yet?Who’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActiona...
Actionable IntelligenceWWYAFLSDEDWTT?What Would Your Average Front-LineService Desk Engineer Do With This Ticket?
Eliminating Superfluous TicketsRebuilding Event Sets – Important QuestionsAre all hard diskerrors actuallylegitimateproble...
Do it now!
Seriously… Are We There Yet?Who’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example C...
Using ALARM/Event VariablesThe data is there… let’s use it!Reference:http://help.kaseya.com/WebHelp/EN/VSA/6030000/index.h...
There is a theory which states thatif ever anyone discovers exactlywhat the Universe is for and why itis here, it will ins...
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination
Upcoming SlideShare
Loading in …5
×

Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination

5,038 views
4,821 views

Published on

This session will show a “case study” of how CWPS uses built-in Kaseya functionality to eliminate over a hundred and fifty superfluous tickets per day, while enlisting utilities to produce “actionable intelligence” for those tickets that need human intervention. The “building blocks” present in Kaseya will be detailed, and content takeaways will be provided for attendees’ forays into the world of “automatic remediation.” Special emphasis will be placed on auditing exceptions when executing automatic remediation, as well as “WWYAFLSDEDWTT?”— the meaning of which will be revealed during the session.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
5,038
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • http://www.sxc.hu/photo/1383851/?forcedownload=1
  • http://www.sxc.hu/photo/1383851/?forcedownload=1
  • Duck is from Microsoft clipart
  • ScroogeMcDuck’s Vault = 3 square acres of DuckburgAssuming each coin is silver-dollar sized, the vault contains $27 trillion US dollarsDoes not include all of McDuck Industries(image from DeviantArt)
  • http://fc05.deviantart.net/fs71/f/2012/206/b/e/scrooge_mcduck_by_theblack_kat-d58iogw.png
  • http://www.sadtrombone.com
  • http://www.morguefile.com/archive/display/141445
  • http://en.wikipedia.org/wiki/File:Mazda3-pi.jpg
  • http://www.morguefile.com/archive/display/723699
  • http://www.morguefile.com/archive/display/141445
  • http://www.morguefile.com/archive/display/141445
  • http://www.morguefile.com/archive/display/141445
  • http://www.morguefile.com/archive/display/141445
  • http://www.morguefile.com/archive/display/833942
  • http://www.eventid.net/display.asp?eventid=11&amp;source=Disk
  • http://www.thejakartapost.com/news/2012/10/24/psy-speak-oxford-union.html
  • http://www.thejakartapost.com/news/2012/10/24/psy-speak-oxford-union.html
  • http://archive.org/details/ADTWhenE1958 (public domain)From 1958
  • http://www.morguefile.com/archive/display/62058
  • http://www.morguefile.com/archive/display/71834 (wolf)
  • http://www.morguefile.com/archive/display/840144
  • http://www.morguefile.com/archive/display/167655
  • http://www.morguefile.com/archive/display/2999
  • http://www.morguefile.com/archive/display/833942http://en.wikipedia.org/wiki/File:SchwarzeneggerJan2010.jpg
  • http://fc05.deviantart.net/fs71/f/2012/206/b/e/scrooge_mcduck_by_theblack_kat-d58iogw.png
  • http://fc05.deviantart.net/fs71/f/2012/206/b/e/scrooge_mcduck_by_theblack_kat-d58iogw.pnghttp://en.wikipedia.org/wiki/File:Douglas_adams_portrait_cropped.jpg
  • Kaseya Connect 2013: Automatic Remediation & Superfluous Ticket Elimination

    1. 1. Automatic Remediation &Superfluous Ticket Elimination
    2. 2. Presentation OutlineWho’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActionable IntelligenceAdditional Resources &Brief Q&A
    3. 3. Who’s That Guy?• Brian Dagan from• Based in Chantilly, VA• Approximately 5,400 endpoints
    4. 4. Presentation OutlineWho’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActionable IntelligenceAdditional Resources &Brief Q&A
    5. 5. Superfluous Tickets/so͞oˈpər-fləəs/• More than is sufficient or required• Unnecessary or needless
    6. 6. Source: http://goo.gl/SYoDK
    7. 7. Superfluous Tickets Eliminated2011-06-10 through 2013-04-01
    8. 8. $45,000/year(Source: http://www.indeed.com)50 workweeks/year40 hours/week2,000 hours/year$22.50/hour75% efficiency (25% Reddit )$30/hour5 minutes/ticketx 189 tickets/day15.75 hours/day= 2+ Engineers (~5,000 Agents)
    9. 9. Eliminating Superfluous Tickets• Rebuilt Event Sets, Monitor Sets• Implemented Policy Management• Wrote Automatic Remediation AgentProcedures for:– Hard disk errors– Low drive space– Service stoppages– Anti-virus removal & installation– Inheritance-based Policy application
    10. 10. Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• Name your Event Sets for the severity– Systems Management Pack does this:
    11. 11. Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• Use EID, Source & Description if possible– Systems Management Pack does this:
    12. 12. Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• Use EID, Source & Description if possible– Systems Management Pack also… doesn’t:– Be aware of why this is a problem!
    13. 13. Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• What’s worth waking up an Engineer?– “Critical” priority• What’s going to ruin someone’s day?– “High” priority• What needs addressed in a day or so?– “Monitoring” priority• What’s going to get you sued?– “Auditing” priority
    14. 14. Eliminating Superfluous TicketsRebuilding Event Sets – Best Practices• Removing superfluous Events:– Which ticket queue was the Alert in?• Locate the Event Set…– What was the Event ID?• Locate the exact Event in the Event Set…– What’s *unique* about the Alert• Modify the Event (or add another Event withspecific info to match the superfluous alertticket, and set it to Ignore):
    15. 15. Eliminating Superfluous TicketsAre you pondering what I’m pondering, Pinky?• Before you add an Event… think!Can I do somethingmore intelligentwith this alert?
    16. 16. Presentation OutlineWho’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActionable IntelligenceAdditional Resources &Brief Q&A
    17. 17. Automatic Remediation Case #1Are hard disk errors actually legitimate problems?Use the “Run Script” option first:Note: For automatic remediation to take place, the machine must be online touse “Run Script,” so do not use this for “Agent Offline” alerts 
    18. 18. Don’t get SMART with me!I will pull this car over right now!S.M.A.R.T. = Self-Monitoring, Analysis and ReportingTechnology; often written as SMART, is a monitoring systemfor computer hard disk drives to detect and report onvarious indicators of reliability, in the hope of anticipatingfailures. Reference: http://en.wikipedia.org/wiki/S.M.A.R.T.
    19. 19. Automatic Remediation Case #1Are hard disk errors actually legitimate problems?Is the disk fixed or removable?• We don’t care about Jim Bob’s iPod…Is SMART enabled on the disk?• If not, can we enable it?How’s the disk SMART health?• If it’s “PASSED,” ignore the Event Log!
    20. 20. Use The Variables, Luke!A true Jedi will always RTFM…Event Log Alerts populate variables when aparticular Event is encounteredhttp://help.kaseya.com/WebHelp/EN/VSA/6030000/index.htm#4853.htmMonitor Alarms populate variables whenthe Counter/Service/Process goes “out ofacceptable operating range”http://help.kaseya.com/WebHelp/EN/VSA/6030000/index.htm#1936.htm
    21. 21. Using Event Log Alert VariablesThe data is there… let’s use it!Within an Email Within a Procedure Description<at> #at# alert time<cg> #cg# event category<cn> #cn# computer name<db-view.column> not available Include a view.column from the database. Forexample, to include the computer name of themachine generating the alert in an email, use <db-vMachine.ComputerName><ed> #ed# event description<ei> #ei# event id<es> #es# event source<et> #et# event time<eu> #eu# event user<ev> #ev# event set name<gr> #gr# group ID<id> #id# machine ID<lt> #lt# log type (Application, Security, System)<tp> #tp# event type - (Error, Warning, Informational, SuccessAudit, or Failure Audit)Note: #subject# and #body# are also available but wouldn’t fit on this slide
    22. 22. Using Monitor Alarm VariablesThe data is there… let’s use it!Within an Email Within a Procedure Description<ad> #ad# alarm duration<ao> #ao# alarm operator<at> #at# alert time<av> #av# alarm threshold<cg> #cg# event category<db-view.column> not available Include a view.column from the database. Forexample, to include the computer name of themachine generating the alert in an email, use <db-vMachine.ComputerName><dv> #dv# SNMP device name<gr> #gr# group ID<id> #id# machine ID<ln> #ln# monitoring log object name<lo> #lo# monitoring log object type: counter, process, object<lv> #lv# monitoring log value<mn> #mn# monitor set namenot available #subject# subject text of the email message, if an email wassent in response to an alertnot available #body# body text of the email message, if an email was sentin response to an alert
    23. 23. Automatic Remediation Case #1Are hard disk errors actually legitimate problems?• Make an Event Set that catches diskevents only (excluding tape errors)• Set the Re-Arm time to an hour (recommended)
    24. 24. Phase Phase Phase1 2 3RebuildEventSetsProfit
    25. 25. Automatic Remediation Case #1Are hard disk errors actually legitimate problems?
    26. 26. Automatic Remediation Case #1Are hard disk errors actually legitimate problems?Let’s make some Agent Procedures!• Use a “parent” procedure that calls“child” procedures to better pinpointexceptions & iterate through all drives• Use the Agent Procedure Log to log keydata points for later troubleshooting• Perform output validation to confirm allcommands and third-party utilities beingemployed produce expected output
    27. 27. Automatic Remediation Case #1Assembling your utility belt• SMART Health Checker:– Checks SMART health of a fixed disk– http://smartmontools.sourceforge.net• Head.exe & Tail.exe– Text & file manipulation– http://unxutils.sourceforge.net/• Built-in Windows utilities like DiskPart:
    28. 28. Automatic Remediation Case #1Best Practices - Re-using “generic” proceduresInitialization script:
    29. 29. Automatic Remediation Case #1Are hard disk errors actually legitimate problems?What can we “key off of” in these events?
    30. 30. Automatic Remediation Case #1The ciiiiircle of liiiife…Errors inconsistently identify the drive, sowe’ll check all fixed disks in a loop:Can’t enableSMART? Drivefailure imminent?Fire a ticket!Unexpectedreturn value?Fire a ticket!Unexpectedreturn value?Fire a ticket!More than 10 disks?Fire a ticket!Removabledisk? Skipto the nextone…Query disk #__for typeInterpretresults of queryFixed disk?Check SMARTInterpretSMART resultsIncrement diskcounter
    31. 31. Automatic Remediation Case #1Oppan Agent Procedure Style!
    32. 32. Automatic Remediation Case #11) Turn local Event Log Alert variables intoGlobal variables:2) Fire the “Initialization” Agent Procedure3) Fire the “Copy Disk Utilities To Agent”Agent Procedure
    33. 33. Automatic Remediation Case #1Best Practices – Re-Using Common Agent ProceduresThe “Initialization” Agent Procedure:• Defines alert e-mail addresses:• Copies over a common suite of utilities:• Initializes the command output files:
    34. 34. Automatic Remediation Case #11) Defines abbreviated paths to theDiskPart script files:2) Copies the DiskPart script files,smartctl.exe (x86 or x64) anddiskpart.exe (if Win2K) to the machine
    35. 35. Automatic Remediation Case #11) Uses DiskPart.exe to request a list ofphysical disks on the machine:2) Starts the Loop that checks disks 0-9 tosee if they’re present in above output:
    36. 36. Automatic Remediation Case #11) If there are 10+ physicaldisks, procedure will alert theglobal:monitoringAlertEMailAddress:2) Fires the next Agent Procedure whichconfigures the query variables:
    37. 37. Automatic Remediation Case #11) Sets the disk count,smartctl.exe query letter (uses lettersinstead of disk numbers, no idea why), andpath to the DiskPart script for that disk:
    38. 38. Automatic Remediation Case #11) Checks the disk typeand proceeds with the next AgentProcedure only if it’s not a USB disk2) If it is a USB disk, we brag about havingprevented a Superfluous Ticket:
    39. 39. Automatic Remediation Case #1
    40. 40. Automatic Remediation Case #11) Checks if SMART is enableda. Attempts to enable it if not (next slide)2) Formulates the SMART health check:3) Runs the SMART health check commandand calls the next Agent Procedure tointerpret the output:
    41. 41. Automatic Remediation Case #11) Attempts to enable SMART usingsmartctl.exe:2) Failure to enable SMART is accounted inthe next Agent Procedure
    42. 42. Automatic Remediation Case #11) Interprets the output ofsmartctl.exe and:a. …alerts if SMART can’t be enabled on a diskwith issues (next slide) –or –b. …boasts if SMART reports that the drive hasa “healthy” SMART status (following slide)2) Accounts for potential failure ofsmartctl.exe by sending failures toglobal:monitoringAlertEMailAddress
    43. 43. Automatic Remediation Case #1
    44. 44. Automatic Remediation Case #1
    45. 45. Automatic Remediation Case #1Are hard disk errors actually legitimate problems?• If SMART can be enabled, is the diskhealthy? Or is failure imminent?
    46. 46. Are We There Yet?Who’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActionable IntelligenceAdditional Resources &Brief Q&A
    47. 47. Actionable IntelligenceWWYAFLSDEDWTT?What Would Your Average Front-LineService Desk Engineer Do With This Ticket?
    48. 48. Eliminating Superfluous TicketsRebuilding Event Sets – Important QuestionsAre all hard diskerrors actuallylegitimateproblems?Does that stoppedService need actualHuman interventionor just a swift kick?Is the disk that’s low on drive space aniPod, iPad or pagefile-only volume?Do I need to run an“Update Lists By Scan”on this server to catchrecently removed ordeactivated Services?What is the answer to the Ultimate Questionof Life, The Universe and Everything?Has the client toldus to not monitorsomething?
    49. 49. Do it now!
    50. 50. Seriously… Are We There Yet?Who’s That Guy?Eliminating Superfluous TicketsAutomatic Remediation BuildingBlocks & Example CasesActionable IntelligenceAdditional Resources &Brief Q&A
    51. 51. Using ALARM/Event VariablesThe data is there… let’s use it!Reference:http://help.kaseya.com/WebHelp/EN/VSA/6030000/index.htm#1936.htmhttp://help.kaseya.com/WebHelp/EN/VSA/6030000/index.htm#4853.htmAll presentation materials are available at:
    52. 52. There is a theory which states thatif ever anyone discovers exactlywhat the Universe is for and why itis here, it will instantly disappearand be replaced by something evenmore bizarre and inexplicable.There is another theory, whichstates that this has alreadyhappened.

    ×