DevOps for Windows Azure –
Surviving Failure
Michele Leroux Bustamante
michelebusta@solliance.net
Michele Leroux Bustamante
Managing Partner
Solliance (solliance.net)
CEO and Cofounder
Snapboard (snapboard.com)
Microsoft Regional Director
Microsoft MVP
Author, Speaker
Pluralsight courses on the way!
Blog: michelebusta.com
michelebusta@solliance.net
@michelebusta

2

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
If it’s not there
you can’t see it
Logs
 IIS Logs




HTTP Errors
Redirects
Internal Server Errors

 Windows Event Logs (Cloud Service Only)
 Performance Counters




Memory, CPU, Disk Access, Requests
Custom Counters (Cloud Service Only)

 Trace Logs


Trace.XXX()

4

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Setting up Web Site Logging

5

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Site Diagnostics

6

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Application Logs

7

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Demo

Web Site Logs
Setting up Cloud Service Logging
 Programmatically





During role initialization
Enable diagnostics and storage transfer settings
Enable windows logs
Enable performance counters

 WADcfg (preferred)



Used to be XML raw
Now VS has user experience

11

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Demo

Cloud Service Logs
Managing Logs
 What if you have many sites and cloud services


Aggregate into single table

 You’ll need filters




By application
By type of log
By deployment

 Can create filters with management tool


I.e., Cloud Storage Studio

13

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Make logging
EASY
Encapsulate Logging
 Simplify logging for the developer
 Create a DiagnosticsHelper




Methods for Debug, Info, Warning, Exception tracing
Methods for throwing exceptions throughout application
A method for serious exceptions, send email on exception to admin

 Be sure to swallow DiagnosticHelper exceptions


Logging should not introduce new issues, wrap that up!

 Can be shared



Web Sites
Cloud Services

15

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Demo

Encapsulating Logging
ETW is fast
DO NOT be afraid to use it
Event Tracing for Windows (ETW)
 Part of the Windows operating system since Windows 2000


Tools have improved significantly since then

 Tools for developers, for IT and for customers to work with
 Highly efficient tracing mechanism with decoupled architecture
 Enable or disable dynamically


No application restarts required

 Available to Azure Cloud Services and Web Sites


Trace.XXX()

18

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
ETW Tracing Components
ETW Consumer
Application

Events
(Persisted)

Events
(Real-Time)

Trace
Logs

Events

Enable/
Disable

ETW Session

ETW
Controller

Trace
Events

Trace
Events

Event
Provider
A

Event
Provider
B

Application

Associate
Providers with
Session

19

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
If you don’t look
you won’t know it’s there
Monitoring
 Performance Counters




Memory
CPU
Requests

21

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Demo

Monitoring
23

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Notably missing: RFC 418!!!!

24

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Be nice if someone
told ME!
Demo

Alerts
27

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Service Levels and Reports
 Customers want proof of SLA compliance
 Your average up time measured monthly




Not including scheduled downtime
Sometimes scheduled downtime is limited
99.95 availability is typical

 Your average request time measured monthly




Should exclude special services/page (i.e. data upload, large reports)
2s per request average is usually desired, an SLA may state 5s (whew)

28

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Analytics Products to Consider








Azure Portal (built in)
Paraleap
Stackify
NewRelic
Pingdom
Right Scale
JumpCloud

29

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
30

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
31

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
32

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
33

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
34

© DEVintersection. All rights reserved.
http://www.DEVintersection.com
It all comes down to
one thing…
CUSTOMERS
References
 Conference resources:


http://michelebusta.com

 See my snapboards:




Currently at the alpha site:
http://snapboardalpha.cloudapp.net/michelebusta
Will move these to snapboard.com/michelebusta when we go live on the
main site (SOON watch my blog for announcement)

 Contact me:



michelebusta@solliance.net
@michelebusta

37

© DEVintersection. All rights reserved.
http://www.DEVintersection.com

DevOps for Azure

  • 1.
    DevOps for WindowsAzure – Surviving Failure Michele Leroux Bustamante michelebusta@solliance.net
  • 2.
    Michele Leroux Bustamante ManagingPartner Solliance (solliance.net) CEO and Cofounder Snapboard (snapboard.com) Microsoft Regional Director Microsoft MVP Author, Speaker Pluralsight courses on the way! Blog: michelebusta.com michelebusta@solliance.net @michelebusta 2 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 3.
    If it’s notthere you can’t see it
  • 4.
    Logs  IIS Logs    HTTPErrors Redirects Internal Server Errors  Windows Event Logs (Cloud Service Only)  Performance Counters   Memory, CPU, Disk Access, Requests Custom Counters (Cloud Service Only)  Trace Logs  Trace.XXX() 4 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 5.
    Setting up WebSite Logging 5 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 6.
    Site Diagnostics 6 © DEVintersection.All rights reserved. http://www.DEVintersection.com
  • 7.
    Application Logs 7 © DEVintersection.All rights reserved. http://www.DEVintersection.com
  • 8.
  • 9.
    Setting up CloudService Logging  Programmatically     During role initialization Enable diagnostics and storage transfer settings Enable windows logs Enable performance counters  WADcfg (preferred)   Used to be XML raw Now VS has user experience 11 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 10.
  • 11.
    Managing Logs  Whatif you have many sites and cloud services  Aggregate into single table  You’ll need filters    By application By type of log By deployment  Can create filters with management tool  I.e., Cloud Storage Studio 13 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 12.
  • 13.
    Encapsulate Logging  Simplifylogging for the developer  Create a DiagnosticsHelper    Methods for Debug, Info, Warning, Exception tracing Methods for throwing exceptions throughout application A method for serious exceptions, send email on exception to admin  Be sure to swallow DiagnosticHelper exceptions  Logging should not introduce new issues, wrap that up!  Can be shared   Web Sites Cloud Services 15 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 14.
  • 15.
    ETW is fast DONOT be afraid to use it
  • 16.
    Event Tracing forWindows (ETW)  Part of the Windows operating system since Windows 2000  Tools have improved significantly since then  Tools for developers, for IT and for customers to work with  Highly efficient tracing mechanism with decoupled architecture  Enable or disable dynamically  No application restarts required  Available to Azure Cloud Services and Web Sites  Trace.XXX() 18 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 17.
    ETW Tracing Components ETWConsumer Application Events (Persisted) Events (Real-Time) Trace Logs Events Enable/ Disable ETW Session ETW Controller Trace Events Trace Events Event Provider A Event Provider B Application Associate Providers with Session 19 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 18.
    If you don’tlook you won’t know it’s there
  • 19.
    Monitoring  Performance Counters    Memory CPU Requests 21 ©DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 20.
  • 21.
    23 © DEVintersection. Allrights reserved. http://www.DEVintersection.com
  • 22.
    Notably missing: RFC418!!!! 24 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 23.
    Be nice ifsomeone told ME!
  • 24.
  • 25.
    27 © DEVintersection. Allrights reserved. http://www.DEVintersection.com
  • 26.
    Service Levels andReports  Customers want proof of SLA compliance  Your average up time measured monthly    Not including scheduled downtime Sometimes scheduled downtime is limited 99.95 availability is typical  Your average request time measured monthly   Should exclude special services/page (i.e. data upload, large reports) 2s per request average is usually desired, an SLA may state 5s (whew) 28 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 27.
    Analytics Products toConsider        Azure Portal (built in) Paraleap Stackify NewRelic Pingdom Right Scale JumpCloud 29 © DEVintersection. All rights reserved. http://www.DEVintersection.com
  • 28.
    30 © DEVintersection. Allrights reserved. http://www.DEVintersection.com
  • 29.
    31 © DEVintersection. Allrights reserved. http://www.DEVintersection.com
  • 30.
    32 © DEVintersection. Allrights reserved. http://www.DEVintersection.com
  • 31.
    33 © DEVintersection. Allrights reserved. http://www.DEVintersection.com
  • 32.
    34 © DEVintersection. Allrights reserved. http://www.DEVintersection.com
  • 33.
    It all comesdown to one thing…
  • 34.
  • 35.
    References  Conference resources:  http://michelebusta.com See my snapboards:   Currently at the alpha site: http://snapboardalpha.cloudapp.net/michelebusta Will move these to snapboard.com/michelebusta when we go live on the main site (SOON watch my blog for announcement)  Contact me:   michelebusta@solliance.net @michelebusta 37 © DEVintersection. All rights reserved. http://www.DEVintersection.com