This document discusses best practices for developing cloud services on Windows Azure. It recommends: 1. Storing state in Windows Azure storage and using loose coupling between components through queues to improve reliability given unreliable networks and hardware failures. 2. Versioning schemas and using rolling upgrades to minimize downtime when deploying updates. 3. Separating code and configuration, using configurable logging and alerts, to aid in debugging when things go wrong in the cloud.

1. Windows Azure: Cloud service development best practices  Sriram Krishnan Program Manager Microsoft Corporation sriramk@microsoft.com

2. 1980

4. void quicksort(int* array, int left, int right) { if(left >= right) return; int index = partition(array, left, right); quicksort(array, left, index - 1); quicksort(array, index + 1, right); }

7. simplicity

8. Act I Architecture

9. Networks are unreliable

10. Hardware fails

13. A few design choices

14. Big, reliable, expensive machine

15. Several commodity machines

16. Lots and lots of commodity machines

17. A few challenges

18. What do you do about that pesky thing called state?

20. Go horizontal Go stateless

21. Store state in Windows Azure storage

22. And it is the default out of the box!

23. Session state provider <system.web>... <sessionStatemode="Custom"customProvider="TableStorageSessionStateProvider"> <providers> <addname="TableStorageSessionStateProvider“ type="Microsoft.Samples.ServiceHosting.AspProviders.TableStorageSessionStateProvider" applicationName=“Foo” /> </providers> </sessionState> </system.web>

24. How do you deal with unreliable components?

26. Be loosely coupled

27. Use Windows Azure queues for separation of work

28. Default.aspx(Input + Making Coffee) LB

29. Tight coupling : Default.aspx.cs publicpartialclass_Default : System.Web.UI.Page { protectedvoid Button1_Click(objectsender,EventArgs e) { var order = txtOrder.Text; ProcessOrder(order); } protectedvoidProcessOrder(string order) { //Make some coffee! ... } }

30. LB Default.aspx(Input) Worker.cs(Make Coffee) Windows Azure Queues

31. Loose coupling : Default.aspx.cs publicpartialclass_Default : System.Web.UI.Page { protectedvoid Button1_Click(objectsender,EventArgs e) { var order = txtOrder.Text; QueueStorageqStore = QueueStorage.Create(_account); MessageQueueorderQ = qStore.GetQueue("OrderQueue"); orderQ.PutMessage(newMessage(order)); } }

32. Loose coupling : WorkerRole.cs public class WorkerRole : RoleEntryPoint { public override void Start() { QueueStorageqStore = QueueStorage.Create(_account); MessageQueueorderQ = qStore.GetQueue("OrderQueue"); while (true) { Message msg=orderQ.GetMessage(); if( msg != null) ProcessOrder(msg.ContentAsString()); } } protected void ProcessOrder(string order) { //Make some coffee! ... }

33. How do you deal with varying load?

34. Build a thermostat for your service

35. How do you deal with failures?

36. Use Windows Azure storage for valuable data

37. Be prepared to reconstruct local state…

38. …since it can disappear any time

39. Retry on transient failuresBut…

40. Be idempotent

41. Don’t launch a DoS attack on yourself

42. Be predictable

43. Avoid shutdown code

44. Know when to throttle and shed load

45. Case study: SmugMug and SkyNet

46. When is ‘good enough’ good enough?

47. The resiliency of email

48. Do all apps need the same guarantees?

49. It's a knob

52. Stateless front-ends Loose coupling Building a thermostat Retrying on failures Loosening consistency Recap

53. End of Act I

54. Act II Updates

55. Updates are hard

56. Hard to ‘undo’ a failed deployment

57. Need to deal with both code and schema changes

58. Code + data Update only one at a time

59. Code vN Data vN

60. Code vN +1 Data vN Code vN

61. Data vN+1 Code vN Data vN

62. Be compatible

63. If it looks like a duck and walks like a duck… http://www.flickr.com/photos/gaetanlee/298160415/

64. Use version numbers in schema

65. } Schema without versioning classEmployee : TableStorageEntity { public Employee(stringfirstName, stringlastName) : base(firstName, lastName) //partition key, row key {} publicstringJobTitle { get; set; } } ... varqResult = fromempin svc.CreateQuery<Employee>(EmployeeDataServiceContext.EmployeeTable) whereemp.PartitionKey == "Steve" && emp.RowKey == "Marx selectemp;

66. } Schema *with* versioning classEmployee : TableStorageEntity { public Employee(stringfirstName, stringlastName) : base(firstName, lastName) {} publicstringJobTitle { get; set; } publicint Version { get; set; } } ... varqResult = fromempin svc.CreateQuery<Employee>(EmployeeDataServiceContext.EmployeeTable) whereemp.PartitionKey == "Steve" && emp.RowKey == "Marx && emp.Version == 1 selectemp;

67. How do you do upgrades without downtime?

68. Windows Azure’s rolling upgrades

69. Stage Deployment Production Deployment Swap for zero downtime upgrade

70. + Stop + start for big changes or if downtime isn’t an issue

71. Future: Precise control

72. When is the best time to update a service?

73. Use the Pacific Ocean

74. Case study: Windows Live ID

75. Update code or data Maintain compat Versioning in schemas Rolling upgrades Recap

76. End of Act II

77. December 4th, 1996

81. Oh Oh!

83. Trace logs

84. Trickiest patch ever

86. Act III When things go wrong…

87. How do I debug?

88. Use the Windows Azure SDK and debug locally to find bugs

89. Separate code and config

90. Configuration files ServiceDefinition.csdef <ServiceDefinitionname="DemoService"> <WebRolename="WebRole"> <ConfigurationSettings> <Setting name="Color"/> </ConfigurationSettings> </WebRole> </ServiceDefinition> ServiceConfiguration.cscfg <ServiceConfigurationserviceName="DemoService"> <Rolename="WebRole"> <ConfigurationSettings> <Setting name ="Color" value ="Red"/> </ConfigurationSettings> </Role> </ServiceConfiguration>

91. Windows Azure’s configuration update mechanism

92. How do I debug the cloud?

93. Logging

94. Configurable logging <?xmlversion="1.0"?> <ServiceConfigurationserviceName=“DemoService”> <Rolename="WebRole"> <Instancescount="1"/> <ConfigurationSettings> <Settingname ="LogLevel"value ="Verbose"/> </ConfigurationSettings> </Role> </ServiceConfiguration> ... if (RoleManager.GetConfigurationSetting("LogLevel") == "Verbose") RoleManager.WriteToLog("Information", "Some log message");

95. {XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX} Tag data with unique ID to track across the system

96. How do I get notified when something bad happens?

97. ! Windows Azure’s alerts

98. ! Email / IM / Phone

99. The Big Red Button

100. Use the SDK Separate code and config Configurable logging Alerts The Big Red Button Recap

101. End of Act III

102. Credits & Acknowledgements James Hamilton http://research.microsoft.com/~jamesrh/ Emre Kicimanhttp://research.microsoft.com/~emrek/ Pat Hellandhttp://blogs.msdn.com/pathelland/ What really happened on Mars http://research.microsoft.com/~mbj/mars_pathfinder/ Flickr blog post http://code.flickr.com/blog/2008/09/26/flickr-engineers-do-it-offline/ Don MacAskillhttp://blogs.smugmug.com/don/

103. One final story

104. William of Ockhamc. 1288 - c. 1348

105. Numquamponendaestpluralitas sine necessitate Plurality ought never be posited without necessity

106. KISS

107. simplicity

108. sriramk@microsoft.comwww.sriramkrishnan.com

109. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.