Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making
Next
Download to read offline and view in fullscreen.

Share

Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production

Download to read offline

Link to video: https://youtu.be/aHHfq4WK9Jw

At Indeed, we're growing quickly, from our engineer headcount to the number of features we deploy. Over the last three years, we’ve had a 6x increase in engineers, and a 15x increase in number of deploys. We’re currently deploying over 700 new features each week. In this talk, we'll describe the infrastructure built to support, scale and automate our software development and product releases, and how any organization can use these tools and techniques to improve release velocity in the face of rapid growth. Specifically, we will discuss Hobo — an easy, standardized way for developers to run our application stacks in Docker. We’ll also describe Control Tower, which manages software releases by unifying all of the information about application features into a single interface. These tools allow engineers to focus on product development, while moving their work from idea to production as efficiently as possible.

Related Books

Free with a 30 day trial from Scribd

See all

Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production

  1. 1. Automation and Developer Infrastructure Empowering Engineers to Move from Idea to Production
  2. 2. Alex Thurlow Engineering Manager
  3. 3. I help people get jobs.
  4. 4. Indeed is the #1 external source of hire Unique Visitors (millions) Million unique visitors 2009 2011 2012 2013 2014 2015 0 20 40 60 80 100 120 140 160 180 2010 180M 180 million unique users 80.2M unique US visitors per month 16M jobs 60+ countries 28 languages 64% of US job searchers search on indeed each month
  5. 5. Alex Thurlow
  6. 6. Plus many more!
  7. 7. What should we build?
  8. 8. What’s best for the jobseeker?
  9. 9. How do we know?
  10. 10. Data-Driven Product Design go.indeed.com/ddpd
  11. 11. Learn Measure Build Lean Startup Cycle
  12. 12. Deliver Learn Measure
  13. 13. Data driven product design requires quick feedback
  14. 14. Engineering Velocity go.indeed.com/evtt
  15. 15. Engineering Velocity go.indeed.com/evtt Developer Productivity Software Architecture Release Process
  16. 16. Engineering Velocity go.indeed.com/evtt Developer Productivity Software Architecture Release Process
  17. 17. Service-Oriented Architecture
  18. 18. Make small, independent applications
  19. 19. Deliver Learn Measure Quickly!
  20. 20. Deliver Learn Measure
  21. 21. Engineering Velocity go.indeed.com/evtt Developer Productivity Software Architecture Release Process
  22. 22. Jon Hanks Software Engineer
  23. 23. I help people get jobs.
  24. 24. Hobo Automated developer provisioning
  25. 25. First goal for a new developer Push code to production in week 1
  26. 26. Deliver Learn Measure
  27. 27. 1:1 Mentorship
  28. 28. Setup = 2-3 Days
  29. 29. 1-2 days left to get to production
  30. 30. Team Rotation
  31. 31. Helps new developer understand Indeed
  32. 32. Rotate every 4 weeks through 3 teams
  33. 33. Rotate every 4 weeks through 3 teams and face the same setup challenges in each team. Frustration!!!
  34. 34. Experienced devs face a similar problem working on multiple projects or switching projects
  35. 35. Service-Oriented Architecture
  36. 36. Job Database Search Webapp
  37. 37. Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database
  38. 38. Need to run several interconnected applications
  39. 39. Total Developers Producing Code By Month 2014 Apr Jul Oct 2015 Apr Jul Oct 2016
  40. 40. 1:1 mentorship didn’t scale
  41. 41. Use technology to speed developer provisioning
  42. 42. Vagrant automates creating virtualized development environments
  43. 43. Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database Vagrantfile Virtual Machine
  44. 44. Success! (kind of)
  45. 45. Reproducible environments Our Solution—The Good
  46. 46. Our Solution—The Good Sharable between devs
  47. 47. Our Solution—The Good <1 day setup time
  48. 48. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  49. 49. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  50. 50. Download full machine image
  51. 51. Download and install required software
  52. 52. Start every application
  53. 53. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  54. 54. Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database Vagrantfile Virtual Machine
  55. 55. Job Service Resume service Location Service Employer Webapp Job Search Index Resume Database GeoIP Database Virtual Machine Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database Virtual Machine
  56. 56. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  57. 57. Shared RAM
  58. 58. Remaining Memory Virtual Machine
  59. 59. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  60. 60. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  61. 61. We’re all getting Macbooks! Hooray?
  62. 62. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  63. 63. Hobo (hō-bō) synonyms: vagrant
  64. 64. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  65. 65. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  66. 66. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  67. 67. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  68. 68. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  69. 69. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  70. 70. What is Docker?
  71. 71. Hypervisor Host OS Guest OS Bins & Libs Guest OS Bins & Libs VM VM Server with Virtual Machines Bins & Libs Host OS Server with Docker Containers DockerEngine Container1 Container2 Container3
  72. 72. Docker terminology Image An OS filesystem
  73. 73. Docker terminology Container A running instance of an image
  74. 74. Docker terminology Registry A server for storing images
  75. 75. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  76. 76. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  77. 77. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  78. 78. Our solution Create and publish a base image for each application type
  79. 79. Java CentOS
  80. 80. Java CentOS hobo-java Publish
  81. 81. Docker allows “Inheriting” Images
  82. 82. Java CentOS Tomcat
  83. 83. hobo-java Tomcat hobo-tomcat Publish
  84. 84. Ready to use Hobo Images T O M C A T
  85. 85. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  86. 86. Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database Virtual Machine
  87. 87. Docker is designed to have one process per container
  88. 88. Search Webapp Job Service Resume service Location Service Job Search Index Resume Database GeoIP Database
  89. 89. Search Webapp Job Service Resume service Location Service Job Search Index Resume Database GeoIP Database Employer Webapp
  90. 90. Networking is easy with One machine
  91. 91. Many machines = we need service discovery
  92. 92. Service Discovery Enabling services to find other services automatically
  93. 93. What is Consul? A key/value store geared towards service discovery Consul has both an HTTP interface and a DNS interface
  94. 94. PUT /v1/catalog/register {"Node": "user-service", "Address": "10.0.0.3"} $ dig +short user-service.node.consul 10.0.0.3
  95. 95. Now all config files can reference other containers by name!
  96. 96. How do we start a stack of multiple dependent applications?
  97. 97. [HOBO-SETTINGS] depends = user-service search-service location-service [job-service] type = application [job-service-mysql] type = resource [job-service-mongo] type = resource Hobo Config File
  98. 98. [HOBO-SETTINGS] depends = user-service search-service location-service [job-service] type = application [job-service-mysql] type = resource [job-service-mongo] type = resource Hobo Config File
  99. 99. [HOBO-SETTINGS] depends = user-service search-service location-service [job-service] type = application [job-service-mysql] type = resource [job-service-mongo] type = resource Hobo Config File
  100. 100. Hobo Config File user-service [HOBO-SETTINGS] depends = [user-service] type = application [user-service-mysql] type = resource
  101. 101. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  102. 102. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  103. 103. src test README build.xml
  104. 104. hobo src test README build.xml
  105. 105. hobo job-service job-service-mysql job-service-mongo hobo_config
  106. 106. job-service config Dockerfile hobo run.sh
  107. 107. Dockerfile FROM dockerreg.indeed.net/hobo/hobo-tomcat
  108. 108. job-service hobo config tomcat.xml mysql-conn.properties job-service.properties
  109. 109. job-service hobo config myconfig job-service.properties tomcat.xml mysql-conn.properties job-service.properties
  110. 110. job-service config Dockerfile hobo run.sh
  111. 111. run.sh Start/Stop commands
  112. 112. Environment variables run.sh
  113. 113. Volume mounts run.sh
  114. 114. What are volume mounts? They mount parts of the host filesystem into the container
  115. 115. What are volume mounts? They mount parts of the host filesystem into the container
  116. 116. What do we mount?
  117. 117. What do we mount? Built code
  118. 118. What do we mount? Configuration files
  119. 119. What do we mount? Log directories
  120. 120. What do we mount? Data directories
  121. 121. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  122. 122. git clone repository ant hobo-deploy Using Hobo
  123. 123. Results
  124. 124. 2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Apr Active Usage Increase Over Time
  125. 125. Projects Configured For Hobo 330
  126. 126. “Sometimes I need to run apps from other teams and the first thing I look for is ‘Is this hobo’d?’. If so I’m happy!” - Indeed Developer
  127. 127. 15 minutes 30 seconds
  128. 128. 10 times per week means 120 hours per year for each developer
  129. 129. Lessons Learned and Takeaways
  130. 130. Lessons Learned and Takeaways Volume mounts are useful
  131. 131. Lessons Learned and Takeaways Make it easy to use for the default case but fully customizable
  132. 132. Lessons Learned and Takeaways Docker is moving quickly
  133. 133. Engineering Velocity go.indeed.com/evtt Developer Productivity Software Architecture Release Process
  134. 134. Control Tower Automating Release Management
  135. 135. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version Yes Does the feature meet all release criteria? Yes
  136. 136. Complex and manual
  137. 137. 1 2 3 4 Goals of Our Release Process Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  138. 138. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  139. 139. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  140. 140. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  141. 141. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  142. 142. Goals of Our Release Process 4 Record release details for tracking, analysis, and repeatability 3 Quickly address bugs in releases 2 Understand cross-product and cross-team dependencies 1 Understand which features are being released
  143. 143. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version Yes Does the feature meet all release criteria? Yes
  144. 144. The life of a feature
  145. 145. Test Deploy & Record Develop Feature Define Release
  146. 146. We use standard development and project management tools
  147. 147. &
  148. 148. Git: with great power comes great complexity
  149. 149. Git flow
  150. 150. master
  151. 151. master jira/IND-123
  152. 152. master jira/IND-123
  153. 153. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Develop Feature =
  154. 154. Feature code can also be in libraries
  155. 155. common-webapp common-database jobservice-api Search Webapp
  156. 156. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Develop Feature =
  157. 157. Release Managers Execute Releases
  158. 158. Goals of Our Release Process 4 Record release details for tracking, analysis, and repeatability 3 Quickly address bugs in releases 2 Understand cross-product and cross-team dependencies 1 Understand which features are being released
  159. 159. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version Yes Does the feature meet all release criteria? Yes
  160. 160. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  161. 161. master release
  162. 162. common-webapp common-database jobservice-api Search Webapp
  163. 163. common-webapp 1.2.6 common-database 3.1.5 jobservice-api 1.0.1 Search Webapp
  164. 164. Build Run Tests Deploy to QA
  165. 165. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  166. 166. What’s in the box?
  167. 167. Look at a dashboard in Jira with everything that’s supposed to be in the release.
  168. 168. Look at Git log to see commits for features
  169. 169. Look at Git log for each new library version
  170. 170. Look at code reviews for each feature to make sure they’re complete
  171. 171. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  172. 172. Use Jira for Release Tracking
  173. 173. Bug Bug Feature Fix Version
  174. 174. Bug Bug Feature Fix Version Deploy Ticket
  175. 175. Bug Bug Feature Fix Version Other App Feature Deploy Ticket
  176. 176. Bug Bug Feature Fix Version Other App Feature Deploy Ticket Depends upon
  177. 177. Bug Bug Feature Fix Version Other App Feature Deploy Ticket Depends upon Depends upon
  178. 178. Test
  179. 179. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  180. 180. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  181. 181. Deploy!
  182. 182. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  183. 183. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  184. 184. master release jira/IND-123
  185. 185. master deploy/2016-06-22 jira/IND-123
  186. 186. jira/IND-123 master deploy/2016-06-22
  187. 187. master deploy/2016-06-22
  188. 188. JIRA updates Close released issues
  189. 189. JIRA updates Close deploy ticket
  190. 190. JIRA updates Mark Fix Version as “Released”
  191. 191. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  192. 192. Goals of Our Release Process 4 Record release details for tracking, analysis, and repeatability 3 Quickly address bugs in releases 2 Understand cross-product and cross-team dependencies 1 Understand which features are being released
  193. 193. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  194. 194. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  195. 195. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  196. 196. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  197. 197. Clicks and Git Commands >100 Release with 4 features
  198. 198. Extra Actions Per Feature ~13
  199. 199. 2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Total Developers Producing Code By Month
  200. 200. 2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Unique projects deployed per month
  201. 201. We needed to automate
  202. 202. Why automate rather than simplify? Data History Transparency
  203. 203. We needed to automate
  204. 204. Control Tower
  205. 205. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  206. 206. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  207. 207. What feature code is in libraries?
  208. 208. common-webapp 1.2.6 common-database 3.1.5 jobservice-api 1.0.1 Indeed Webapp
  209. 209. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  210. 210. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  211. 211. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  212. 212. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  213. 213. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  214. 214. master release jira/IND-123
  215. 215. master release jira/IND-123 IND-123 commits In release IND-123 commit Not in release
  216. 216. master release jira/IND-123
  217. 217. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  218. 218. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  219. 219. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  220. 220. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  221. 221. Test DeployDevelop Feature Define Release
  222. 222. Results
  223. 223. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  224. 224. Clicks and Git Commands >100 Release with 4 features
  225. 225. Control Tower Clicks <15 Release with 4 features
  226. 226. Extra Actions Per Feature ~13
  227. 227. Extra Actions Per Feature 0
  228. 228. Jul Oct 2015 Apr Jul Oct 2016 Apr Deploys per month
  229. 229. Deploys Per Project 1.9X
  230. 230. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  231. 231. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  232. 232. More knowledge = Less risk
  233. 233. of developers are release managers 57%
  234. 234. To start doing releases ~3 months
  235. 235. Lessons Learned and Takeaways
  236. 236. Lessons Learned and Takeaways Everything can be automated
  237. 237. Lessons Learned and Takeaways Engineers don’t like mystery
  238. 238. Lessons Learned and Takeaways Automate the process you have, not the process you want
  239. 239. Where do we go from here?
  240. 240. Too much human interaction
  241. 241. Train machines to handle releases
  242. 242. Engineering blog & talks - indeed.tech Open Source - opensource.indeedeng.io Careers - indeed.jobs Twitter - @IndeedEng Learn More
  • Gudimetla

    May. 11, 2020

Link to video: https://youtu.be/aHHfq4WK9Jw At Indeed, we're growing quickly, from our engineer headcount to the number of features we deploy. Over the last three years, we’ve had a 6x increase in engineers, and a 15x increase in number of deploys. We’re currently deploying over 700 new features each week. In this talk, we'll describe the infrastructure built to support, scale and automate our software development and product releases, and how any organization can use these tools and techniques to improve release velocity in the face of rapid growth. Specifically, we will discuss Hobo — an easy, standardized way for developers to run our application stacks in Docker. We’ll also describe Control Tower, which manages software releases by unifying all of the information about application features into a single interface. These tools allow engineers to focus on product development, while moving their work from idea to production as efficiently as possible.

Views

Total views

2,392

On Slideshare

0

From embeds

0

Number of embeds

1,777

Actions

Downloads

10

Shares

0

Comments

0

Likes

1

×