From NASA to Startups to Big Commerce

72,832 views

Published on

From NASA to Startups to Big Commerce

  1. 1. FROM NASA TO STARTUPS TO BIG COMMERCE Building Maintainable, Scalable Projects DANIEL GREENFELD
  2. 2. SUCCESS!!! SUCCESS!!! SUCCESS!!! SUCCESS!!!
  3. 3. FAILURE.
  4. 4. Let’s get started
  5. 5. I ❤aerospace (you’ve been warned)
  6. 6. BIGPLANS
  7. 7. http://en.wikipedia.org/wiki/Soyuz_(spacecraft)
  8. 8. The Reality
  9. 9. http://en.wikipedia.org/wiki/Conestoga_(rocket) Conestega First commercial rocket! 1996!
  10. 10. https://www.flickr.com/photos/archer10/2215343914/
  11. 11. https://www.flickr.com/photos/ramcguire/19953122
  12. 12. Early Design Mistakes
  13. 13. Early Design Mistakes • Not understanding or knowing the requirements • Over or under architecting • Choosing the wrong tools • Premature Optimization • Management fiat made with ignorance or bad data
  14. 14. Technical Debt
  15. 15. Loan Against the Future • Get it done fast! • Grouping data poorly • Hardcoding • Upcoming technology will fix it • Leave testing and documentation for later Technical Debt
  16. 16. Hard to Maintain Enhance Scale Projects
  17. 17. 2006
  18. 18. http://science.nasa.gov More NASA work
  19. 19. Get It Done!!! https://www.djangopackages.com/
  20. 20. Bad Practices @ PyCon 2011 https://www.flickr.com/photos/pydanny/5670716870/
  21. 21. Had mistakes at start
  22. 22. We insisted on rebuild Rejected
  23. 23. Rebuilt and works great
  24. 24. Electronic Giftcards
  25. 25. Epic Volume
  26. 26. Our Challenge
  27. 27. Refactor
  28. 28. SUCCESS!!! SUCCESS!!! SUCCESS!!!
  29. 29. Challenges
  30. 30. Great Engineering Team
  31. 31. Scaling?
  32. 32. no-name projects can go viral But even
  33. 33. So Many Things to Get Wrong
  34. 34. is a constant Failure
  35. 35. But that’s okay
  36. 36. How Do We Build for the Future?
  37. 37. Starting a Project
  38. 38. Greenfield Projects http://commons.wikimedia.org/wiki/File:A_Green_field_in_the_countryside_-_geograph.org.uk_-_193455.jpg
  39. 39. Greenfield Projects Or Greenfeld Projects?
  40. 40. Greenfield Projects http://commons.wikimedia.org/wiki/File:Green_fields_in_Gramado.jpg
  41. 41. Starting Basics
  42. 42. Kelly Johnson Reknowned and prolific aircraft design engineer Invented Seriously Awesome Planes!!!
  43. 43. P-38 Lightning http://en.wikipedia.org/wiki/Lockheed_P-38_Lightning#mediaviewer/File:Lockheed_P-38J_Lightning_-_1.jpg
  44. 44. U-2 http://commons.wikimedia.org/wiki/File:Usaf.u2.750pix.jpg
  45. 45. SR-71 http://commons.wikimedia.org/wiki/File:Lockheed_SR-71_Blackbird.jpg
  46. 46. Kelly Johnson Reknowned and prolific aircraft design engineer AREA 51 Invented Seriously Awesome Planes!!!
  47. 47. Kelly Johnson Reknowned and prolific aircraft design engineer AREA 51 Invented Seriously Awesome Planes!!! Really Good Manager
  48. 48. Keep It Simple, Stupid Kelly Johnson Said:
  49. 49. KISS Kelly Johnson Said: Keep It Simple, Stupid
  50. 50. Keep it Simple Stupid Keep It Simple, Stupid KISS Kelly Johnson Said:
  51. 51. Keep it Simple Stupid If Kelly Johnson kept it simple, why can’t we?
  52. 52. Simple != Simplistic
  53. 53. Needless Complexity is Bad
  54. 54. Bad Complexity Swap two position values in a list
  55. 55. Simplicity is a Virtue Foundations: Simple But not Simplistic Feature Construction: Simple Bug Fixes: Simple
  56. 56. Be careful with databases that make simplistic approaches easy. JSON => SQL
  57. 57. Pro-Tip: Keep management of data relations out of the code!
  58. 58. http://en.wikipedia.org/wiki/File:Map_of_USA_with_state_names_2.svg http://upload.wikimedia.org/wikipedia/commons/thumb/b/b2/ CourtGavel.JPG/600px-CourtGavel.JPG http://commons.wikimedia.org/wiki/File:Benz-velo.jpg
  59. 59. Reasons to consider Non-Relational Databases • Ephemeral Data (caching) • Hierarchical Data • Other data structures that don’t easily map to relations • Seating charts (Eventbrite!) • Organization charts in dysfunctional companies
  60. 60. Pro-Tip Would it work in a spreadsheet? relational databaseYES! No! non-relational database
  61. 61. Simplicity isn’t always the answer
  62. 62. Sometimes you do need complexity
  63. 63. Right at the Beginning
  64. 64. Valid Reason for Complexity: Database Transactions Transactions aren’t complex, but engineers often think they are. Note:
  65. 65. http://commons.wikimedia.org/wiki/ File:WinonaSavingsBankVault.JPG
  66. 66. http://commons.wikimedia.org/wiki/File:Okayama_Red_Cross_Hospital.jpg
  67. 67. Database transactions protect finances and lives Valid Cause of Complexity
  68. 68. transactions If one operation fails, revert revert revert
  69. 69. Don’t code solutions to handle transactions
  70. 70. Use databases with transactions PostgreSQL MySQL
  71. 71. Avoiding Database Transactions is Simplistic You have to build special workarounds to make things reliable
  72. 72. You have to build special workarounds to make things reliable Avoiding Database Transactions is Simplistic
  73. 73. You have to build special workarounds to make things reliable Avoiding Database Transactions is Simplistic
  74. 74. Is a simplistic approach to avoid complexity… A) Bad Design Decision? ! B) Technical Debt?
  75. 75. Answer • Plan to fix later? • Otherwise: Technical Debt Bad Design Decision?
  76. 76. More Thoughts on Complexity
  77. 77. Complex Foundations: Harder to maintain
  78. 78. Complexity2 = Complex Foundations Complex Business +
  79. 79. Complexity2 = Hard to Debug Hard to Enhance +
  80. 80. Complex Code Causes Edge Cases • Are you testing all the BRANCH (if/switch) statements? • How do you test that ‘brilliant’ class heirarchy?
  81. 81. Your business logic is going to be enough complexity Building finance project? Spend your time making it make money? Fighting the project’s design?
  82. 82. Don’t Get Fancy • Build a solid foundation first. • “prototype” projects often aren’t. • Beware virality • No money • Fancy stuff can come later.
  83. 83. Choosing Tools
  84. 84. Choosing Tools is Hard Hard Hard
  85. 85. Do it the Right Way
  86. 86. Management Fiat
  87. 87. Don’t Use Management Fiat with ignorance or bad data
  88. 88. Mistake: Management Fiat Choosing Enterprise Software because it’s expensive
  89. 89. Your engineer’s recommendations ! ! ! ! ! vs Expensive White Papers written by well- paid mostly non-technical analysts that either state the obvious, expensive, or non-sensical Mistake: Management Fiat
  90. 90. HIPAA Health Insurance and Accountability Act Mistake: Management Fiat Wrong advice for
  91. 91. Your engineer’s recommendations vs Anything in a business management magazine. Mistake: Management Fiat
  92. 92. Salesperson’ vs That person you just met who might possibly has a vested financial interest in what they are recommending. Your engineer’s recommendations Mistake: Management Fiat
  93. 93. Your engineer’s recommendations when they constantly tell you how smart they are. ! vs Burning your money in a fire. Same result, but faster Mistake: Management Fiat http://commons.wikimedia.org/wiki/File:Burning-money-and- yuanbao-at-the-cemetery-3249.JPG
  94. 94. Your engineer’s recommendations when they want to continue using a 10+ year old legacy platform vs The Bus Factor Mistake: Management Fiat http://en.wikipedia.org/wiki/File:GMBus.jpg
  95. 95. http://en.wikipedia.org/wiki/Bus_factor Bus Factor! http://commons.wikimedia.org/wiki/File:Arriva_T6_nearside.JPG
  96. 96. Is documented? Does the job? Has traction? Choosing tools
  97. 97. • Ignore corporate marketing hype • Even for corporate open source projects • Unless… Choosing tools It’s software for popular proprietary hardware
  98. 98. Toolkit Ecosystems
  99. 99. Is there an Ecosystem? • Python, Django, Flask, et al • Python package Index • Django Packages • Language + Framework means lots of optimized packages already exist to do much of the work. Tools with traction have extendable components
  100. 100. Case Study:
  101. 101. 5000+ for Django 46000+ for Python
  102. 102. Parallels 78K+ 83K+
  103. 103. Don’t Reinvent the Wheel http://commons.wikimedia.org/wiki/File:London-Eye-2009.JPG
  104. 104. Example: CMS Don’t build your own!
  105. 105. Unless your project is about reinventing wheels http://commons.wikimedia.org/wiki/File:Triple_Rotacaster_commercial_industrial_omni_wheel.jpg
  106. 106. Pro-Tip #1 Toolkit Ecosystems and Engineers If engineers don’t know that tool ecosystems exist, consider them junior. http://commons.wikimedia.org/wiki/File:Fawnpuppy.jpg
  107. 107. Educate them!
  108. 108. Pro-Tip #2 Toolkit Ecosystems and Engineers If engineers refuse to work with an ecosystem, ! don’t waste your time with them.
  109. 109. They purposefully lower the Bus Factor http://commons.wikimedia.org/wiki/File:Keiseibus-twinbus-20071013.jpg
  110. 110. Best Practices
  111. 111. Insist on Best Practices
  112. 112. Cleaner Consistent Stronger Elegant CODE
  113. 113. Easier to Debug
  114. 114. Easier to Add Features
  115. 115. New Staff Ramp Up Faster Faster Faster
  116. 116. How to Best Practice Research and find the common consensus PEP-8 PEP-20 Example: Python
  117. 117. Find References Research and find the common consensus
  118. 118. Research and find the common consensus Find References
  119. 119. Research and find the common consensus • MySQL • PostgreSQL • MongoDB • Redis • CouchDB • Cassandra • BigTable Find References
  120. 120. Management Pro-Tip Part 1 If your engineers consistently refuse to follow defined best practices…
  121. 121. They purposefully lower the Bus Factor http://commons.wikimedia.org/wiki/File:B43OleBillatIWMLondon.jpg
  122. 122. Management Pro-Tip Part 2 Get new ones
  123. 123. Tests
  124. 124. Always have them! Even some coverage is better than no coverage.
  125. 125. Always have a working test harness And a few tests that check really obvious things.
  126. 126. Tests on existing projects without test harnesses seleniumhq.org
  127. 127. Add a test for everybug fix.
  128. 128. Every Programming Tool has Test Harnesses
  129. 129. Management Pro-Tip Engineers who consistently refuse to write tests lower their bus factor to 1. Prepare to keep them foreverhttp://commons.wikimedia.org/wiki/File:Ride_On_5312_at_Glenmont.jpg
  130. 130. Version Control Works oneveryoperatingsystem Tutorials andbookseverywhere Just startwithitoraddit Wehavenoexcusesnottouseit!
  131. 131. Version Control Pro-Tips
  132. 132. Pro-Tip #1 Dropbox is NOT software-project-acceptable version control!
  133. 133. Pro-Tip #2 An FTP server for backing up files isn’t either
  134. 134. Pro-Tip #3 Most good engineers won’t work on a project that doesn’t have version control
  135. 135. Pro-Tip #4 Most good engineers won’t work at companies without version control
  136. 136. Scaling
  137. 137. Scaling usually isn’t a problem
  138. 138. 97% Chance You Ain’t Gonna Need it
  139. 139. A 3% Case “Register here” http://commons.wikimedia.org/wiki/ File:President_Barack_Obama.jpg
  140. 140. Other 3% Cases A major celebrity tweets about your project You are launching the official site for a blockbuster movie You just go viral
  141. 141. What if your foundations are bad?
  142. 142. Bad Foundations are Fixable W. Lloyd MacKenzie, via Flickr @ http://www.flickr.com/photos/saffron_blaze/ With enough engineers, money, and headaches, you can fix anything
  143. 143. Two Quick Scaling Fixes Based on the fact that the early scaling problems are almost always database-related.
  144. 144. It’s almost always about the data
  145. 145. First Fix: Caching The first healthcare.gov patch implemented by their tiger team added caching.
  146. 146. Second Fix: Indexing Almost every datastore supports indexes. Apply them to places with the most data reads.
  147. 147. The Big Scaling Mistake to Avoid
  148. 148. Switching Tools Too Early • Changing to the unusual datastore de-jour • Trying a new programming language • Switching out template languages. • Changing Application Frameworks before attempting to improve the database
  149. 149. Careful of the hype Replacing Tools mobile site Went from Rails to Node.js. Giant Improvements! ! (A coincidence, perhaps?)
  150. 150. Node.js was just part of the story They also modernized servers and added more. Replacing Tools Careful of the hype
  151. 151. Replacing Frameworks But recognize excitement Clearly Linkedin Engineers were excited to work with Node.js Engineer interest in tools is important. It inspires us to do great things.
  152. 152. Management Pro-Tip Uninspired Engineers don’t do the best work and are much more prone to leave.
  153. 153. Don’t Optimize Prematurely
  154. 154. Instead…
  155. 155. Follow Best Practices
  156. 156. By following best practices, you are laying a foundation that makes scaling a project easier.
  157. 157. Long Term Scaling Fixes
  158. 158. Add More Servers
  159. 159. DevOps!
  160. 160. DevOps! Ansible Puppet Chef Docker Salt Stack
  161. 161. DevOps! Catch: Devops != Cheap Ansible Puppet Chef Docker Salt Stack
  162. 162. Pro-Tip Undocumented DevOps lowers the bus factor of operations staff. Prepare to keep them forever http://commons.wikimedia.org/wiki/File:E85bus.jpg
  163. 163. Software as a Service Heroku Engineyard Google App Engine Firebase Various AWS Products
  164. 164. Gets Expensive Software as a Service
  165. 165. Vendor Lock-In Software as a Service
  166. 166. How to replace tools safely
  167. 167. Don’t switch all at once! Version 4
  168. 168. Language Switches Identify bottlenecks in the code Try applying new language there.
  169. 169. Example Switches Aim for giantspeed boosts • Numpy • Scipy • Pandas Move slow big data/scientific calculations to:
  170. 170. More Example Switches Port components to compiled languages such as: • PyPy • Go • C • Java Aim for giantspeed boosts
  171. 171. Maintenance Over Time
  172. 172. Case Study: Every Project • Your project has been around for a while • Problems • Fixing bugs is hard • Adding features is hard
  173. 173. Why is it harder? • Project is no longer the bright and shiny • Adding features adds to complexity • Bugs caused by unforeseen edge cases • Not enough tests make catching developer introduced bugs harder • Mistakes at the beginning are really starting to show Original Engineer(s) was an Idiot
  174. 174. Original Engineer(s) are always idiots
  175. 175. I’ve yet to join a project where I didn’t feel like ranting all the time
  176. 176. Rant Time! • Why use a relational database? • Why not use a non-relational database? • Why this programming language? • Why not use OO or functional programming techniques? • Why use OO or functional programming techniques? • What the heck is this programming pattern anyway?
  177. 177. Caveat: The Constant of the Worst Code Ever
  178. 178. Hindsight is 20/20 • No one predicts with 100% accuracy • Not on software projects • It’s easy with hindsight for us to complain about the decisions made.
  179. 179. Reality Check • Making accurate predictions is hard • Projects grow organically • At least you are getting paid to work on this, right?
  180. 180. Be Understanding • Don’t be a jerk.! • Try to understand why things evolved the way they did. • Forgive your predecessor • They can provide useful information! • Circumstances can and will be reversed
  181. 181. Maintenance Over Time
  182. 182. Why is it harder? • Project is no longer the bright and shiny • Adding features adds to complexity • Bugs caused by unforeseen edge cases • Not enough tests make catching developer introduced bugs harder • Mistakes at the beginning are really starting to show Original Engineer(s) was an Idiot HOW DO WE FIX THIS?
  183. 183. The Basics Maintenance Over Time
  184. 184. The Basics Make engineer setup and system config as fast/easy as possible Vagrant Docker Chef PuppetAnsible Salt Stack
  185. 185. More Basics: Tests No more code changes without tests. New code releases must maintain test coverage or increase it.
  186. 186. More Basics: Document! Document everything as you come across it Markdown, RestructuredText, Google Docs, even Wikis Don’t use Word or Pages!
  187. 187. Refactor Mercilessly Once the basics are in place…
  188. 188. Reward Successful Refactoring
  189. 189. Give them a gnome! Courtesy Daniel Roy Greenfeld
  190. 190. Code Reduction Gnome Courtesy Daniel Roy Greenfeld
  191. 191. New code must follow best practices Basic: Follow Best Practices
  192. 192. Bug-fixed code might be moved to best practices
  193. 193. Embrace Code Reuse
  194. 194. 3+ Identical Lines of Code in the same place?
  195. 195. Stick it in a function or method!
  196. 196. Write a test for that function or method!
  197. 197. Any place this function or method is used… Write a test!
  198. 198. Release Early and Often • Rather than do massiveupdates periodically • Do incrementalreleases constantly • Users respond more favorably • Bug management is easier
  199. 199. Use Feature Flags • Turn new features off and on as needed • Libraries can help you • Use it for A/B testing • In use at Eventbrite and others
  200. 200. http://en.wikipedia.org/wiki/Falcon_9 Summary
  201. 201. pydanny.com Engineer at Co-Author of Architecture Team

×