From NASA to Startups to Big Commerce
Jul. 15, 2014•0 likes•93,238 views
Download to read offline
Report
Daniel GreenfeldFollow
Recommended
Managing and Versioning Machine Learning Models in PythonSimon Frid
Scaling Analysis ResponsiblyWork-Bench
Using dataset versioning in data scienceVenkata Pingali
Version Control in Machine Learning + AI (Stanford)Anand Sampat
Provenance in Production-Grade Machine LearningAnand Sampat
From Obvious to Ingenius: Incrementally Scaling Web Apps on PostgreSQLKonstantin Gredeskoul
More Related Content
What's hot
Software Analytics - Achievements and ChallengesTao Xie
H2O 3 REST API OverviewRaymond Peck
Capital onehadoopintroDoug Chang
Deploying ML models to production (frequently and safely) - PYCON 2018David Tan
Capgemini - Project industrialization with apache sparkJean-Baptiste Martin
Mark Hughes Annual Seminar Presentation on Open Source Tracy Kent
Similar to From NASA to Startups to Big Commerce
Software Carpentry and the Hydrological Sciences @ AGU 2013Aron Ahmadia
The Lost Tales of Platform Design (February 2017)Julien SIMON
A modern architecturereview–usingcodereviewtools-ver-3.5SSW
GameDay - Achieving resilience through Chaos EngineeringDiUS
Doing Analytics Right - Building the Analytics EnvironmentTasktop
Agile+Lean, Alum Meet Apr23 2016 - Short VersionHemant Elhence
More from Daniel Greenfeld
How to Write a Popular Python Library by AccidentDaniel Greenfeld
10 more-things-you-can-do-with-pythonDaniel Greenfeld
Thinking hard about_pythonDaniel Greenfeld
Intro to Data VisualizationsDaniel Greenfeld
An Extreme Talk about the Zen of PythonDaniel Greenfeld
PyCon Philippines 2012 KeynoteDaniel Greenfeld
From NASA to Startups to Big Commerce
- 1. FROM NASA TO STARTUPS TO BIG COMMERCE Building Maintainable, Scalable Projects DANIEL GREENFELD
- 3. FAILURE.
- 5. I ❤aerospace (you’ve been warned)
- 6. BIGPLANS
- 8. The Reality
- 13. Early Design Mistakes • Not understanding or knowing the requirements • Over or under architecting • Choosing the wrong tools • Premature Optimization • Management fiat made with ignorance or bad data
- 14. Technical Debt
- 15. Loan Against the Future • Get it done fast! • Grouping data poorly • Hardcoding • Upcoming technology will fix it • Leave testing and documentation for later Technical Debt
- 18. 2006
- 21. Bad Practices @ PyCon 2011 https://www.flickr.com/photos/pydanny/5670716870/
- 22. Had mistakes at start
- 23. We insisted on rebuild Rejected
- 24. Rebuilt and works great
- 26. Epic Volume
- 27. Our Challenge
- 28. Refactor
- 30. Challenges
- 32. Scaling?
- 35. So Many Things to Get Wrong
- 37. But that’s okay
- 38. How Do We Build for the Future?
- 41. Greenfield Projects Or Greenfeld Projects?
- 43. Starting Basics
- 44. Kelly Johnson Reknowned and prolific aircraft design engineer Invented Seriously Awesome Planes!!!
- 48. Kelly Johnson Reknowned and prolific aircraft design engineer AREA 51 Invented Seriously Awesome Planes!!!
- 49. Kelly Johnson Reknowned and prolific aircraft design engineer AREA 51 Invented Seriously Awesome Planes!!! Really Good Manager
- 50. Keep It Simple, Stupid Kelly Johnson Said:
- 51. KISS Kelly Johnson Said: Keep It Simple, Stupid
- 52. Keep it Simple Stupid Keep It Simple, Stupid KISS Kelly Johnson Said:
- 53. Keep it Simple Stupid If Kelly Johnson kept it simple, why can’t we?
- 56. Bad Complexity Swap two position values in a list
- 57. Simplicity is a Virtue Foundations: Simple But not Simplistic Feature Construction: Simple Bug Fixes: Simple
- 58. Be careful with databases that make simplistic approaches easy. JSON => SQL
- 59. Pro-Tip: Keep management of data relations out of the code!
- 61. Reasons to consider Non-Relational Databases • Ephemeral Data (caching) • Hierarchical Data • Other data structures that don’t easily map to relations • Seating charts (Eventbrite!) • Organization charts in dysfunctional companies
- 62. Pro-Tip Would it work in a spreadsheet? relational databaseYES! No! non-relational database
- 63. Simplicity isn’t always the answer
- 64. Sometimes you do need complexity
- 65. Right at the Beginning
- 66. Valid Reason for Complexity: Database Transactions Transactions aren’t complex, but engineers often think they are. Note:
- 69. Database transactions protect finances and lives Valid Cause of Complexity
- 70. transactions If one operation fails, revert revert revert
- 71. Don’t code solutions to handle transactions
- 73. Avoiding Database Transactions is Simplistic You have to build special workarounds to make things reliable
- 74. You have to build special workarounds to make things reliable Avoiding Database Transactions is Simplistic
- 75. You have to build special workarounds to make things reliable Avoiding Database Transactions is Simplistic
- 76. Is a simplistic approach to avoid complexity… A) Bad Design Decision? ! B) Technical Debt?
- 77. Answer • Plan to fix later? • Otherwise: Technical Debt Bad Design Decision?
- 79. Complex Foundations: Harder to maintain
- 81. Complexity2 = Hard to Debug Hard to Enhance +
- 82. Complex Code Causes Edge Cases • Are you testing all the BRANCH (if/switch) statements? • How do you test that ‘brilliant’ class heirarchy?
- 83. Your business logic is going to be enough complexity Building finance project? Spend your time making it make money? Fighting the project’s design?
- 84. Don’t Get Fancy • Build a solid foundation first. • “prototype” projects often aren’t. • Beware virality • No money • Fancy stuff can come later.
- 85. Choosing Tools
- 88. Do it the Right Way
- 89. Management Fiat
- 90. Don’t Use Management Fiat with ignorance or bad data
- 91. Mistake: Management Fiat Choosing Enterprise Software because it’s expensive
- 92. Your engineer’s recommendations ! ! ! ! ! vs Expensive White Papers written by well- paid mostly non-technical analysts that either state the obvious, expensive, or non-sensical Mistake: Management Fiat
- 93. HIPAA Health Insurance and Accountability Act Mistake: Management Fiat Wrong advice for
- 94. Your engineer’s recommendations vs Anything in a business management magazine. Mistake: Management Fiat
- 95. Salesperson’ vs That person you just met who might possibly has a vested financial interest in what they are recommending. Your engineer’s recommendations Mistake: Management Fiat
- 96. Your engineer’s recommendations when they constantly tell you how smart they are. ! vs Burning your money in a fire. Same result, but faster Mistake: Management Fiat http://commons.wikimedia.org/wiki/File:Burning-money-and- yuanbao-at-the-cemetery-3249.JPG
- 97. Your engineer’s recommendations when they want to continue using a 10+ year old legacy platform vs The Bus Factor Mistake: Management Fiat http://en.wikipedia.org/wiki/File:GMBus.jpg
- 100. Is documented? Does the job? Has traction? Choosing tools
- 101. • Ignore corporate marketing hype • Even for corporate open source projects • Unless… Choosing tools It’s software for popular proprietary hardware
- 102. Toolkit Ecosystems
- 103. Is there an Ecosystem? • Python, Django, Flask, et al • Python package Index • Django Packages • Language + Framework means lots of optimized packages already exist to do much of the work. Tools with traction have extendable components
- 104. Case Study:
- 105. 5000+ for Django 46000+ for Python
- 106. Parallels 78K+ 83K+
- 107. Don’t Reinvent the Wheel http://commons.wikimedia.org/wiki/File:London-Eye-2009.JPG
- 108. Example: CMS Don’t build your own!
- 109. Unless your project is about reinventing wheels http://commons.wikimedia.org/wiki/File:Triple_Rotacaster_commercial_industrial_omni_wheel.jpg
- 110. Pro-Tip #1 Toolkit Ecosystems and Engineers If engineers don’t know that tool ecosystems exist, consider them junior. http://commons.wikimedia.org/wiki/File:Fawnpuppy.jpg
- 111. Educate them!
- 112. Pro-Tip #2 Toolkit Ecosystems and Engineers If engineers refuse to work with an ecosystem, ! don’t waste your time with them.
- 114. Best Practices
- 117. Easier to Debug
- 119. New Staff Ramp Up Faster Faster Faster
- 120. How to Best Practice Research and find the common consensus PEP-8 PEP-20 Example: Python
- 121. Find References Research and find the common consensus
- 122. Research and find the common consensus Find References
- 123. Research and find the common consensus • MySQL • PostgreSQL • MongoDB • Redis • CouchDB • Cassandra • BigTable Find References
- 124. Management Pro-Tip Part 1 If your engineers consistently refuse to follow defined best practices…
- 126. Management Pro-Tip Part 2 Get new ones
- 127. Tests
- 128. Always have them! Even some coverage is better than no coverage.
- 129. Always have a working test harness And a few tests that check really obvious things.
- 130. Tests on existing projects without test harnesses seleniumhq.org
- 131. Add a test for everybug fix.
- 132. Every Programming Tool has Test Harnesses
- 133. Management Pro-Tip Engineers who consistently refuse to write tests lower their bus factor to 1. Prepare to keep them foreverhttp://commons.wikimedia.org/wiki/File:Ride_On_5312_at_Glenmont.jpg
- 134. Version Control Works oneveryoperatingsystem Tutorials andbookseverywhere Just startwithitoraddit Wehavenoexcusesnottouseit!
- 136. Pro-Tip #1 Dropbox is NOT software-project-acceptable version control!
- 137. Pro-Tip #2 An FTP server for backing up files isn’t either
- 138. Pro-Tip #3 Most good engineers won’t work on a project that doesn’t have version control
- 139. Pro-Tip #4 Most good engineers won’t work at companies without version control
- 140. Scaling
- 141. Scaling usually isn’t a problem
- 142. 97% Chance You Ain’t Gonna Need it
- 143. A 3% Case “Register here” http://commons.wikimedia.org/wiki/ File:President_Barack_Obama.jpg
- 145. Other 3% Cases A major celebrity tweets about your project You are launching the official site for a blockbuster movie You just go viral
- 147. What if your foundations are bad?
- 148. Bad Foundations are Fixable W. Lloyd MacKenzie, via Flickr @ http://www.flickr.com/photos/saffron_blaze/ With enough engineers, money, and headaches, you can fix anything
- 150. Two Quick Scaling Fixes Based on the fact that the early scaling problems are almost always database-related.
- 151. It’s almost always about the data
- 152. First Fix: Caching The first healthcare.gov patch implemented by their tiger team added caching.
- 153. Second Fix: Indexing Almost every datastore supports indexes. Apply them to places with the most data reads.
- 154. The Big Scaling Mistake to Avoid
- 155. Switching Tools Too Early • Changing to the unusual datastore de-jour • Trying a new programming language • Switching out template languages. • Changing Application Frameworks before attempting to improve the database
- 156. Careful of the hype Replacing Tools mobile site Went from Rails to Node.js. Giant Improvements! ! (A coincidence, perhaps?)
- 157. Node.js was just part of the story They also modernized servers and added more. Replacing Tools Careful of the hype
- 158. Replacing Frameworks But recognize excitement Clearly Linkedin Engineers were excited to work with Node.js Engineer interest in tools is important. It inspires us to do great things.
- 159. Management Pro-Tip Uninspired Engineers don’t do the best work and are much more prone to leave.
- 161. Instead…
- 163. By following best practices, you are laying a foundation that makes scaling a project easier.
- 165. Add More Servers
- 166. DevOps!
- 168. DevOps! Catch: Devops != Cheap Ansible Puppet Chef Docker Salt Stack
- 169. Pro-Tip Undocumented DevOps lowers the bus factor of operations staff. Prepare to keep them forever http://commons.wikimedia.org/wiki/File:E85bus.jpg
- 170. Software as a Service Heroku Engineyard Google App Engine Firebase Various AWS Products
- 171. Gets Expensive Software as a Service
- 172. Vendor Lock-In Software as a Service
- 173. How to replace tools safely
- 174. Don’t switch all at once! Version 4
- 175. Language Switches Identify bottlenecks in the code Try applying new language there.
- 176. Example Switches Aim for giantspeed boosts • Numpy • Scipy • Pandas Move slow big data/scientific calculations to:
- 177. More Example Switches Port components to compiled languages such as: • PyPy • Go • C • Java Aim for giantspeed boosts
- 179. Case Study: Every Project • Your project has been around for a while • Problems • Fixing bugs is hard • Adding features is hard
- 180. Why is it harder? • Project is no longer the bright and shiny • Adding features adds to complexity • Bugs caused by unforeseen edge cases • Not enough tests make catching developer introduced bugs harder • Mistakes at the beginning are really starting to show Original Engineer(s) was an Idiot
- 181. Original Engineer(s) are always idiots
- 182. I’ve yet to join a project where I didn’t feel like ranting all the time
- 183. Rant Time! • Why use a relational database? • Why not use a non-relational database? • Why this programming language? • Why not use OO or functional programming techniques? • Why use OO or functional programming techniques? • What the heck is this programming pattern anyway?
- 184. Caveat: The Constant of the Worst Code Ever
- 185. Hindsight is 20/20 • No one predicts with 100% accuracy • Not on software projects • It’s easy with hindsight for us to complain about the decisions made.
- 186. Reality Check • Making accurate predictions is hard • Projects grow organically • At least you are getting paid to work on this, right?
- 187. Be Understanding • Don’t be a jerk.! • Try to understand why things evolved the way they did. • Forgive your predecessor • They can provide useful information! • Circumstances can and will be reversed
- 189. Why is it harder? • Project is no longer the bright and shiny • Adding features adds to complexity • Bugs caused by unforeseen edge cases • Not enough tests make catching developer introduced bugs harder • Mistakes at the beginning are really starting to show Original Engineer(s) was an Idiot HOW DO WE FIX THIS?
- 191. The Basics Make engineer setup and system config as fast/easy as possible Vagrant Docker Chef PuppetAnsible Salt Stack
- 192. More Basics: Tests No more code changes without tests. New code releases must maintain test coverage or increase it.
- 193. More Basics: Document! Document everything as you come across it Markdown, RestructuredText, Google Docs, even Wikis Don’t use Word or Pages!
- 194. Refactor Mercilessly Once the basics are in place…
- 196. Give them a gnome! Courtesy Daniel Roy Greenfeld
- 197. Code Reduction Gnome Courtesy Daniel Roy Greenfeld
- 198. New code must follow best practices Basic: Follow Best Practices
- 199. Bug-fixed code might be moved to best practices
- 200. Embrace Code Reuse
- 201. 3+ Identical Lines of Code in the same place?
- 202. Stick it in a function or method!
- 203. Write a test for that function or method!
- 204. Any place this function or method is used… Write a test!
- 205. Release Early and Often • Rather than do massiveupdates periodically • Do incrementalreleases constantly • Users respond more favorably • Bug management is easier
- 206. Use Feature Flags • Turn new features off and on as needed • Libraries can help you • Use it for A/B testing • In use at Eventbrite and others