The document provides tips for building maintainable and scalable projects. It discusses the importance of following best practices like writing tests, using version control, and avoiding premature optimization. It also warns against technical debt and recommends focusing on simplicity over complexity when starting a new project.
Managing and Versioning Machine Learning Models in PythonSimon Frid
Practical machine learning is becoming messy, and while there are lots of algorithms, there is still a lot of infrastructure needed to manage and organize the models and datasets. Estimators and Django-Estimators are two python packages that can help version data sets and models, for deployment and effective workflow.
Version Control in Machine Learning + AI (Stanford)Anand Sampat
Starting with outlining the history of conventional version control before diving into explaining QoDs (Quantitative Oriented Developers) and the unique problems their ML systems pose from an operations perspective (MLOps). With the only status quo solutions being proprietary in-house pipelines (exclusive to Uber, Google, Facebook) and manual tracking/fragile "glue" code for everyone else.
Datmo works to solve this issue by empowering QoDs in two ways: making MLOps manageable and simple (rather than completely abstracted away) as well as reducing the amount of glue code so to ensure more robust end-to-end pipelines.
This goes through a simple example of using Datmo with an Iris classification dataset. Later workshops will expand to show how Datmo can work with other data pipelining tools.
Provenance in Production-Grade Machine LearningAnand Sampat
Over the next few years, every company must develop a strategy to leverage artificial intelligence and machine learning to stay relevant and beat out competitors. This requires hiring talented data scientists as well as DevOps and data engineers who can put these into production. Today, finding that perfect combination of talent can be difficult, but a focus on retraining and productivity tools can increase a small team’s impact on business ROI by over 10x. In this technical talk, we discuss how enterprises can better prepare their employees to deploy artificial intelligence and machine learning into production by using the same techniques used in software to add provenance, reliability, and efficiency to these processes. Specifically, we describe the benefits of adding provenance including reliable deployments and builds, A/B testing, continuous deployment, and automation and show how they can decrease the time to business ROI by over 10x.
From Obvious to Ingenius: Incrementally Scaling Web Apps on PostgreSQLKonstantin Gredeskoul
In this exciting and informative talk, presented at PgConf Sillicon Valley 2015, Konstantin cut through the theory to deliver a clear set of practical solutions for scaling applications atop PostgreSQL, eventually supporting millions of active users, tens of thousands concurrently, and with the application stack that responds to requests with a 100ms average. He will share how his team solved one of the biggest challenges they faced: effectively storing and retrieving over 3B rows of "saves" (a Wanelo equivalent of Instagram's "like" or Pinterest's "pin"), all in PostgreSQL, with highly concurrent random access.
Over the last three years, the team at Wanelo optimized the hell out of their application and database stacks. Using PostgreSQL version 9 as their primary data store, Joyent Public Cloud as a hosting environment, the team re-architected their backend for rapid expansion several times over, as the unrelenting traffic kept climbing up. This ultimately resulted in a highly efficient, horizontally scalable, fault tolerant application infrastructure. Unimpressed? Now try getting there without the OPS or DBA teams, all while deploying seven times per day to production, with an application measuring 99.999% uptime over the last 6 months.
Google AppEngine (GAE/J) - Introduction and Overview from a Java GuyMax Völkel
You know Java, but what is AppEngine? In this session Max will walk you from a brief introduction of Servlets to an overview of the Google AppEngine. We learn about the basics such as the data store – which is quite different from SQL -, application versions, back-ends, scheduler, instances and logging. Services such as mail, url fetch, task queue are also explained. Since we use GAE/J since 2010 in Calpano we also talk about costs and experience in practice.
Managing and Versioning Machine Learning Models in PythonSimon Frid
Practical machine learning is becoming messy, and while there are lots of algorithms, there is still a lot of infrastructure needed to manage and organize the models and datasets. Estimators and Django-Estimators are two python packages that can help version data sets and models, for deployment and effective workflow.
Version Control in Machine Learning + AI (Stanford)Anand Sampat
Starting with outlining the history of conventional version control before diving into explaining QoDs (Quantitative Oriented Developers) and the unique problems their ML systems pose from an operations perspective (MLOps). With the only status quo solutions being proprietary in-house pipelines (exclusive to Uber, Google, Facebook) and manual tracking/fragile "glue" code for everyone else.
Datmo works to solve this issue by empowering QoDs in two ways: making MLOps manageable and simple (rather than completely abstracted away) as well as reducing the amount of glue code so to ensure more robust end-to-end pipelines.
This goes through a simple example of using Datmo with an Iris classification dataset. Later workshops will expand to show how Datmo can work with other data pipelining tools.
Provenance in Production-Grade Machine LearningAnand Sampat
Over the next few years, every company must develop a strategy to leverage artificial intelligence and machine learning to stay relevant and beat out competitors. This requires hiring talented data scientists as well as DevOps and data engineers who can put these into production. Today, finding that perfect combination of talent can be difficult, but a focus on retraining and productivity tools can increase a small team’s impact on business ROI by over 10x. In this technical talk, we discuss how enterprises can better prepare their employees to deploy artificial intelligence and machine learning into production by using the same techniques used in software to add provenance, reliability, and efficiency to these processes. Specifically, we describe the benefits of adding provenance including reliable deployments and builds, A/B testing, continuous deployment, and automation and show how they can decrease the time to business ROI by over 10x.
From Obvious to Ingenius: Incrementally Scaling Web Apps on PostgreSQLKonstantin Gredeskoul
In this exciting and informative talk, presented at PgConf Sillicon Valley 2015, Konstantin cut through the theory to deliver a clear set of practical solutions for scaling applications atop PostgreSQL, eventually supporting millions of active users, tens of thousands concurrently, and with the application stack that responds to requests with a 100ms average. He will share how his team solved one of the biggest challenges they faced: effectively storing and retrieving over 3B rows of "saves" (a Wanelo equivalent of Instagram's "like" or Pinterest's "pin"), all in PostgreSQL, with highly concurrent random access.
Over the last three years, the team at Wanelo optimized the hell out of their application and database stacks. Using PostgreSQL version 9 as their primary data store, Joyent Public Cloud as a hosting environment, the team re-architected their backend for rapid expansion several times over, as the unrelenting traffic kept climbing up. This ultimately resulted in a highly efficient, horizontally scalable, fault tolerant application infrastructure. Unimpressed? Now try getting there without the OPS or DBA teams, all while deploying seven times per day to production, with an application measuring 99.999% uptime over the last 6 months.
Google AppEngine (GAE/J) - Introduction and Overview from a Java GuyMax Völkel
You know Java, but what is AppEngine? In this session Max will walk you from a brief introduction of Servlets to an overview of the Google AppEngine. We learn about the basics such as the data store – which is quite different from SQL -, application versions, back-ends, scheduler, instances and logging. Services such as mail, url fetch, task queue are also explained. Since we use GAE/J since 2010 in Calpano we also talk about costs and experience in practice.
On a Big Data project, validating a promising idea through a prototype is a crucial step. Moving from a prototype to a production-ready application is another one.
Discover how Capgemini has implemented Apache Spark to build People Analytics, a Big Data Analytics platform dedicated to human resources departments.
Learn from our experience, code and best practices to strengthen the reliability, performances and maintainability of your application.
Good Practices for Developing Scientific Software Frameworks: The WRENCH fram...Rafael Ferreira da Silva
What is the best way to write secure and reliable applications?
- Good Enough Practices in Scientific Computing
(https://arxiv.org/abs/1609.00037)
- Best Practices for Scientific Computing (https://doi.org/10.1371/journal.pbio.1001745)
- Scientific Software Best Practices (https://scientific-software-best-practices.readthedocs.io/en/latest/)
- Best Practices for Scientific Software (https://software.ac.uk/blog/2017-11-29-best-practices-scientific-software)
WRENCH enables novel avenues for scientific workflow use, research, development, and education. WRENCH capitalizes on recent and critical advances in the state of the art of distributed platform/application simulation. WRENCH builds on top of the open-source SimGrid simulation framework. SimGrid enables the simulation of large-scale distributed applications in a way that is accurate (via validated simulation models), scalable (low ratio of simulation time to simulated time, ability to run large simulations on a single computer with low compute, memory, and energy footprints), and expressive (ability to simulate arbitrary platform, application, and execution scenarios). WRENCH provides directly usable high-level simulation abstractions using SimGrid as a foundation. More information on https://wrench-project.org
In a nutshell, WRENCH makes it possible to:
- Prototype implementations of Workflow Management System (WMS) components and underlying algorithms;
- Quickly, scalably, and accurately simulate arbitrary workflow and platform scenarios for a simulated WMS implementation; and
- Run extensive experimental campaigns to conclusively compare workflow executions, platform architectures, and WMS algorithms and designs.
Big data made easy with a Spark is the presentation I did for Opensource 101 in Columbia, SC, on April 18th 2019. It is a hands-on tutorial on Apache Spark with Java, walking through 3 different labs.
Maintaining your whole infrastructure using Terraform and reusable modules makes most of our lives easier, but when those less familiar with DevOps want to create or update resources, you usually either have to train and enable them to use Terraform, or handle the request yourself.
However what if you could offload the execution of those changes to a centralised tool and just review both the code and output being submitted for review? Atlantis, Terraform Cloud or env0 can act as a PR-based feedback loop for a hosted Terraform executor to make self-service a little bit easier.
However what if you could offload the execution of those changes to a centralised tool and just review both the code and output being submitted for review? Atlantis, Terraform Cloud or env0 can act as a PR-based feedback loop for a hosted Terraform executor to make self-service a little bit easier.
Presentation given on the 15th July 2021 at the Airflow Summit 2021
Conference website: https://airflowsummit.org/sessions/2021/clearing-airflow-obstructions/
Recording: https://www.crowdcast.io/e/airflowsummit2021/40
Bridging Concepts and Practice in eScience via Simulation-driven EngineeringRafael Ferreira da Silva
The CyberInfrastructure (CI) has been the object of intensive research and development in the last decade, re- sulting in a rich set of abstractions and interoperable software implementations that are used in production today for supporting ongoing and breakthrough scientific discoveries. A key challenge is the development of tools and application execution frameworks that are robust in current and emerging CI configurations, and that can anticipate the needs of upcoming CI applications. This paper presents WRENCH, a framework that enables simulation-driven engineering for evaluating and developing CI application execution frameworks. WRENCH provides a set of high- level simulation abstractions that serve as building blocks for developing custom simulators. These abstractions rely on the scalable and accurate simulation models that are provided by the SimGrid simulation framework. Consequently, WRENCH makes it possible to build, with minimum software development effort, simulators that that can accurately and scalably simulate a wide spectrum of large and complex CI scenarios. These simulators can then be used to evaluate and/or compare alternate platform, system, and algorithm designs, so as to drive the development of CI solutions for current and emerging applications.
A modern architecturereview–usingcodereviewtools-ver-3.5SSW
For any project that is critical to the business, it’s important to do ‘Modern Architecture Reviews’. Being an architect is fun, you get to design the system, do ongoing code reviews, and play the bad ass. It is even more fun when using modern cool tools.
GameDay - Achieving resilience through Chaos EngineeringDiUS
http://dius.com.au/resources/game-day/
Agility has brought us iterative software development, independent feature teams, nimble architectures and distributed, scalable infrastructure. But how do you maintain confidence in these systems in the face of this emergent complexity and fast paced change? The answer is to anticipate and practice failure!
In this session we explore GameDays, a collaborative exercise where teams safely introduce chaos into their systems, in order to make them better.
Doing Analytics Right - Building the Analytics EnvironmentTasktop
Implementing analytics for development processes is challenging. As in discussed in the previous webinars, the right analytics are determined by the goals of the organization, not by the available data. So implementing your analytics solutions will require an efficient analytics and data architecture, including the ability to combine and stage data from heterogeneous sources. An architecture that excludes the ability to gain access to the necessary data will create a barrier to deploying your newly designed analytics program, and will force you back into the “light is brighter here” anti-pattern.
This webinar will describe the technical considerations of implementing the data architecture for your analytics program, and explain how Tasktop can help.
On a Big Data project, validating a promising idea through a prototype is a crucial step. Moving from a prototype to a production-ready application is another one.
Discover how Capgemini has implemented Apache Spark to build People Analytics, a Big Data Analytics platform dedicated to human resources departments.
Learn from our experience, code and best practices to strengthen the reliability, performances and maintainability of your application.
Good Practices for Developing Scientific Software Frameworks: The WRENCH fram...Rafael Ferreira da Silva
What is the best way to write secure and reliable applications?
- Good Enough Practices in Scientific Computing
(https://arxiv.org/abs/1609.00037)
- Best Practices for Scientific Computing (https://doi.org/10.1371/journal.pbio.1001745)
- Scientific Software Best Practices (https://scientific-software-best-practices.readthedocs.io/en/latest/)
- Best Practices for Scientific Software (https://software.ac.uk/blog/2017-11-29-best-practices-scientific-software)
WRENCH enables novel avenues for scientific workflow use, research, development, and education. WRENCH capitalizes on recent and critical advances in the state of the art of distributed platform/application simulation. WRENCH builds on top of the open-source SimGrid simulation framework. SimGrid enables the simulation of large-scale distributed applications in a way that is accurate (via validated simulation models), scalable (low ratio of simulation time to simulated time, ability to run large simulations on a single computer with low compute, memory, and energy footprints), and expressive (ability to simulate arbitrary platform, application, and execution scenarios). WRENCH provides directly usable high-level simulation abstractions using SimGrid as a foundation. More information on https://wrench-project.org
In a nutshell, WRENCH makes it possible to:
- Prototype implementations of Workflow Management System (WMS) components and underlying algorithms;
- Quickly, scalably, and accurately simulate arbitrary workflow and platform scenarios for a simulated WMS implementation; and
- Run extensive experimental campaigns to conclusively compare workflow executions, platform architectures, and WMS algorithms and designs.
Big data made easy with a Spark is the presentation I did for Opensource 101 in Columbia, SC, on April 18th 2019. It is a hands-on tutorial on Apache Spark with Java, walking through 3 different labs.
Maintaining your whole infrastructure using Terraform and reusable modules makes most of our lives easier, but when those less familiar with DevOps want to create or update resources, you usually either have to train and enable them to use Terraform, or handle the request yourself.
However what if you could offload the execution of those changes to a centralised tool and just review both the code and output being submitted for review? Atlantis, Terraform Cloud or env0 can act as a PR-based feedback loop for a hosted Terraform executor to make self-service a little bit easier.
However what if you could offload the execution of those changes to a centralised tool and just review both the code and output being submitted for review? Atlantis, Terraform Cloud or env0 can act as a PR-based feedback loop for a hosted Terraform executor to make self-service a little bit easier.
Presentation given on the 15th July 2021 at the Airflow Summit 2021
Conference website: https://airflowsummit.org/sessions/2021/clearing-airflow-obstructions/
Recording: https://www.crowdcast.io/e/airflowsummit2021/40
Bridging Concepts and Practice in eScience via Simulation-driven EngineeringRafael Ferreira da Silva
The CyberInfrastructure (CI) has been the object of intensive research and development in the last decade, re- sulting in a rich set of abstractions and interoperable software implementations that are used in production today for supporting ongoing and breakthrough scientific discoveries. A key challenge is the development of tools and application execution frameworks that are robust in current and emerging CI configurations, and that can anticipate the needs of upcoming CI applications. This paper presents WRENCH, a framework that enables simulation-driven engineering for evaluating and developing CI application execution frameworks. WRENCH provides a set of high- level simulation abstractions that serve as building blocks for developing custom simulators. These abstractions rely on the scalable and accurate simulation models that are provided by the SimGrid simulation framework. Consequently, WRENCH makes it possible to build, with minimum software development effort, simulators that that can accurately and scalably simulate a wide spectrum of large and complex CI scenarios. These simulators can then be used to evaluate and/or compare alternate platform, system, and algorithm designs, so as to drive the development of CI solutions for current and emerging applications.
A modern architecturereview–usingcodereviewtools-ver-3.5SSW
For any project that is critical to the business, it’s important to do ‘Modern Architecture Reviews’. Being an architect is fun, you get to design the system, do ongoing code reviews, and play the bad ass. It is even more fun when using modern cool tools.
GameDay - Achieving resilience through Chaos EngineeringDiUS
http://dius.com.au/resources/game-day/
Agility has brought us iterative software development, independent feature teams, nimble architectures and distributed, scalable infrastructure. But how do you maintain confidence in these systems in the face of this emergent complexity and fast paced change? The answer is to anticipate and practice failure!
In this session we explore GameDays, a collaborative exercise where teams safely introduce chaos into their systems, in order to make them better.
Doing Analytics Right - Building the Analytics EnvironmentTasktop
Implementing analytics for development processes is challenging. As in discussed in the previous webinars, the right analytics are determined by the goals of the organization, not by the available data. So implementing your analytics solutions will require an efficient analytics and data architecture, including the ability to combine and stage data from heterogeneous sources. An architecture that excludes the ability to gain access to the necessary data will create a barrier to deploying your newly designed analytics program, and will force you back into the “light is brighter here” anti-pattern.
This webinar will describe the technical considerations of implementing the data architecture for your analytics program, and explain how Tasktop can help.
Building and Scaling High Performing Technology Organizations by Jez Humble a...Agile India
High performing organizations don't trade off quality, throughput, and reliability: they work to improve all of these and use their software delivery capability to drive organizational performance. In this talk, Jez presents the results from DevOps Research and Assessment's five-year research program, including how continuous delivery and good architecture produce higher software delivery performance, and how to measure culture and its impact on IT and organizational culture. They explain the importance of knowing how (and what) to measure so you focus on what’s important and communicate progress to peers, leaders, and stakeholders. Great outcomes don’t realize themselves, after all, and having the right metrics gives us the data we need to keep getting better at building, delivering, and operating software systems.
More details:
https://confengine.com/agile-india-2019/proposal/8524/building-and-scaling-high-performing-technology-organizations
Conference link: https://2019.agileindia.org
Building a successful DevOps solution requires a holistic view of your development ecosystem plus solid technology that can support your organization today and in the future. Learn how to start defining your own successful DevOps solution and how to position Helix to be at the center of it all.
Designing for Performance: Database Related Worst PracticesChristian Antognini
Optimal performance is not simply a product one can buy but rather the results of accurate planning and a correct implementation. Given that applications should be designed for performance, it would be useful to cover an approach to doing that in great detail. However, for obvious reasons, it is not a subject that can be covered in a presentation. For this reason, I limit myself to briefly describing the top most common database-related design problems that frequently lead to suboptimal performance.
AMIS 25: DevOps Best Practice for Oracle SOA and BPMMatt Wright
DevOps and Cloud are transforming the software release process, one which spans multiple teams across development and operations (including testing, infrastructure management), into a collaborative process, with all teams working together to deliver solutions into production faster.
This session details how to implement a continuous delivery process for Oracle SOA/BPM projects, both on-premise and in the cloud, which transform the release process into an automated, reliable, high quality delivery pipeline that that deliver projects faster, with less risk and less cost.
It details the processes and best practices that need to be established, how to use tools to automate and govern the build, deployment and configuration of code from our first initial environment through to production.
1. Learn how DevOps and Continuous Delivery can stream-line the delivery of integration / bpm projects into production.
2. Learn how DevOps plus the Cloud service can accelerate the implementation of on-premise Oracle SOA .
3. Learn best practice for implementing DevOps or Continuous Delivery for Oracle SOA projects on-cloud and on-premise.
4. How to use tools to automate and govern the build, deployment and configuration of code from dev through to production
5. How to leverage the Cloud for Dev and Test, and the benefits this provides.
For numerous large enterprises, the alignment of hardware and software processes is critical to managing an Agile environment. Agile Hardware implementations can be put in place by using the same framework as our typical Agile Software Development transformations. Start off with assessing the organization’s current state, then move to planning and preparing by and putting together a transition backlog, start execution with training and coaching, spread the cultural shift with change management and maintain and scale the transformation.
Machine learning applications are typically stitched together from hopes and dreams, shell scripts, cron jobs, home-grown schedulers, snippets of configuration clipped from multiple blog posts, thousands of hard-coded business rules, a.k.a. "our SQL corpus," and a few lines of training and testing code. Organizing all the moving parts into something maintainable and supportive of ongoing development is a challenge most teams have on their TODO list, roadmap, or tech debt pile. Getting ahead of the day-to-day demands and settling into a sane architecture often seems like an unattainable goal. The past several years have seen an explosion of tool-building in the data engineering and analytics area, including in Apache projects spanning the areas of search and information retrieval, job orchestration, file and stream formats, and machine learning libraries. In this talk we will cover our product and development teams' choices of architecture and tools, from data ingestion and storage, through transformations and processing, to presentation of results and publishing to web services, reports, and applications.
Let's say you're a data scientist, and you've been asked to build infrastructure. Here I've distilled some best practices as an introduction for people who are new to DevOps.
What happens when a company either doesn’t fully empower the Security team, or have one at all? Stuff like Goto fail, Equifax, unsandboxed AVs and infinite other buzz, or yet to be buzzed, words describe failures of not adequately protecting customers or services they rely on. Having a solid security team enables a company to set a bar, ensure security exists within the design, insert tooling at various stages of the process and continuously iterate on such results. Working with the folks building the products to give them solutions instead of just problems allows one to scale, earn trust and most importantly be effective and actually ship.
There’s a whole security industry out there with folks wearing every which hat you can think of. They have influence and the ability to find a bug one day and disclose it the next, so companies must adapt both engineering practices and perspectives in order to ‘navigate the waters of reality’ and not just hope one doesn’t take a look at their product. Having processes in place that reduce attack surface, automate testing and set a minimum bar can reduce bugs therefore randomization for devs therefore cost of patching and create a culture where security makes more sense as it demonstratively solves problems.
Nvidia is evolving in this space. Focused on the role of product security, I’ll go through the various components of a security team and how they each interact and complement each other, commodity and niche tooling as well as how relationships across organizations can give one an edge in this area. This talk balances the perspective of security engineers working within a large company with the independent nature of how things work in the industry.
Attendees will walk away with a breadth of knowledge, an inside view of the technical workings, tooling and intricacies of finding and fixing bugs and finding balance within a product-first world.
Automate Application Quality Detection. Use Key Application Quality Metrics (# of SQL, Memory Allocated, CPU & GC Times, ...) captured during Automated Test Executions.
Let these Metrics act as Quality Gates. Leads to better quality software reaching the end of the Pipeline
Similar to From NASA to Startups to Big Commerce (20)
How to Write a Popular Python Library by AccidentDaniel Greenfeld
We gave this talk as the opening keynote speech at PyCon Singapore. The theme of the talk is that most complex projects begin from humble origins. That you should create your own projects, sharing your knowledge and expertise.
10 more things you can do with the Python programming language. Python is used in science, engineer, and forward thinking entrepreneurial projects like Instagram and Pinterest.
My keynote speech from EuroPython, this talk explores what it is like being a developer in a community filled with experts from around the world. The goal of the talk is to provide useful content for beginners and topics of discussion for more advanced developers, while also focusing on Python’s strengths. Video of this talk is at http://www.youtube.com/watch?v=7TImWbnUDeI
In the Python community we are taught from the outset of learning the language that the Zen of Python serves as a guide for how we should construct our codebases and projects. Rather than go into the zen-like meanings of each statement, this talk will explore how individual koans are implemented via detailed displays of sophisticated code examples.
An advanced forms presentation given with Miguel Araujo (marajop) at DjangoCon 2011. The transcript and slides is aimed at getting into Django Core, and Jacob Kaplan-Moss has stated this is his plan.
Admitting my flaws and turning them into virtues! This is a full length talk given it at DjangoCon US 2011, PyCon Australia 2011, and LA Django. The earliest version was a lightning talk given at the 2011 Hollywood hackathon.
Just to be clear, all of these things suck. Please DO NOT use these techniques! It was a joke, and if you attempt any of these things you are making a big mistake.
The slides printed for attendees of the Pycon 2010 Long Pinax Tutorial. James Tauber was the co-author and deserves tons of credit for his work. So does Brian Rosner and the rest of the Pinax team.
13. Early Design Mistakes
• Not understanding or knowing the requirements
• Over or under architecting
• Choosing the wrong tools
• Premature Optimization
• Management fiat made with ignorance or bad data
15. Loan Against the Future
• Get it done fast!
• Grouping data poorly
• Hardcoding
• Upcoming technology will fix it
• Leave testing and documentation for later
Technical Debt
61. Reasons to consider
Non-Relational Databases
• Ephemeral Data (caching)
• Hierarchical Data
• Other data structures that don’t easily map to relations
• Seating charts (Eventbrite!)
• Organization charts in dysfunctional companies
62. Pro-Tip
Would it work in a spreadsheet?
relational databaseYES!
No! non-relational database
95. Salesperson’
vs
That person you just met who might
possibly has a vested financial interest in
what they are recommending.
Your engineer’s recommendations
Mistake: Management Fiat
96. Your engineer’s recommendations when they
constantly tell you how smart they are.
!
vs
Burning your
money in a fire.
Same result, but faster
Mistake: Management Fiat
http://commons.wikimedia.org/wiki/File:Burning-money-and-
yuanbao-at-the-cemetery-3249.JPG
97. Your engineer’s recommendations when they want to
continue using a 10+ year old legacy platform
vs
The Bus Factor
Mistake: Management Fiat
http://en.wikipedia.org/wiki/File:GMBus.jpg
101. • Ignore corporate marketing hype
• Even for corporate open source projects
• Unless…
Choosing tools
It’s software for popular proprietary hardware
103. Is there an Ecosystem?
• Python, Django, Flask, et al
• Python package Index
• Django Packages
• Language + Framework means lots of optimized
packages already exist to do much of the work.
Tools with traction have extendable components
109. Unless your project is
about reinventing wheels
http://commons.wikimedia.org/wiki/File:Triple_Rotacaster_commercial_industrial_omni_wheel.jpg
110. Pro-Tip #1
Toolkit Ecosystems and Engineers
If engineers don’t know that tool ecosystems exist,
consider them junior.
http://commons.wikimedia.org/wiki/File:Fawnpuppy.jpg
133. Management Pro-Tip
Engineers who consistently
refuse to write tests lower
their bus factor to 1.
Prepare to
keep them foreverhttp://commons.wikimedia.org/wiki/File:Ride_On_5312_at_Glenmont.jpg
148. Bad
Foundations
are Fixable
W. Lloyd MacKenzie, via Flickr @ http://www.flickr.com/photos/saffron_blaze/
With enough
engineers, money, and
headaches, you can fix
anything
155. Switching Tools Too Early
• Changing to the unusual datastore de-jour
• Trying a new programming language
• Switching out template languages.
• Changing Application Frameworks
before attempting to improve the database
156. Careful of the hype
Replacing Tools
mobile site
Went from Rails to Node.js.
Giant Improvements!
!
(A coincidence, perhaps?)
157. Node.js was just part of the story
They also modernized servers
and added more.
Replacing Tools
Careful of the hype
158. Replacing Frameworks
But recognize excitement
Clearly Linkedin Engineers were
excited to work with Node.js
Engineer interest in tools is important.
It inspires us to do great things.
179. Case Study: Every Project
• Your project has been around for a while
• Problems
• Fixing bugs is hard
• Adding features is hard
180. Why is it harder?
• Project is no longer the bright and shiny
• Adding features adds to complexity
• Bugs caused by unforeseen edge cases
• Not enough tests make catching developer introduced
bugs harder
• Mistakes at the beginning are really starting to show
Original Engineer(s) was an Idiot
182. I’ve yet to join a project
where I didn’t feel like
ranting all the time
183. Rant Time!
• Why use a relational database?
• Why not use a non-relational database?
• Why this programming language?
• Why not use OO or functional programming techniques?
• Why use OO or functional programming techniques?
• What the heck is this programming pattern anyway?
185. Hindsight is 20/20
• No one predicts with 100% accuracy
• Not on software projects
• It’s easy with hindsight for us to complain about the
decisions made.
186. Reality Check
• Making accurate predictions is hard
• Projects grow organically
• At least you are getting paid to work on this, right?
187. Be Understanding
• Don’t be a jerk.!
• Try to understand why things evolved the way they did.
• Forgive your predecessor
• They can provide useful information!
• Circumstances can and will be reversed
189. Why is it harder?
• Project is no longer the bright and shiny
• Adding features adds to complexity
• Bugs caused by unforeseen edge cases
• Not enough tests make catching developer introduced
bugs harder
• Mistakes at the beginning are really starting to show
Original Engineer(s) was an Idiot
HOW
DO WE FIX THIS?
204. Any place this function
or method is used…
Write a test!
205. Release Early and Often
• Rather than do massiveupdates periodically
• Do incrementalreleases constantly
• Users respond more favorably
• Bug management is easier
206. Use Feature Flags
• Turn new features off and on as needed
• Libraries can help you
• Use it for A/B testing
• In use at Eventbrite and others