The Hard and Soft SkillsRequired to Avoid Outages              @pascallouis
Code   Bless   Ship   Maintain   Profit!   git rm
Tactics       • Entity bound ids (e.g. Id<T>)       • Textual ids MWDN-YP89-OLVL-USER       • Testable configurations      ...
TDD       • Not controversial (anymore)       • Living code documentation       • Enables collaboration       • Technique ...
Gold Tests       • Tests which can be changed by a (small)         subset of engineering       • Enforced via policy or te...
Expressive Tests       • “Change your language and you change         your thoughts” — Karl Albrecht       • Can be implem...
...             Given feed PaymentEventFeedListener receives:       """               {                   "payment_id": "E...
Automated            Manual  Quality                               or              or                                     ...
Code Analysis       • In theory: static vs dynamic       • In practice: pre vs post-productionCode       Bless     Ship   ...
Pre Analysis       • Type Checking       • Testing, CI       • Linters       • Forbidden Call AnalysisCode       Bless    ...
Post Analysis       • Logging       • Metrics       • Invariant CheckingCode       Bless     Ship     Maintain   Profit!   ...
Alerting & Reporting                   Sign               Res                  pon                          al            ...
Metrics vs Checks    OK                   ?WARNING     1     0  200ms   0ms Code     Bless        Ship   Maintain   Profit!...
Code   Bless   Ship   Maintain   Profit!   git rm
Post-Mortem       • When Anytime there are issues!       • Why Learn and avoid mistakes of the past       • How BlamelessC...
Post-Mortem       • Go through the timeline       • The Good, The Bad and the Ugly       • Action ItemsCode        Bless  ...
Root Cause AnalysisCode   Bless   Ship   Maintain   Profit!   git rm
Code   Bless   Ship   Maintain   Profit!   git rm
Proportional Investing       • When you lose N hours to maintenance, you         spend an equivalent N hours on improving ...
Wrap Up• Safety unleashes creativity• Many touch points, many ways to do it —  find what’s right for you• It’s not all abou...
Developing an Immune System — The Hard and Soft Skills required to avoid Outages
Developing an Immune System — The Hard and Soft Skills required to avoid Outages
Developing an Immune System — The Hard and Soft Skills required to avoid Outages
Developing an Immune System — The Hard and Soft Skills required to avoid Outages
Upcoming SlideShare
Loading in …5
×

Developing an Immune System — The Hard and Soft Skills required to avoid Outages

689 views

Published on

In just three years, Square has achieved ubiquitous recognition for mobile card processing grossing over $10B a year in credit card transactions. At the heart of Square's technology are many financial systems which must operate safely, correctly, and sustain rapid growth.

During this tech talk, Pascal will describe the concept of an immune system, go over best practices, share lessons leaned and provide a detailed layering of best practices. This talk will cover non-controversial topics such ad TDD, but from new angles. We'll also cover emerging practices like continuous deployment, and softer areas such as engineering management practices geared towards safety. You'll come out of this session with a fresh perspective on how to build software.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
689
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Developing an Immune System — The Hard and Soft Skills required to avoid Outages

  1. 1. The Hard and Soft SkillsRequired to Avoid Outages @pascallouis
  2. 2. Code Bless Ship Maintain Profit! git rm
  3. 3. Tactics • Entity bound ids (e.g. Id<T>) • Textual ids MWDN-YP89-OLVL-USER • Testable configurations • etc.Code Bless Ship Maintain Profit! git rm
  4. 4. TDD • Not controversial (anymore) • Living code documentation • Enables collaboration • Technique to encode invariantsCode Bless Ship Maintain Profit! git rm
  5. 5. Gold Tests • Tests which can be changed by a (small) subset of engineering • Enforced via policy or technologyCode Bless Ship Maintain Profit! git rm
  6. 6. Expressive Tests • “Change your language and you change your thoughts” — Karl Albrecht • Can be implementation agnosticCode Bless Ship Maintain Profit! git rm
  7. 7. ... Given feed PaymentEventFeedListener receives: """ { "payment_id": "EPT-300", "isTivoReplay": false, "merchant": { "token": "m-1" }, ... } """ Then expect table balance_changing_events order by id: | event_type | status | process_attempts | | HOLD | UNPROCESSED | 1 | | CAPTURE | UNPROCESSED | 0 | When then the time is 2012-01-06 17:10:00 And balance changing event queue processes items Then expect table balance_changing_events order by id: | event_type | status | process_attempts | | HOLD | UNPROCESSED | 2 | | CAPTURE | PROCESSED | 1 |Code Bless Ship Maintain Profit! git rm
  8. 8. Automated Manual Quality or or TimeCode Bless Ship Maintain Profit! git rm
  9. 9. Code Analysis • In theory: static vs dynamic • In practice: pre vs post-productionCode Bless Ship Maintain Profit! git rm
  10. 10. Pre Analysis • Type Checking • Testing, CI • Linters • Forbidden Call AnalysisCode Bless Ship Maintain Profit! git rm
  11. 11. Post Analysis • Logging • Metrics • Invariant CheckingCode Bless Ship Maintain Profit! git rm
  12. 12. Alerting & Reporting Sign Res pon al Precise Imprecise se Immediate Alert Oups! Deferred Report ReportCode Bless Ship Maintain Profit! git rm
  13. 13. Metrics vs Checks OK ?WARNING 1 0 200ms 0ms Code Bless Ship Maintain Profit! git rm
  14. 14. Code Bless Ship Maintain Profit! git rm
  15. 15. Post-Mortem • When Anytime there are issues! • Why Learn and avoid mistakes of the past • How BlamelessCode Bless Ship Maintain Profit! git rm
  16. 16. Post-Mortem • Go through the timeline • The Good, The Bad and the Ugly • Action ItemsCode Bless Ship Maintain Profit! git rm
  17. 17. Root Cause AnalysisCode Bless Ship Maintain Profit! git rm
  18. 18. Code Bless Ship Maintain Profit! git rm
  19. 19. Proportional Investing • When you lose N hours to maintenance, you spend an equivalent N hours on improving things.Code Bless Ship Maintain Profit! git rm
  20. 20. Wrap Up• Safety unleashes creativity• Many touch points, many ways to do it — find what’s right for you• It’s not all about technology, most is about the culture and the people

×