Software engineering
essentials
A primer
Bio
● 16+ years IT stuff
● 1981 - 2011 in Germany
● since 21.07.2011 in Brazil, married, 2 sons
Currently
● Head of Innovation, Dafiti
● IT Consultant, independent
Contact
● Twitter (follow me for mostly geek stuff): https://twitter.com/georg.buske
echo -n $(whoami); Georg.Buske
Today’s topics (not agenda)
● Process and team organization
○ Agile - what's all about?
○ Small projects VS. large scale projects
○ Real world examples of important IT guidelines in an enterprise
● Software Design & Architecture
○ Clean Code - an evergreen in Software Development
○ abbreviations you need to know: DDD, TDD, KISS, DRY, YAGNI, SOLID
○ A deeper look into the meaning of…
Disclaimer(s)
● Slides in english, talk in portuguese
● This is in no means a deep dive - a primer
○ We will cover a lot of stuff
○ Each topic could easily fill its own talk (or workshop)
○ There is no shortcut
● Slides will be available afterwards
Process and team organization
Fact: There is no real difference
Everything of the talk can be applied to small and big [software] projects [and
products]
Wait, there is “really small”:
● < 3 developers on a private project or something you own
● then it is OK to ignore a lot of topics
● Use at least a VCS, like Git (even you are working alone, source control
makes your life easier)
Small projects VS. large scale projects
Try to split large projects into a lot of small ones!
E.g. Amazon has 2 Pizza teams
Average team size in other tech companies is between 3 - 10 (which is already a
lot!)
Communication paths by team size
Source: https://www.solutionsiq.com/learning/blog-post/lines-of-communication/
Agile - what's all about?
Agile Manifesto
Different ways of being agile: Scrum
Scrum, quite old: 1993
“Real scrum” has more than 30 rules
Because of this technically a lot of companies are doing “ScrumBut”
But this is fine, as long as we are efficient
Some important aspects of Scrum:
Prioritized Backlog
Timeboxed Sprints (Work organized and delivered in Sprints, e.g. always 2 weeks,
no changes) -> deliver value
Scrum
Roles:
● scrum master
● product owner
● participants (qa, devs)
Meetings:
● Daily (same place, same time, everyday)
● Retrospective (what was good, what was bad in the last Sprint)
● Planning (what we can accomplish next Sprint)
Different ways of being agile: Kanban
● part of lean movement (popularized by toyota, 1940s/1950s)
● Less rules than Scrum, important aspects:
○ Visualize (KanbanBoard)
○ limit WIP
○ optimize value streams
○ Implement Feedback loops
Different ways of being agile: XP
● extreme programming framework
● by Kent Beck (also creator of TDD)
● First used in C3 (Chrysler Comprehensive Compensation project) - Kent
Beck, Martin Fowler
● “XP was one of the first agile methods, indeed XP was the dominant agile
method in the late 90s and early 00s before Scrum became dominant”
● Contains:
○ Continuous Integration
○ Refactoring
Real world examples of important IT
guidelines in an enterprise
Why is it important?
● Consistency
● Best practices (we know it works)
● blueprint / howto for everyone (new and old), everybody knows how to do
something
● Standard (from Wikipedia): established norm, specification, system of market
dominance
● Guideline (from Google search): a general rule, principle or piece of advice
Dafiti’s IT Docs...
● General development workflow
● Architecture guidelines
● Ticket guidelines
● Coding standards
○ Style Guide templates
● Logging guidelines
● Commit message standard
● Pull request guidelines
Dafiti’s IT Docs...
● Post-mortem template
● DevOps best practices and failure mode
● Change requests
● POCs
● RFCs
● Service guidelines
● Component integration guidelines
● Component testing best practices
Examples
We will opensource our Docs, follow @Dafiti_Tech or me (@georgbuske) on
Twitter if you are interested ;-)
Feedback and Deliver Value
Software Design & Architecture
Design VS. Architecture
Definition
If the Design is the interior, the architecture is the building
Therefore:
Design: The API (how to open the door) how looks the dining room)
Architecture: The Big Picture, high level (this is how the house [or city] looks like)
So in short, Software architecture is more about the design of the entire system,
while software design emphasizes on module / component / class level. (Source:
https://stackoverflow.com/questions/704855/software-design-vs-software-architecture)
Avoid Ivory Tower Architecture
Should there be a dedicated team of Software architectures?
No!
Better:
● committee of senior technologists
● clear set of allowed technologies (tools like Technology Radar:
https://www.thoughtworks.com/radar )
Clean Code - First things first
● Clean Code Book, by Robert “Uncle Bob” Martin
● “Programs must be written for people to read, and only incidentally for
machines to execute.”
○ ― Harold Abelson, Structure and Interpretation of Computer Programs
● “No surprises”
○ ― Ward Cunningham
● “Clean code is a code that is written by someone who cares”
○ ― MIchael Feathers
Clean Code characteristics
● is language independent
● method length
● I should be able to read and understand
● Interfaces
● Error handling
● Boy Scout Rule
○ Always leave the campground cleaner than you found it.
● Unit and acceptance tests
Naming is hard
There are only two hard things in
Computer Science: cache invalidation
and naming things.
-- Phil Karlton
An extension
My personal extension
Naming
● this applies to everything: variables, classes, functions, interfaces - even
repository names
● Rule: Use Intention-Revealing Names
● Rule: Avoid Disinformation
○ E.g. IP as variable name for Identity Provider
● Rule: Make meaningful Distinctions
○ ProductInfo and ProductData
● Classes: Nouns
Naming Examples
SOLID
● subset of clean code
● 5 principles
○ Single Responsibility Principle
○ Open Close Principle
○ Liskov Substitution Principle
○ Interface Segregation Principle
○ Dependency Inversion Principle
Single Responsibility Principle
Cohesion, a class should have only a single responsibility
Should be changed for only one reason
minimal and does one thing good (cf. Unix philosophy) and least possible
dependencies
Open Closed Principle
software entities should be open for extension, but closed for modification
Liskov Substitution Principle
Objects in a program should be replaceable with instances of their subtypes
without altering the correctness of that program
Interface Segregation Principle
Many client specific interfaces are better than one big one
Dependency Inversion Principle
One should depend upon abstractions, not concretions
A.High-level modules should not depend on low-level modules. Both should
depend on abstractions.
B.Abstractions should not depend on details. Details should depend on
abstractions.
Example Patterns: Plugin, ServiceLocator, Dependency Injection -> Inversion of
control
KISS = Keep it simple, stupid
● no overly complex solutions
● straightforward is often easier, cleaner and better to understand
● if you do TDD, this helps
● code duplication is bad [copy paste]
● now, this can conflict with microservices (more on microservices later)
DRY = Don’t repeat yourself
YAGNI = You ain’t gonna need it
very close related to KISS
For example: unnecessary complexity
Except you are writing library code probably don't need to add 4 different types of
authentication methods (Strategies, Adapters) - if there is the requirement to
change, it is often long-term
DDD = Domain Driven Design
● term coined by Eric Evans
● focus on problem domain
● Combines lots of best
practices and patterns
○ Refactoring
○ Continuous Integration
Context mapping & bounded contexts
● Core domain, generic subdomain, supporting domain
○ Best forces on most important part of the system (your product’s USP), core domains
○ Generic subdomains could be bought off-the-shelf
Ubiquitous language
● Everyone in team has the same vocabulary
Tactical Patterns
● Entities
○ Has identity
● Aggregate Roots
○ Are the entry points
● Value Objects
○ Immutable
● Repositories
○ Data Layer
TDD = Test Driven Development
Test Driven Development
● Red - green - refactor
● test first development (unit testing)
● write failing test, implement, refactor, iterate
● you can certainly test after, but advantages for test first are focus on features
● evolving and iterative design, to achieve KISS, DRY and YAGNI
Bonus: SOA = Service Oriented
Architecture
SOA is a kind of 20 year old concept
● Boundaries are explicit
● Services are autonomous (Microservice extension: Can be deployed
independent)
● Services share schema and contract, not class
● Service compatibility is based on policy
And nowadays, everybody calls it “Microservices”
This is now not 100 % accurate, but generally speaking Microservices are an old
SOA and Microservices
A deeper look into the meaning of:
make it work make it right make it fast
What does it mean?
● Remember TDD Cycle: make it work, make it right
○ test pass - refactor - nice API
● And then: performance (make it fast)
○ profiling bottlenecks, optimize code
● And sometimes this is not enough...
● make it work, make it right, make it fast, make it scale, make it again -
lifecycle in #softwaredevelopment
○ Google: Design for 10fold, rewrite for 2 orders of magnitude...
Anti patterns
● Big Ball of Mud: A system with no recognizable structure
● Cargo Cult Programming: Using patterns and methods without understanding
why
● Accidental complexity: Programming tasks which could be eliminated with
better tools (as opposed to essential complexity inherent in the problem being
solved)
● Dependency hell: Problems with versions of required products
And much more: https://en.wikipedia.org/wiki/Anti-pattern
● I found the silver bullet: There is no silver bullet
● The wrong abstractions: Better RY than the wrong abstraction!
● Being “too clever”: Better clear than clever (sometimes weird and
Communication and maintainability
Final thoughts
Life is about trade-offs
● Do you test? How much do you test?
○ Good discussion about TDD and trade-offs Kent Beck, David Heinemeier Hansson, and Kent
Beck (Is TDD Dead? https://martinfowler.com/articles/is-tdd-dead/ )
● We need to refactor X, but we need feature Y.
● This won’t perform good when we have more users.
● As technologists we want to use cool tech, as employees we need to deliver
value as well
Language recommendations
● Master at least one language
● proficiency (Dreyfus) in a language is nothing one can accomplish in a month
or a year
○ 10 years to learn a language is a good time frame (Peter Norvig): http://norvig.com/21-
days.html
● Stay up-to-date and look into new languages and paradigms: TIOBE Top 50 (
https://www.tiobe.com/tiobe-index/ ) is a good place to look for
● Read papers about algorithms and concepts that matter to you (good sources
are Google Scholar, https://arxiv.org/ or https://blog.acolyer.org/ )
Book recommendations
● Coders at Work (Peter Seibel)
● Clean Code (Robert Martin)
● Working Effectively with Legacy Code (Michael Feathers)
● Domain Driven Design (Eric Evans)
● The Phoenix Project (Gene Kim, George Spafford, and Kevin Behr)
● Mythical Man Month (Fred Brooks)
● Release It! (Michael T. Nygard)
● Building Microservices: Designing Fine-Grained Systems (Sam Newman)
Book recommendations
● Design Patterns (GoF) (Erich Gamma, John Vlissides, Ralph Johnson, and
Richard Helm)
● Enterprise patterns for Application Architecture (Martin Fowler)
● How Google Tests Software (James A. Whittaker, Jason Arbon, and Jeff
Carollo)
● Implementing Domain Driven Design (Vaughn Vernon)
● Beyond Blame (Dave Zwieback)
● 7 languages in 7 weeks (Bruce Tate)
Book recommendations
● 97 Things Every Software Architect Should Know (Richard Monson-Haefel)
● Data Science from Scratch (Joel Grus)
● Building Scalable Websites (Cal Henderson)
○ Some topics might be a bit outdated in the era of Cloud and SaaS, though a lot of good topics
if you are working on websites
● Pragmatic Thinking and Learning: Refactor Your Wetware (Andy Hunt)
● Introduction to Algorithms (Charles E. Leiserson, Clifford Stein, Ronald
Rivest, and Thomas H. Cormen)
● Refactoring (Martin Fowler)
Recap
Communication
Fast Feedback
Continuous Improvement (also live-long learning)
Every Rule has its exception
your mileage may vary
@Dafiti_Tech is hiring…
Want to work in Sao Paulo? Or know
some friend?
Thank you!
Q & A

Software Engineering Primer

  • 1.
  • 2.
    Bio ● 16+ yearsIT stuff ● 1981 - 2011 in Germany ● since 21.07.2011 in Brazil, married, 2 sons Currently ● Head of Innovation, Dafiti ● IT Consultant, independent Contact ● Twitter (follow me for mostly geek stuff): https://twitter.com/georg.buske echo -n $(whoami); Georg.Buske
  • 3.
    Today’s topics (notagenda) ● Process and team organization ○ Agile - what's all about? ○ Small projects VS. large scale projects ○ Real world examples of important IT guidelines in an enterprise ● Software Design & Architecture ○ Clean Code - an evergreen in Software Development ○ abbreviations you need to know: DDD, TDD, KISS, DRY, YAGNI, SOLID ○ A deeper look into the meaning of…
  • 4.
    Disclaimer(s) ● Slides inenglish, talk in portuguese ● This is in no means a deep dive - a primer ○ We will cover a lot of stuff ○ Each topic could easily fill its own talk (or workshop) ○ There is no shortcut ● Slides will be available afterwards
  • 5.
    Process and teamorganization
  • 6.
    Fact: There isno real difference Everything of the talk can be applied to small and big [software] projects [and products] Wait, there is “really small”: ● < 3 developers on a private project or something you own ● then it is OK to ignore a lot of topics ● Use at least a VCS, like Git (even you are working alone, source control makes your life easier) Small projects VS. large scale projects
  • 7.
    Try to splitlarge projects into a lot of small ones! E.g. Amazon has 2 Pizza teams Average team size in other tech companies is between 3 - 10 (which is already a lot!)
  • 8.
    Communication paths byteam size Source: https://www.solutionsiq.com/learning/blog-post/lines-of-communication/
  • 9.
    Agile - what'sall about?
  • 10.
  • 11.
    Different ways ofbeing agile: Scrum Scrum, quite old: 1993 “Real scrum” has more than 30 rules Because of this technically a lot of companies are doing “ScrumBut” But this is fine, as long as we are efficient Some important aspects of Scrum: Prioritized Backlog Timeboxed Sprints (Work organized and delivered in Sprints, e.g. always 2 weeks, no changes) -> deliver value
  • 12.
    Scrum Roles: ● scrum master ●product owner ● participants (qa, devs) Meetings: ● Daily (same place, same time, everyday) ● Retrospective (what was good, what was bad in the last Sprint) ● Planning (what we can accomplish next Sprint)
  • 13.
    Different ways ofbeing agile: Kanban ● part of lean movement (popularized by toyota, 1940s/1950s) ● Less rules than Scrum, important aspects: ○ Visualize (KanbanBoard) ○ limit WIP ○ optimize value streams ○ Implement Feedback loops
  • 14.
    Different ways ofbeing agile: XP ● extreme programming framework ● by Kent Beck (also creator of TDD) ● First used in C3 (Chrysler Comprehensive Compensation project) - Kent Beck, Martin Fowler ● “XP was one of the first agile methods, indeed XP was the dominant agile method in the late 90s and early 00s before Scrum became dominant” ● Contains: ○ Continuous Integration ○ Refactoring
  • 15.
    Real world examplesof important IT guidelines in an enterprise
  • 16.
    Why is itimportant? ● Consistency ● Best practices (we know it works) ● blueprint / howto for everyone (new and old), everybody knows how to do something ● Standard (from Wikipedia): established norm, specification, system of market dominance ● Guideline (from Google search): a general rule, principle or piece of advice
  • 17.
    Dafiti’s IT Docs... ●General development workflow ● Architecture guidelines ● Ticket guidelines ● Coding standards ○ Style Guide templates ● Logging guidelines ● Commit message standard ● Pull request guidelines
  • 18.
    Dafiti’s IT Docs... ●Post-mortem template ● DevOps best practices and failure mode ● Change requests ● POCs ● RFCs ● Service guidelines ● Component integration guidelines ● Component testing best practices
  • 19.
    Examples We will opensourceour Docs, follow @Dafiti_Tech or me (@georgbuske) on Twitter if you are interested ;-)
  • 20.
  • 21.
    Software Design &Architecture
  • 22.
  • 23.
    Definition If the Designis the interior, the architecture is the building Therefore: Design: The API (how to open the door) how looks the dining room) Architecture: The Big Picture, high level (this is how the house [or city] looks like) So in short, Software architecture is more about the design of the entire system, while software design emphasizes on module / component / class level. (Source: https://stackoverflow.com/questions/704855/software-design-vs-software-architecture)
  • 28.
    Avoid Ivory TowerArchitecture Should there be a dedicated team of Software architectures? No! Better: ● committee of senior technologists ● clear set of allowed technologies (tools like Technology Radar: https://www.thoughtworks.com/radar )
  • 29.
    Clean Code -First things first ● Clean Code Book, by Robert “Uncle Bob” Martin ● “Programs must be written for people to read, and only incidentally for machines to execute.” ○ ― Harold Abelson, Structure and Interpretation of Computer Programs ● “No surprises” ○ ― Ward Cunningham ● “Clean code is a code that is written by someone who cares” ○ ― MIchael Feathers
  • 30.
    Clean Code characteristics ●is language independent ● method length ● I should be able to read and understand ● Interfaces ● Error handling ● Boy Scout Rule ○ Always leave the campground cleaner than you found it. ● Unit and acceptance tests
  • 31.
  • 32.
    There are onlytwo hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton
  • 33.
  • 34.
  • 35.
    Naming ● this appliesto everything: variables, classes, functions, interfaces - even repository names ● Rule: Use Intention-Revealing Names ● Rule: Avoid Disinformation ○ E.g. IP as variable name for Identity Provider ● Rule: Make meaningful Distinctions ○ ProductInfo and ProductData ● Classes: Nouns
  • 36.
  • 37.
    SOLID ● subset ofclean code ● 5 principles ○ Single Responsibility Principle ○ Open Close Principle ○ Liskov Substitution Principle ○ Interface Segregation Principle ○ Dependency Inversion Principle
  • 38.
    Single Responsibility Principle Cohesion,a class should have only a single responsibility Should be changed for only one reason minimal and does one thing good (cf. Unix philosophy) and least possible dependencies
  • 40.
    Open Closed Principle softwareentities should be open for extension, but closed for modification
  • 42.
    Liskov Substitution Principle Objectsin a program should be replaceable with instances of their subtypes without altering the correctness of that program
  • 44.
    Interface Segregation Principle Manyclient specific interfaces are better than one big one
  • 45.
    Dependency Inversion Principle Oneshould depend upon abstractions, not concretions A.High-level modules should not depend on low-level modules. Both should depend on abstractions. B.Abstractions should not depend on details. Details should depend on abstractions. Example Patterns: Plugin, ServiceLocator, Dependency Injection -> Inversion of control
  • 47.
    KISS = Keepit simple, stupid ● no overly complex solutions ● straightforward is often easier, cleaner and better to understand ● if you do TDD, this helps
  • 48.
    ● code duplicationis bad [copy paste] ● now, this can conflict with microservices (more on microservices later) DRY = Don’t repeat yourself
  • 49.
    YAGNI = Youain’t gonna need it very close related to KISS For example: unnecessary complexity Except you are writing library code probably don't need to add 4 different types of authentication methods (Strategies, Adapters) - if there is the requirement to change, it is often long-term
  • 50.
    DDD = DomainDriven Design ● term coined by Eric Evans ● focus on problem domain ● Combines lots of best practices and patterns ○ Refactoring ○ Continuous Integration
  • 51.
    Context mapping &bounded contexts ● Core domain, generic subdomain, supporting domain ○ Best forces on most important part of the system (your product’s USP), core domains ○ Generic subdomains could be bought off-the-shelf
  • 52.
    Ubiquitous language ● Everyonein team has the same vocabulary
  • 53.
    Tactical Patterns ● Entities ○Has identity ● Aggregate Roots ○ Are the entry points ● Value Objects ○ Immutable ● Repositories ○ Data Layer
  • 54.
    TDD = TestDriven Development
  • 55.
    Test Driven Development ●Red - green - refactor ● test first development (unit testing) ● write failing test, implement, refactor, iterate ● you can certainly test after, but advantages for test first are focus on features ● evolving and iterative design, to achieve KISS, DRY and YAGNI
  • 56.
    Bonus: SOA =Service Oriented Architecture
  • 57.
    SOA is akind of 20 year old concept ● Boundaries are explicit ● Services are autonomous (Microservice extension: Can be deployed independent) ● Services share schema and contract, not class ● Service compatibility is based on policy And nowadays, everybody calls it “Microservices” This is now not 100 % accurate, but generally speaking Microservices are an old SOA and Microservices
  • 58.
    A deeper lookinto the meaning of: make it work make it right make it fast
  • 59.
    What does itmean? ● Remember TDD Cycle: make it work, make it right ○ test pass - refactor - nice API ● And then: performance (make it fast) ○ profiling bottlenecks, optimize code ● And sometimes this is not enough... ● make it work, make it right, make it fast, make it scale, make it again - lifecycle in #softwaredevelopment ○ Google: Design for 10fold, rewrite for 2 orders of magnitude...
  • 60.
  • 61.
    ● Big Ballof Mud: A system with no recognizable structure ● Cargo Cult Programming: Using patterns and methods without understanding why ● Accidental complexity: Programming tasks which could be eliminated with better tools (as opposed to essential complexity inherent in the problem being solved) ● Dependency hell: Problems with versions of required products And much more: https://en.wikipedia.org/wiki/Anti-pattern ● I found the silver bullet: There is no silver bullet ● The wrong abstractions: Better RY than the wrong abstraction! ● Being “too clever”: Better clear than clever (sometimes weird and
  • 62.
  • 63.
  • 64.
    Life is abouttrade-offs ● Do you test? How much do you test? ○ Good discussion about TDD and trade-offs Kent Beck, David Heinemeier Hansson, and Kent Beck (Is TDD Dead? https://martinfowler.com/articles/is-tdd-dead/ ) ● We need to refactor X, but we need feature Y. ● This won’t perform good when we have more users. ● As technologists we want to use cool tech, as employees we need to deliver value as well
  • 65.
    Language recommendations ● Masterat least one language ● proficiency (Dreyfus) in a language is nothing one can accomplish in a month or a year ○ 10 years to learn a language is a good time frame (Peter Norvig): http://norvig.com/21- days.html ● Stay up-to-date and look into new languages and paradigms: TIOBE Top 50 ( https://www.tiobe.com/tiobe-index/ ) is a good place to look for ● Read papers about algorithms and concepts that matter to you (good sources are Google Scholar, https://arxiv.org/ or https://blog.acolyer.org/ )
  • 66.
    Book recommendations ● Codersat Work (Peter Seibel) ● Clean Code (Robert Martin) ● Working Effectively with Legacy Code (Michael Feathers) ● Domain Driven Design (Eric Evans) ● The Phoenix Project (Gene Kim, George Spafford, and Kevin Behr) ● Mythical Man Month (Fred Brooks) ● Release It! (Michael T. Nygard) ● Building Microservices: Designing Fine-Grained Systems (Sam Newman)
  • 67.
    Book recommendations ● DesignPatterns (GoF) (Erich Gamma, John Vlissides, Ralph Johnson, and Richard Helm) ● Enterprise patterns for Application Architecture (Martin Fowler) ● How Google Tests Software (James A. Whittaker, Jason Arbon, and Jeff Carollo) ● Implementing Domain Driven Design (Vaughn Vernon) ● Beyond Blame (Dave Zwieback) ● 7 languages in 7 weeks (Bruce Tate)
  • 68.
    Book recommendations ● 97Things Every Software Architect Should Know (Richard Monson-Haefel) ● Data Science from Scratch (Joel Grus) ● Building Scalable Websites (Cal Henderson) ○ Some topics might be a bit outdated in the era of Cloud and SaaS, though a lot of good topics if you are working on websites ● Pragmatic Thinking and Learning: Refactor Your Wetware (Andy Hunt) ● Introduction to Algorithms (Charles E. Leiserson, Clifford Stein, Ronald Rivest, and Thomas H. Cormen) ● Refactoring (Martin Fowler)
  • 69.
  • 70.
    Every Rule hasits exception your mileage may vary
  • 71.
    @Dafiti_Tech is hiring… Wantto work in Sao Paulo? Or know some friend?
  • 72.