Buzzword Deathmatch: Agile vs SOA formerly “ Agile Development with SOA”

About me 10 years experience in IT, mainly as a consultant Took part in many large scale projects Government(s) Banking Insurances A foot in the process, the other one in the architecture. My blog: http://ziobrando.blogspot.com

10 years experience in IT, mainly as a consultant

Took part in many large scale projects

Government(s)

Banking

Insurances

A foot in the process, the other one in the architecture.

My blog: http://ziobrando.blogspot.com

Agenda Starting from the dinosaurs… The Agile Landscape The SOA Landscape

Starting from the dinosaurs…

The Agile Landscape

The SOA Landscape

The Agile Rationale Waterfall software development proved itself inefficient and unsatisfactory Waterfall is “traditionally” associated with high cost, long development time Result unpredictability and low success ratio Difficulties in managing and accommodating change If asked, nobody is doing waterfall anymore (…or so they think)

Waterfall software development proved itself inefficient and unsatisfactory

Waterfall is “traditionally” associated with

high cost,

long development time

Result unpredictability and low success ratio

Difficulties in managing and accommodating change

If asked, nobody is doing waterfall anymore (…or so they think)

Unified Process The unified process changed the scenario Iterations as a fundamental part of the process Fine grained roles and artefact definition UML as the official representation language A comprehensive definition of all development process activities Unfortunately, it also set the ground for A family of UML modelling tools A lot of documentation templates

The unified process changed the scenario

Iterations as a fundamental part of the process

Fine grained roles and artefact definition

UML as the official representation language

A comprehensive definition of all development process activities

Unfortunately, it also set the ground for

A family of UML modelling tools

A lot of documentation templates

The Dark side of Unified Process The tools became more and more important, ultimately twisting the process Analysis and design turned into solo activities Paper, paper, paper, and more paper

The tools became more and more important, ultimately twisting the process

Analysis and design turned into solo activities

Paper, paper, paper, and more paper

Developers Developers were considered “interchangeable resources” whose only goal was to implement the specification, though iteratively .

Developers were considered “interchangeable resources” whose only goal was to implement the specification, though iteratively .

Roles and responsibilities The fine grained roles framework eventually turned into a slow down factor: People took over only “the right tasks” (implicit waterfall) Slower response Bottlenecks Blame game ...

The fine grained roles framework eventually turned into a slow down factor:

People took over only “the right tasks”

(implicit waterfall)

Slower response

Bottlenecks

Blame game

...

Rebel forces gathered

The three flavours of Agility XP made a breakthrough focusing on extreme software development practices Scrum defined a framework in which Agile practices could be applied within an organization Lean software development introduced concepts and principles from the manufacturing industry into software development (defining also a theoretical background)

XP made a breakthrough focusing on extreme software development practices

Scrum defined a framework in which Agile practices could be applied within an organization

Lean software development introduced concepts and principles from the manufacturing industry into software development (defining also a theoretical background)

eXtreme Programming User Stories Less formal than Use cases, act like placeholder for a real discussion Frequent small releases Iterations are shorter and targeted to production, Frequent planning and estimation Each iteration is re-planned according to the currently available information No anticipated development No frameworks or layered architecture Test first Test suites run preserving application integrity, and producing better interfaces Customer availability Real feedback is the key to make the right thing No code ownership Continuous integration The whole system is frequently integrated and tested Pair programming Programmers work in pairs, in coding sessions No overtime ...

User Stories

Less formal than Use cases, act like placeholder for a real discussion

Frequent small releases

Iterations are shorter and targeted to production,

Frequent planning and estimation

Each iteration is re-planned according to the currently available information

No anticipated development

No frameworks or layered architecture

Test first

Test suites run preserving application integrity, and producing better interfaces

Customer availability

Real feedback is the key to make the right thing

No code ownership

Continuous integration

The whole system is frequently integrated and tested

Pair programming

Programmers work in pairs, in coding sessions

No overtime

...

XP Feedback seems to be the driving factor for many of the proposed practices From the code From peers From the customer From the project From the team The team is largely empowered and encouraged to propose original solutions Process itself might be modified or improved

Feedback seems to be the driving factor for many of the proposed practices

From the code

From peers

From the customer

From the project

From the team

The team is largely empowered and encouraged to propose original solutions

Process itself might be modified or improved

Scrum Scrum does not refer strictly to software development, but provides a framework for adaptive project management Unlike UP, only 3 roles are defined in Scrum The Team The Product Owner The Scrum Master

Scrum does not refer strictly to software development, but provides a framework for adaptive project management

Unlike UP, only 3 roles are defined in Scrum

The Team

The Product Owner

The Scrum Master

Development team landscape

The agile team Small units Cross-functional The team is free to take any design decision In Scrum, the team reports only to the product owner A well defined ceremony and set of actions to prevent the team to drift in dangerous directions

Small units

Cross-functional

The team is free to take any design decision

In Scrum, the team reports only to the product owner

A well defined ceremony and set of actions to prevent the team to drift in dangerous directions

Dealing with several stakeholders The Product Owner is the ONE and ONLY responsible for the development team.

The Product Owner is the ONE and ONLY responsible for the development team.

XP vs Scrum XP Frequent iterations of working software Frequent status update and re-planning Focus on the development team internal organization and practices Scrum Frequent iterations of working software Frequent status update and re-planning Focus on the relations between the development team and the organization

XP

Frequent iterations of working software

Frequent status update and re-planning

Focus on the development team internal organization and practices

Scrum

Frequent iterations of working software

Frequent status update and re-planning

Focus on the relations between the development team and the organization

XP and Scrum The two perspectives are largely complementary: XP does not say much about what happens outside of the development team Scrum lets the team free to organize, and XP might be a sensible choice Scrum has definitely a better marketing

The two perspectives are largely complementary:

XP does not say much about what happens outside of the development team

Scrum lets the team free to organize, and XP might be a sensible choice

Scrum has definitely a better marketing

Lean software development principles Eliminate waste Everything that is not delivering any value to the user Amplify learning Developing is discovery, Decide as late as possible Avoid irreversible decisions or take them only when you have as much information as needed to decide Deliver as fast as possible Fast development cycles help feedback. Speed is better than quality Empower the team The team is or will become the maximum expert on the matter Build integrity in Software must be useful, used and right for the job See the whole Failing to see the whole picture will lead to local optimizations

Eliminate waste

Everything that is not delivering any value to the user

Amplify learning

Developing is discovery,

Decide as late as possible

Avoid irreversible decisions or take them only when you have as much information as needed to decide

Deliver as fast as possible

Fast development cycles help feedback. Speed is better than quality

Empower the team

The team is or will become the maximum expert on the matter

Build integrity in

Software must be useful, used and right for the job

See the whole

Failing to see the whole picture will lead to local optimizations

Waste Instances of waste that can show up in different shapes Unnecessary documentation Anticipated design Over engineering Waiting Communication leaks Defects Handoff Complexity Blame Quality (?) ...

Instances of waste that can show up in different shapes

Unnecessary documentation

Anticipated design

Over engineering

Waiting

Communication leaks

Defects

Handoff

Complexity

Blame

Quality (?)

...

The lo-fi approach As a consequence, some tools were deprecated, favouring a non-tech approach: Paper Post-it Whiteboards Information radiators took advantage of physical proximity to allow a more efficient exchange of meaningful information (where possible) Some tools were basically banned (es. MSProject), others entered the show (es. Wikis)

As a consequence, some tools were deprecated, favouring a non-tech approach:

Paper

Post-it

Whiteboards

Information radiators took advantage of physical proximity to allow a more efficient exchange of meaningful information (where possible)

Some tools were basically banned (es. MSProject), others entered the show (es. Wikis)

The bottom-up revolution Agile sets the ground for interesting proposal to emerge from the team The team learns and focuses on a given problem space eventually turning into the best available expert on the matter

Agile sets the ground for interesting proposal to emerge from the team

The team learns and focuses on a given problem space eventually turning into the best available expert on the matter

Breeding collaboration Isolation happens seldom Many activities are performed in groups, or pairing, or in a clearly visible way Transparency enables trust and a more effective form of collaboration Information exchange happens in both ways

Isolation happens seldom

Many activities are performed in groups, or pairing, or in a clearly visible way

Transparency enables trust and a more effective form of collaboration

Information exchange happens in both ways

The Ideal Agile scenario Not every condition is optimal to turn to agile: to fully benefit of agile’s potential we should (ideally) Be employed by a company in whose top priority is software development. Be located in the same place Have access to customers (whatever this means) Be free to grow as a team and assume responsibilities Free to arrange logistics, hardware, etc.

Not every condition is optimal to turn to agile: to fully benefit of agile’s potential we should (ideally)

Be employed by a company in whose top priority is software development.

Be located in the same place

Have access to customers (whatever this means)

Be free to grow as a team and assume responsibilities

Free to arrange logistics, hardware, etc.

Agenda Setting the Background The Agile Landscape The SOA Landscape

Setting the Background

The Agile Landscape

The SOA Landscape

SOA Rationale Large organizations were burdened with the weight of Different legacy systems Mergers and acquisitions Necessity to integrate heterogeneous systems Duplicate and redundant systems High (unnecessary) complexity Operation costs Evolution costs Extremely slow reaction times, or ...basically being stuck, or failing to deliver value ...Does all this sound familiar?

Large organizations were burdened with the weight of

Different legacy systems

Mergers and acquisitions

Necessity to integrate heterogeneous systems

Duplicate and redundant systems

High (unnecessary) complexity

Operation costs

Evolution costs

Extremely slow reaction times, or ...basically being stuck, or failing to deliver value

...Does all this sound familiar?

Pursuing uniformity Standards, frameworks, rules and tight integration failed to deliver the needed business agility, resulting in a heavier burden to take into account, before even designing a possible solution to a given problem.

Standards, frameworks, rules and tight integration failed to deliver the needed business agility, resulting in a heavier burden to take into account, before even designing a possible solution to a given problem.

Service Oriented Architecture SOA aims to dramatically reduce coupling between applications in a given landscape: Language coupling  XML Protocol coupling  WS, SOAP Network coupling  ESB Availability coupling  messages Standards and low coupling defined an enterprise-level architecture made up of replaceable components

SOA aims to dramatically reduce coupling between applications in a given landscape:

Language coupling  XML

Protocol coupling  WS, SOAP

Network coupling  ESB

Availability coupling  messages

Standards and low coupling defined an enterprise-level architecture made up of replaceable components

Low coupling Services are meant to talk to each other, with the lowest possible reciprocal knowledge Enterprise Service Bus

Services are meant to talk to each other, with the lowest possible reciprocal knowledge

The ideal goal SOA was a tool to Allow the long awaited necessary cleanup Allow IT to become a driving factor for the business instead of a burden “ It’s a mess” “ I’ll schedule it for 2010” “ We have no time right now” “ We can do that” “ Why not doing that instead?” “ ...We have an idea” Allow enterprises to reduce vendor lock-in recovering control on IT budget.

SOA was a tool to

Allow the long awaited necessary cleanup

Allow IT to become a driving factor for the business instead of a burden

“ It’s a mess”

“ I’ll schedule it for 2010”

“ We have no time right now”

“ We can do that”

“ Why not doing that instead?”

“ ...We have an idea”

Allow enterprises to reduce vendor lock-in recovering control on IT budget.

Put in another way SOA is a tool to allow large enterprises to achieve business agility Shorter products release cycles Getting feedback straight from the market Experimenting new ways to make business SOA is not a way to recreate an existing structure with a newer technology

SOA is a tool to allow large enterprises to achieve business agility

Shorter products release cycles

Getting feedback straight from the market

Experimenting new ways to make business

SOA is not a way to recreate an existing structure with a newer technology

The “classical” SOA scenario The agile greenfield Be employed by a company in whose top priority is software development. Be located in the same place Have access to customers (whatever this means) Be free to grow as a team and assume responsibilities Free to arrange logistics, hardware, etc. The “classical” SOA Company’s top priority is generally not software development Consultants, contractors (and sub-contractors) are the rule Development takes often place in multiple locations , remote and offshore teams are possible. Smaller incentives to grow and assume responsibilities Low control over logistics, hardware, etc.

The agile greenfield

Be employed by a company in whose top priority is software development.

Be located in the same place

Have access to customers (whatever this means)

Be free to grow as a team and assume responsibilities

Free to arrange logistics, hardware, etc.

The “classical” SOA

Company’s top priority is generally not software development

Consultants, contractors (and sub-contractors) are the rule

Development takes often place in multiple locations , remote and offshore teams are possible.

Smaller incentives to grow and assume responsibilities

Low control over logistics, hardware, etc.

Unleashing freedom Loose coupling and language neutral standards offered the possibility to develop applications in the most appropriate technology Enterprise Service Bus

Loose coupling and language neutral standards offered the possibility to develop applications in the most appropriate technology

... Maybe we’re still coupled...? Technical coupling was only one of the many coupling factor on the stage: Licence budget Operations & Support Standards, Rules and existing prescriptions HR strategies Architecture Corporate culture

Technical coupling was only one of the many coupling factor on the stage:

Licence budget

Operations & Support

Standards, Rules and existing prescriptions

HR strategies

Architecture

Corporate culture

A new Jargon UML was not enough for SOA needs A new Jargon eventually emerged as well as new notations, new diagrams, new stencils, new layers ...

UML was not enough for SOA needs

A new Jargon eventually emerged as well as new notations, new diagrams, new stencils, new layers ...

Agenda Setting the Background The Agile Landscape The SOA Landscape Putting things together

Setting the Background

The Agile Landscape

The SOA Landscape

Putting things together

Pairing Agile and SOA “ Enterprises experience more success with SOA when they eschew big top-down delivery projects and instead get down in the trenches with an evolutionary approach. Agile processes provide a basic structure for this kind of incremental delivery.” Carey Schwaber, Forrester Research Interestingly, not so many articles tell how Agility benefits from SOA.

“ Enterprises experience more success with SOA when they eschew big top-down delivery projects and instead get down in the trenches with an evolutionary approach. Agile processes provide a basic structure for this kind of incremental delivery.”

Carey Schwaber, Forrester Research

Interestingly, not so many articles tell how Agility benefits from SOA.

Agile as a Cooperative Game Software development might be classified as a finite, goal-directed, cooperative game (Cockburn) Carpet wrestling Jazz Tennis Software Development Dolls Competitive Cooperative Organization Survival Career management Finite, Non-goal-directed Finite, goal-directed Infinite

Software development might be classified as a finite, goal-directed, cooperative game (Cockburn)

Software development as a Game Finite: a project starts and ends (somehow) Goal directed: es. deliver on time Collaborative : we’re playing together within a team … but we’re not only doing that: We’re playing career game Playing family game Etc. etc.

Finite: a project starts and ends (somehow)

Goal directed: es. deliver on time

Collaborative : we’re playing together within a team

… but we’re not only doing that:

We’re playing career game

Playing family game

Etc. etc.

A successful team

Some key issues The Team Best of breed selection Team goals before individual ones High motivation The goal Clearly defined goal Time-framed experience Measurable outcome

The Team

Best of breed selection

Team goals before individual ones

High motivation

The goal

Clearly defined goal

Time-framed experience

Measurable outcome

A “not so successful” team

Key issues The team Best of breed? Individual goals before common goals The Goal Not so clearly defined Random time-frame Non measurable outcome (only history will tell)

The team

Best of breed?

Individual goals before common goals

The Goal

Not so clearly defined

Random time-frame

Non measurable outcome (only history will tell)

Limited resources game A Software Development scenario is bounded on several key areas: Budget Time Expertise Talent Manageable Information

A Software Development scenario is bounded on several key areas:

Budget

Time

Expertise

Talent

Manageable Information

SOA game? SOA is generally a different type of game played at a different level: SOA’s goal might be of a longer term strategy Mid term results might not be easily observable and measurable by the team. Es. Budget Suppression of an external system

SOA is generally a different type of game played at a different level:

SOA’s goal might be of a longer term strategy

Mid term results might not be easily observable and measurable by the team.

Es. Budget

Suppression of an external system

The SOA Game A common critic to Agile methods is that they focus on a short-term strategy, in a given project scope. SOA aims to see “the whole picture” focusing on somewhat obscure long-term goals

A common critic to Agile methods is that they focus on a short-term strategy, in a given project scope.

The dominant culture Enterprise culture might be far from agility SOA might have management’s sponsorship but agile might not. Careers, roles and specialities might be put under pressure

Enterprise culture might be far from agility

SOA might have management’s sponsorship but agile might not.

Careers, roles and specialities might be put under pressure

Project Size SOA scope calls for many development teams at a time.

SOA scope calls for many development teams at a time.

A more realistic scenario

Team’s reward Individual bonuses might be counterproductive or out of control Consultants and contractors might not be reached We’ve got to work on something else Increase the team’s knowledge Improve working conditions Bring (honest) positive feedback ...

Individual bonuses might be counterproductive or out of control

Consultants and contractors might not be reached

We’ve got to work on something else

Increase the team’s knowledge

Improve working conditions

Bring (honest) positive feedback

...

Top – Down reprise Some tools and artifacts re-introduce a top-down philosophy Role based game Tool driven design Specification based development Another vicious effect is that ... Bottom up feedback is discouraged Possible good ideas get lost

Some tools and artifacts re-introduce a top-down philosophy

Role based game

Tool driven design

Specification based development

Another vicious effect is that ...

Bottom up feedback is discouraged

Possible good ideas get lost

Starting Agile and SOA? Though “orthogonal”, two revolutions at the same time are probably too much for any team But ...agile performs well where a lot of explorations and uncertainty is involved: Adaptive process spikes For a good Agile team SOA is “Just another Technical Hassle”

Though “orthogonal”, two revolutions at the same time are probably too much for any team

But ...agile performs well where a lot of explorations and uncertainty is involved:

Adaptive process

spikes

For a good Agile team SOA is “Just another Technical Hassle”

Read the scenario Different roles within the organization read buzzwords differently: Team might value agility more (and blame SOA if something goes wrong) Architects might value SOA more (and blame agility if something goes wrong) Management couldn’t care less And value only results

Different roles within the organization read buzzwords differently:

Team might value agility more

(and blame SOA if something goes wrong)

Architects might value SOA more

(and blame agility if something goes wrong)

Management couldn’t care less

And value only results

Set up the scene Don’t raise expectations you cannot fulfil Keep management aware of potential emerging problems Transitions to agile practices (especially Scrum and lean) stress the organization, exposing problems that lived long “under the carpet”. Problems have always being there, but pointing them out might result impolite .

Don’t raise expectations you cannot fulfil

Keep management aware of potential emerging problems

Transitions to agile practices (especially Scrum and lean) stress the organization, exposing problems that lived long “under the carpet”.

Problems have always being there, but pointing them out might result impolite .

Choose a close target first Can’t wait months to prove that choices were good. Look to close targets Achieve them Build on this Confidence Team jellying Reputation

Can’t wait months to prove that choices were good.

Look to close targets

Achieve them

Build on this

Confidence

Team jellying

Reputation

Travel light Availability of tools to manage SOA-related complexity does not mean that complexity has to be encouraged The dark side of SOA would not be better than it was before

Availability of tools to manage SOA-related complexity does not mean that complexity has to be encouraged

The dark side of SOA would not be better than it was before

Adhere to standards As a corollary to “ Decide as late as possible ”: Develop on standards whenever possible Generally better documentation More easily replaceable ... Think long term Avoid temptations from vendor-specific features Keep deviation from standards under control Don’t buy anything until proves necessary.

As a corollary to “ Decide as late as possible ”:

Develop on standards whenever possible

Generally better documentation

More easily replaceable

... Think long term

Avoid temptations from vendor-specific features

Keep deviation from standards under control

Don’t buy anything until proves necessary.

Rethink waste Some obvious form of waste “doing the same thing twice” might be preferable to “document what you’ve been doing” Large scale changes priority Company and location boundaries impact Communication cost Handoff cost

Some obvious form of waste “doing the same thing twice” might be preferable to “document what you’ve been doing”

Large scale changes priority

Company and location boundaries impact

Communication cost

Handoff cost

The cost of communication Information does not propagate with documentation, but by knowing what others are doing. Keep multiple teams in sync Scrum of Scrums Information radiators Proximity End of iteration demo When written words are better, favour Wikis and on-demand documentation.

Information does not propagate with documentation, but by knowing what others are doing.

Keep multiple teams in sync

Scrum of Scrums

Information radiators

Proximity

End of iteration demo

When written words are better, favour Wikis and on-demand documentation.

Share the big plan Preventing developers to see the whole picture severely harms the possibility of a bottom-up contribution Only local optimizations are possible 

Preventing developers to see the whole picture severely harms the possibility of a bottom-up contribution

Only local optimizations are possible 

High level planning SOA is a long term transformation process Every step changes the scenario More knowledge Different weight Different opportunities Frequent high-level re-planning is necessary to achieve the goals (not necessarily the original ones) Measure , don’t expect Assess risks , don’t make predictions Don’t initiate anything that’s not needed now

SOA is a long term transformation process

Every step changes the scenario

More knowledge

Different weight

Different opportunities

Frequent high-level re-planning is necessary to achieve the goals (not necessarily the original ones)

Measure , don’t expect

Assess risks , don’t make predictions

Don’t initiate anything that’s not needed now

Testing a SOA SOA overall design is based on Design by Contract principles Clean interface definition based upon standards “ Official” expected service behaviour Let’s test on the contract!

SOA overall design is based on Design by Contract principles

Clean interface definition based upon standards

“ Official” expected service behaviour

Let’s test on the contract!

Testing a SOA: challenges Language independent test suite  Mocks and Stubs implementing services  Selection of key areas  Test environment management 

Language independent test suite 

Mocks and Stubs implementing services 

Selection of key areas 

Test environment management 

Environment definition Staging environment might be hard to afford in certain context Keep continuous integration tools running and testing the app Extend your integration capabilities as far as you can get Find a balance between correctness and manageability Keep the tests updated

Staging environment might be hard to afford in certain context

Keep continuous integration tools running and testing the app

Extend your integration capabilities as far as you can get

Find a balance between correctness and manageability

Keep the tests updated

Put in two words... Don’t let the “old bad habits” strike back Be prepared to compromises, in the short term (it’s a limited resource game) Keep track the price you pay Be ready to change them if opportunities arise Always aim to your long term goals Involve the right sponsors Communicate! Be ready for a rollercoaster ride!

Don’t let the “old bad habits” strike back

Be prepared to compromises, in the short term (it’s a limited resource game)

Keep track the price you pay

Be ready to change them if opportunities arise

Always aim to your long term goals

Involve the right sponsors

Communicate!

Be ready for a rollercoaster ride!

References Agile Software Development – Alistair Cockburn (Addison Wesley) Extreme Programming Explained – Kent Beck (Addison Wesley) The Unified Software Development Process – Ivar Jacobson, Grady Booch, James Rumbaugh (Addison Wesley) Agile Modeling – Scott Ambler (Wiley)

Agile Software Development – Alistair Cockburn (Addison Wesley)

Extreme Programming Explained – Kent Beck (Addison Wesley)

The Unified Software Development Process – Ivar Jacobson, Grady Booch, James Rumbaugh (Addison Wesley)

Agile Modeling – Scott Ambler (Wiley)

References Lean Software Development: An Agile Toolkit – Mary Poppendieck, Tom Poppendieck (Addison Wesley) User Stories Applied – Mike Cohn (Addison Wesley) Enterprise SOA: Service-Oriented Architecture Best Practices – Dirk Krafzig, Karl Banke, Dirk Slama (Prentice Hall) Service-Oriented Architecture: A Field Guide to Integrating XML and Web Services – Thomas Erl (Prentice Hall)

Lean Software Development: An Agile Toolkit – Mary Poppendieck, Tom Poppendieck (Addison Wesley)

User Stories Applied – Mike Cohn (Addison Wesley)

Enterprise SOA: Service-Oriented Architecture Best Practices – Dirk Krafzig, Karl Banke, Dirk Slama (Prentice Hall)

Service-Oriented Architecture: A Field Guide to Integrating XML and Web Services – Thomas Erl (Prentice Hall)