This document discusses balancing upfront planning and emergence in enterprise architecture. It addresses finding the right balance between anticipating needs through planning and adapting to emerging needs. The key is focusing on the goal of maximizing value delivery and ROI through the appropriate level of planning based on the context. Emergence is defined as harnessing the collective intelligence of an organization through empowering individuals and enabling novel solutions to emerge from interactions between agents rather than top-down control. Fostering emergence involves attending to relationships and enabling small changes to lead to large effects.

1. Enterprise ArchitectureBalancing Upfront Planning and Emergence

2. Published Description of this Talk“How do we handle the balance between the big up front plan, the ivory tower EA team, with the need to inspect and adapt and respond to the emerging needs of our customers? Can you identify the challenges with this description?

3. Survey Question #1What stakeholder groups are represented in the audience?End-UsersThe Business (sales, marketing, key management, etc)CustomersDomain Experts (analyst, architect, systems engineer, etc)Developers (designer, coder, tester, etc)

4. Survey Question #2How many of you consider yourself an “agilist”?Agile DevelopmentWaterfall DevelopmentIterative DevelopmentIterative and Incremental DevelopmentParallel Test DevelopmentAcceptance Test Driven DevelopmentMeasure of SuccessConformance to PlanConstant Flow of Business ValueProcessHandoffs/SequentialTeamwork/ParallelCultureCommand and ControlLeadership and CollaborationDesignBig Design Up FrontContinuousQABig Test on BackendContinuousTool SupportHighly SpecificFully IntegratedLean and Efficient Value Streams

5. Which is Better?Predictive?Adaptive?

6. Which is Better?Trick QuestionObviously Both Are RequiredBalance Is RequiredBalance is the Real Trick!PredictiveAdaptive

7. Balance Is The Trick“Any development effort should be a balance between anticipation (planning based on what we know) and adaptation (responding to what we learn over time).”Jim Highsmith, Embracing Change, 17th March 2011

8. How Do We Find The Right Balance?“It Depends”“It’s Situationally Specific”“It’s All About The Context”Shucks! I Want It To Be Easy!Don’t you hate those answers?

9. How Do We Make It “Easier”?Focus On The Goal!Huh? What’s the Goal?

10. What Is Our Goal?Every Business Is Exactly The SameEvery Business Has Exactly The Same GoalEvery Business Is In Business… TO MAKE MONEY!

11. So, How Much Should We Plan?(How do we find the right balance?)Plan as much as necessary to receive a positive ROI on the planning investmentWe Plan Only ToMaximize ValueDelivery

12. Balance Is The Trick“If one has strong discipline without agility, the result is bureaucracy and stagnation. Agility without discipline is the unencumbered enthusiasm of a start-up company before it has to turn a profit.”Balancing Agility and Discipline, Barry Boehm and Richard Turner

13. Survey Question #3On A Scale of 1 – 5, How Important Is Planning?Planning Is The Purest Form Of Evil………Planning Is The True Secret Sauce of DevelopmentAnother Trick Question. Have You Been Paying Attention? 

14. AgendaIntroductionWhat is Architecture?What is an Architect?What is Emergence?How Can We Foster Emergence?How To Maximize Value DeliveryConclusion and Q&A

15. Survey Question #4How large are the companies represented by this audience?< 500 Associates501 – 1,000 Associates1,001 – 5,000 Associates5,001 – 25,000 Associates> 25,000 Associates

16. Definition of Architecture“Architecture is the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution.” [IEEE 1471]IEEE Computer Society, IEEE Recommended Practice for Architectural Description of Software-Intensive Systems: IEEE Std 1472000. 2000.

17. Definition of Architecture“The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them.”Bass, Len; Clements, Paul; & Kazman, Rick. Software Architecture in Practice, Second Edition. Boston, MA: Addison-Wesley, 2003.

18. Definition of ArchitectureSoftware Architecture encompasses the set of significant decisions about the organization of a software system:Selection of the structural elements and their interfaces, by which a system is composed

19. Behavior as specified in collaborations among those elements

20. Composition of these structural and behavioral elements into larger subsystems

21. Architectural style that guides this organizationG. Booch, P. Krutchen, K. Bittner and R. Reitman. The Rational Unified Process — An Introduction. 1999. Definition derived from Mary Shaw’s definition presented in 1995 at the First International Workshop on Architectures for Software Systems.

22. Definition Of Architecture"Architecture is about the important stuff. Whatever that is.“Who Needs An Architect?, Martin Fowlerhttp://martinfowler.com/ieeeSoftware/whoNeedsArchitect.pdfArchitecture is the stuff that's hard to change later. And there should be as little of that stuff as possible.Martin Fowler via Neil FordEvolutionary architecture and emergent design: Evolutionary architecturehttp://public.dhe.ibm.com/software/dw/java/j-eaed10-pdf.pdf

23. Setting The Record StraightArchitecture Has A Tremendous Impact On ResultsFew properties have as much impact on an organization’s success as does architecture.There is ALWAYS an ArchitectureArchitecture is an inherent property of your organization and your software. You may not understand it. You may not communicate it. You may not have planned it. But it’s there!Architecture <> DocumentationArchitecture is that inherent property of the system – not the artifacts that describe it.Architecture <> Infrastructure / HardwareArchitecture encompasses the myriad of perspectives and concerns that make up a solution - and enable its production. That includes hardware, software, operations, support, organization, etc.These Concepts Apply To Both “Enterprise” and “Application”For purposes of today’s discussion – how does emergence apply to architecture and planning – there is no difference between “Enterprise Architecture” and “Application Architecture”

24. AgendaIntroductionWhat is Architecture?What is an Architect?What is Emergence?How Can We Foster Emergence?How To Maximize Value DeliveryConclusion and Q&A

25. Why Do We Do Architecture?To capture the stakeholder perspectives that affect designTo embrace change and to reduce the cost of solving problemsTo create a shared vision across the team and stakeholdersTo smooth the decision-making process- Lean Architecture, Jim Coplien and Gertrud BjørnvigFocus On The Goal!

26. What Is An Architect?“Leads the development of the system's software architecture, which includes promoting and creating support for the key technical decisions that constrain the overall design and implementation for the project.” - Rational Unified Process

27. What Is An Architect?“An architect is a business-minded person who best understands how to leverage technology to maximize profitability.”- Paul Preiss, IASA (paraphrasing)

28. What Is An Architect?“A true software architect is one who is a domain expert, who knows how to apply the domain expertise to the design of a particular system, and who materially participates in implementation.” - Lean Architecture, Jim Coplien and Gertrud Bjørnvig

29. Competencies By RoleA Perfect 10! 

30. Architect Focus Over The Life of a ProjectDiscoveryInventionImplementationFocusThe Architect is accountable from “Concept to Cash”Project LifecycleIBM – Brian Selic

31. Survey Question #5True of False: My Organization Has Individuals Specifically Assigned To The Architect Role.

32. Why Do We Do Architecture?“Architects” are not required, but these outcomes must be satisfied!To capture the stakeholder perspectives that affect designTo embrace change and to reduce the cost of solving problemsTo create a shared vision across the team and stakeholdersTo smooth the decision-making processLean Architecture, Jim Coplien and Gertrud Bjørnvig

33. AgendaIntroductionWhat is Architecture?What is an Architect?What is Emergence?How Can We Foster Emergence?How To Maximize Value DeliveryConclusion and Q&A

34. Survey Question #6True or False?Emergence is a principle thatencourages us to defer decisions to the“Last Responsible Moment” and avoid“Big Up Front Planning” or “Big Up Front Design”

35. One Definition Of EmergenceEmergence is what happens when the whole is smarter than the sum of its parts. It's what happens when you have a system of relatively simple-minded component parts… and they interact in relatively simple ways. And yet somehow out of all this interaction some higher level structure or intelligence appears, usually without any master planner calling the shots. These kinds of systems tend to evolve from the ground up.http://www.oreillynet.com/lpt/a/1574Another Definition Of Emergence"They are bottom-up systems, not top-down. They get their smarts from below. In a more technical language, they are complex adaptive systems that display emergent behavior. In these systems agents residing on one scale start producing behavior that lies one scale above them: ants create colonies; urbanites create neighborhoods; simple pattern-recognition software learns how to recommend new books. The movement from low-level rules to higher-level sophistication is what we call emergence.“- Emergence, Steven Johnson

36. Shorter Definition Of EmergenceCollective Intelligence

37. Stop The InsanityEmergence is notabout the timing of decisions.Emergence is about harnessing the incredible intelligence that lies dormant in our organizations – trapped in antiquated and outdated philosophies of management.Emergence is about enabling and empowering the people to achieve a greater purpose.

38. AgendaIntroductionWhat is Architecture?What is an Architect?What is Emergence?How Can We Foster Emergence?How To Maximize Value DeliveryConclusion and Q&A

39. Simple Principles Of EmergenceThe source of emergence is the interaction among agents who mutually affect each other.Attend to relationships characterized by mutuality among people, among teams and among companies in order for novelty to emerge.Small change can lead to large effects.Seek to lead change through many small experiments, which search the landscape of possibilities.Emergence is certain, but there is no certainty of what it will be.Create conditions for constructive emergence rather than to plan a strategic goal in detail. Evolve solutions, don’t design them.Greater diversity of agents in a system leads to richer emergent patterns.Seek a diversity of people, cultures, expertise, ages, personalities, gender so that when they interact in teams, creativity has the potential of being enhanced.- Complexity: life at the edge of chaos, Roger LewinSimple to Describe. Hard to Accept!

40. How Do We Foster Emergence?Accept that there is in fact a “Complex Adaptive System” Provide leadership in the design of the systemManage the system – not the peopleQuit thinking linearly Passionately apply a “new management model”

41. A New Model For ManagementManagement Needs ChangingNew model leveraging “Complexity Theory”Acknowledge that organizations are really networks6 Important Practices (“views”)Energize PeopleEmpower TeamsAlign ConstraintsDevelop CompetenceGrow StructureImprove Everything- Management 3.0, JurgenApello

42. View #1: Energize PeoplePeople are the most important parts of an organization and managers must do all they can to keep people active, creative, and motivated.Complexity Versus Lean, JurgenApello

43. “http://www.slideshare.net/jurgenappelo/complexity-versus-leanView #2: Empower TeamsTeams can self-organize, and this requires empowerment, authorization, and trust from management.Complexity Versus Lean, JurgenApello

44. “http://www.slideshare.net/jurgenappelo/complexity-versus-leanView #3: Align ConstraintsSelf-organization can lead to anything, and it’s therefore necessary to protect people and shared resources, and to give people a clear purpose and defined goals.Complexity Versus Lean, JurgenApello

45. “http://www.slideshare.net/jurgenappelo/complexity-versus-leanView #4: Develop CompetenceTeams cannot achieve these goals if team members aren’t capable enough, and managers must therefore contribute to the development of competence.Complexity Versus Lean, JurgenApello

46. “http://www.slideshare.net/jurgenappelo/complexity-versus-leanView #5: Grow StructureMany teams operate within the context of a complex organization, and thus it is important to consider structures that enhance communication .Complexity Versus Lean, JurgenApello

47. “http://www.slideshare.net/jurgenappelo/complexity-versus-leanView #6: Improve EverythingPeople, teams, and organizations need to improve continuously to defer failure for as long as possible.Complexity Versus Lean, JurgenApello

48. “http://www.slideshare.net/jurgenappelo/complexity-versus-lean“Nurture” EmergencePassionately Apply a NewManagement Model That1. Establishes Clear Context, and2. Energizes and Empowers The People

49. AgendaIntroductionWhat is Architecture?What is an Architect?What is Emergence?How Can We Foster Emergence?How To Maximize Value DeliveryConclusion and Q&A

50. Recipe For Value DeliveryCreate The OpportunityDesign The “System”Nurture EmergenceApply A New Management ModelExecution LoopDo Just Enough PlanningCreate The ContextEstablish The Guard RailsLubricate ExecutionGet On With ItSense and RespondRepeat

51. What Planning Do We Need To Do?Plan the right things at the right timeJust enough to “accomplish the goal” by:Establishing Clear ContextEnergizes and Empowering The PeopleTypically Focus On These Outcomes:Capture the stakeholder perspectives that affect the priority aspects of the designEmbrace change and to reduce the cost of solving problemsCreate a shared vision across the team and stakeholdersSmooth the decision-making process

52. Emergence and DecisionsMaking decisions sets the context required to enable emergenceThere are often multiple “scales” involvedDecision cycles are different at different scalesEvery scale can operate in an emergent fashionStrategy/Vision/Mission -> Enterprise -> ApplicationVertical feedback loops are required

53. How Much is “Enough”?“In these systems agents residing on one scale start producing behavior that lies one scale above them.”- Emergence, Steven Johnson

54. Establishing ContextContext for Energizing and EmpoweringContext for SafetyContext for Efficiency

55. Context For Energizing / EmpoweringA compelling description of the goalsSense of urgency and purposeAlignment with vision, mission, and strategy

56. Empowerment“If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.”- Antoine de Saint Exupéry

57. Emergence and UncertaintyEmergence involves some degree of uncertaintyEmpowerment introduces independenceDoes the Heisenberg Uncertainty Principle play?

58. Emergence and Leadership"Good leaders create the conditions for constructive emergence rather than trying to plan every strategic goal in detail.“Step Back & Emerge, Dr Paul Thomas

59. http://businessdoctorme.blogspot.com/2010/07/step-back-emerge.htmlContext For SafetyClarity of enterprise strategy and objectives“Good enough” description of the enterprise target architectureEssential architectural requirements and constraints

60. Emergence and BoundariesBoundaries Establish FreedomIs This contrary to popular belief? Freedom is Required For Speed and Agility

61. Context for EfficiencyNot all decisions are created equalSome things are harder to change than othersSome decisions are more “core” or “fundamental” and get the ball rollingDon’t forget “strategic” vs “tactical”Evolutionary architecture and emergent design: Evolutionary architecturehttp://public.dhe.ibm.com/software/dw/java/j-eaed10-pdf.pdf

62. AgendaIntroductionWhat is Architecture?What is an Architect?What is Emergence?How Can We Foster Emergence?How To Maximize Value DeliveryConclusion and Q&A

63. Can We Let Go?Understanding emergence is about: giving up control; being more tolerant of that exploratory phase where the rules don't make sense and where few goals have been clearly defined; letting the system govern itself as much as possible; and letting it learn from its own footprints.http://www.cleanlanguage.co.uk/articles/articles/216/1/Self-Organising-Systems-Findhorn/Page1.htmlNurture The FreakarFke

64. The World Is A Fractal“Once you start thinking of strategy as an emergent phenomenon, you realize that we have often attacked the wrong end of the problem. Strategists and senior executives have too often worked on "the strategy," rather than on the preconditions that could give rise to strategy innovation. In essence, they've been trying to design complex, multicell organisms, rather than trying to understand and create the conditions from which such organisms will emerge.”- “Strategy Innovation and the Quest for Value, Gary Hamel (http://www.strategos.com/articles/questforvalue.htm)

65. Q&A

66. AgendaIntroductionWhat is Architecture?What is an Architect?What is Emergence?How Can We Foster Emergence?How To Maximize Value DeliveryConclusion and Q&AAppendix (Gartner On Emergent Architecture)

67. Gartners “New” Emergent EA“Enterprise architects must adopt a new style of enterprise architecture (EA) to respond to the growing variety and complexity in markets, economies, nations, networks and companies, according to Gartner, Inc. Analysts advised companies to adopt ‘emergent architecture’, also known as middle-out EA and light EA, and set out definitions of the new approach.”Key Characteristics of the Emergent Approach“Summarised as ‘architect the lines, not the boxes’, which means managing the connections between different parts of the business rather than the actual parts of the business themselves.” “Models all relationships as interactions via some set of interfaces, which can be completely informal and manual – for example, sending handwritten invitations to a party via postal letters - to highly formal and automated, such as credit-card transactions across the Visa network.”Bruce Robertson, research vice president at GartnerGartner Press Release, Egham, UK, August 11, 2009http://www.gartner.com/it/page.jsp?id=1124112

68. Gartner Emergent EA Principles#1 Non-DeterministicNon-deterministic - In the past, enterprise architects applied centralized decision-making to design outcomes. Using emergent architecture, they instead must decentralize decision-making to enable innovation.I've heard about this "deterministic" EA practice. And I've also heard about unicorns. Every effective EA practice I've seen recognized its role as one of leadership - and context. One of the biggest drivers in business today is Agility - the need to respond rapidly to changing needs and opportunities. By definition, then, we are operating in a climate where the future is not pre-determined or predicted. As such, at some scale, the specific outcomes are obviously non-deterministic. But there's a huge risk of this property being abused (see Agile Is Not "Make It Up As You Go"). As a whole, any organization with an EA practice absolutely has some destination in mind... some target. It's our job to create context and to provide leadership that helps the organization translate that target into actionable goals... and to adapt its way to success.http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

69. Gartner Emergent EA Principles#2 Autonomous ActorsAutonomous actors - Enterprise architects can no longer control all aspects of architecture as they once did. They must now recognise the broader business ecosystem and devolve control to constituents.Again, the idea that an Enterprise Architect could ever "control all aspects of architecture" is a farce. The power of an organization always lies with those that are serving the organizations clients - the business units. Our role in EA is to serve those people on the front line and empower them to better meet the needs of their customers - and at the same time advance the organization as a whole towards its targets. This is a role of leadership - not control - and I wrote about it in Leadership - The Secret Sauce. Emphasizing the value of "collective intelligence," that post reminds us that we can "achieve outrageous levels of performance by harnessing the intellect and energy of the people." This also came up in Nurture The Freaks where we contemplated these words from Gary Hamel, "Going forward, no company will be able to afford to waste a single iota of human imagination and intellectual power."http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

70. Gartner Emergent EA Principles#3 Rule-Bound ActorsRule-bound actors - Where in the past enterprise architects provided detailed design specifications for all aspects of the EA, they must now define a minimal set of rules and enable choice. It's a reasonably well accepted principle that an EA practice should never make a decision (or set a constraint) that could be left to the business unit or development team. In fact, enabling choice and encouraging participation are important vehicles for gaining buy-in and goodwill (see Governance Without Goodwill Is Dead). This is yet another reminder of how we need to establish context by creating guard rails that keep the organization out of the ditch.http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

71. Gartner Emergent EA Principles#4 Goal-Oriented ActorsGoal-oriented actors - Previously, the only goals that mattered were the corporate goals but this has now shifted to each constituent acting in their own best interests.With the guard rails in place, responsibility for driving rests on the individual drivers, each in their own vehicle with both hands on the wheel.http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

72. Gartner Emergent EA Principles#5 Local InfluencesLocal Influences - Actors are influenced by local interactions and limited information. Feedback within their sphere of communication alters the behaviour of individuals. No individual actor has data about all of an emergent system. EA must increasingly coordinate.There is a massive amount of information flowing through the modern organization, and the majority of it originates and circulates right on the front line where the dynamic nature of today's agile organization demands a high degree of "in the heat of battle" decision-making. This suggests EA add value by encouraging a broad community that is willing and able to actively contribute to and consume a real-time, high-bandwidth stream of communication. It's not our job to assimilate it all and make decisions. Instead, as individual consumers at the trough of the information stream, we help drive the information out to those who need it the most. I don't really like the use of the word "coordinate" here. We're only coordinating in the indirect sense. Perhaps this role is some combination of the Connector, Maven, and Salesman roles described by Malcolm Gladwell in The Tipping Point.http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

73. Gartner Emergent EA Principles#6 Dynamic or Adaptive SystemsDynamic or Adaptive Systems - The system (the individual actors as well as the environment) changes over time. EA must design emergent systems [that] sense and respond to changes in their environment.This is one of the most important functions of the Enterprise Architecture discipline in a modern organization. We have a responsibility to bring a "systems thinking" perspective to the table and influence the design of flexible and adaptive systems - systems that have the ability to learn from and respond to their experience. When talking about systems, here, it's critical that we deliberately design this adaptive nature into our products AND our organizations. I'm happy to see that Gartner is beginning to recognize organizations as a type of "complex adaptive system".http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

74. Gartner Emergent EA Principles#7 Resource-Constrained EnvironmentResource-Constrained Environment - An environment of abundance does not enable emergence; rather, the scarcity of resources drives emergence.Most of us are operating under the influence of unprecedented economic conditions, and these times demand that we become more creative. In fact, creative isn't really the right word. To respond to the reality of the corporate climate of today (and tomorrow), we're going to need to be "clever" - adroit, nimble, resourceful, and mentally quick. The organizations that thrive in the future will be those that respond today by building a sustainable system of Agile capabilities that maximize the contribution of every associate.http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html