Thanks for coming, all the cool kids are next door in the RestfulErlang session…
Working on large enterprise programs, especially in the federal space, I was always struck by the location of the center of mass of effort. I had done corporate work and also been in startup situations and over and over I found myself contemplating these graphs. It seemed like we enterprise people were spending all of our time writing documentation while those web people were writing code and building toothpick bridges in their agile workspaces.
And that led me to this question: Why is corporate software so hard? Even for the simple stuff? We have access to the same technologies? So what’s the difference?
Of course the easy answer is “Bureaucracy.” And while that might be a bit of a cop out, let’s explore it a bit further.
Does your IT/IS department look anything like this? Architecture, development, requirements, program management, operations, maintenance… All of these separations of concerns sure do make things less pleasant and they take a tax on initiative. Are they necessary or just vestiges of the past? Or, do they need to apply to everything?
We started at the 30k foot level, now we are going to climb some more and look at some meta-drivers of the world we operate in.
Bureaucracy is much maligned, but it grew up to deal with the organizational challenges of the industrial age. Anditwas effective at organizing these guys to run …
…this. To the best of my knowledge there are no emergent factories. (though that may be changing with the makerbot).
And that led to this. While the individual’s experience of bureaucracy may feel like a life sucking vampire squid stuck to your face, the industrial revolution and its organizing methods have led to the first sustained growth in individual wealth in the history of humanity.
And here’s the real meta-context part… James Watt and his contemporaries thought they were inventing a form of locomotion, but they were unleashing a number of social gravitational forces.
The machines and industrialization needed labor – and lots of people left behind their dispersed agrarian lifestyles to concentrate in cities.
And that concentration made radically different kinds of political organization possible. In a sense, the machine led directly political movements around the world. Three out of four of our “big” political isms sprang directly from the modernism made possible by all those machines and the bureaucracies that sprang up to organize us around them. Those structures also informed the corporate structures that we operate in.
They weren’t just creating political systems either, they were creating entire newmodes of thought … For example, reductionism, the notion that a pure understanding could come from the decomposition of complexity into component parts came from our understanding of industrial age machine. Today these threads still run through our thinking – they are cognitive filters we probably aren’t even aware of – and probably interfere with our ability to understand complex dynamic systems where many of the “secrets” are in the time-variant patterns rather than fixed state or structure.
Which leads us back to the modernist enterprise that we are gradually replacing. I work for a very large systems integrator, and we’ve spent the last thirty plus years digitizing the corporate bureaucracy. From the first corporate payroll system to today we’ve been about the reimplementation of traditional bureaucracy in silicon and copper and every step along the way has been essentially hierarchical, reductionist, planned… Even today we to talk about the “industrialization” of IT – both in terms of what we are doing for our customers and how we do it. When we develop software for bureaucracies we do it bureaucratically.
The corporate enterprise, like any system, evolves. And it’s current level evolutionary maturity is probably about at the stage of the nematode; and we’ve been building its neural network. The nematode (or c. elegans) is widely studied because of its simple neural structures, structures that are focused on the very narrow regulation of homeostasis via on stimulus and response. The corporate digitalization of essentially neural processes are at a similar stage – with a focus on corporate homeostasis and the reimplementation of previously human-driven systems with productivity enhancing IT. But in many cases we haven’t changed the fundamental underlying organization or operation, we’ve just digitized them. (Note: there is some over simplification here. Collaboration tools etc. are beginning to make possible shadow organizations and networked modes of interaction, but the fundamental “big systems” like ERP’s are about digitizing former structures).
Some 30 years ago or so Leonard Kleinrock, Lawrence Roberts, Robert Kahn, and Vint Cerf thought they were building a computer network…
But the changes they have wrought are every bit is as dramatic as those caused by the steam engine and industrial revolution before it. The network age is ushering in a period of decentralized organizations…
strained national sovereignty
long tail enablement…
surprising forms of collective production…
new forms of political participation…
complex non-linear (social) systems…
non-state (or tacitly-state-sponsored) cyber conflict and on and on…
The network age doesn’t just extend industrialization, it leaves it behind. Modernism and reductionism may be in their final steep decline. They are being supplanted by a new kind of network empiricism that will exert its own gravitational force on our social and organizational fabric. George Bush, with his perfect manifestations of reductionism (“let’s roll!”, “You are with us or against us”) will probably be our last president of the modernist era. Reductionism remains intuitively appealing (and not just in the political sphere) but is increasingly out of step with our complex networked world.
This might seem kind of out there, but I just want to establish the context that the modern corporate enterprise operates in. We’ve all watched the U.S. Army try to figure out how to deal with decentralized non-state enemies and how it is challenging their deepest held notions of organization and process. Corporations face similar challenges. The world is changing and we need to take a hard look at how we organize to operate in it, and that includes how we build IT.
This leads us to a new, and more important, question.
For example. We are in the habit of thinking of agile methods as in contrast to the waterfall model. But agile software development isn’t a software development methodology, it’s the post-bureaucratic shop floor quality circle applied to software. It’s as much (or more) about post-bureaucratic organizational models as it is about software per se.
But back to this broader question of the impact of the network age on the enterprise. We are used to thinking of the corporate enterprise as nice neat packets of planning immersed in the broader emergent market. This separation of planned from emergent has worked well, and seems to have a natural balance that shifts with technology, organizational methods etc. But if there weren’t some natural limits to the expansion of planning and control Russia would still be a communist endeavor with a planned economy and General Electric would have stayed on an acquisition path until they owned everything.
As the corporate ecosystem becomes more connected the emergent challenges go beyond price. Increased corporate size and complexity, and more networked post-bureaucratic internal structures are making it very difficult for corporations to deal with all of the complexity they face. In fact, the bureaucratic model is further strained when the complexity exceeds the ability of the strategic corporate leadership to deal with in their decision making. (C is the complexity that management must deal with, C0 is the ability of a single person or small group to deal with complexity). When C is above C0 things come apart. The worst possible case is that C > C0 and you either don’t recognize it or refuse to accept it.
Hayak was talking about markets here, but as complexity becomes more and more part of our landscape it seems to be applicable to the internals of the corporate enterprise and the IT systems that enable it.
Implication of network age – these boundaries are getting fuzzier, and the internal contents of the planned entities are getting blurrier.
So, here’s the core point of Intentional Emergence, at least as it applies to organizational and IT architecture, design the edge to better “impedance match” with the surrounding ecosystem. This means giving it both planning (intentional) and emergent properties.
The result is a new aspirational enterprise software zeitgeist.
As technologists we have a natural tendency toward reductionist thought. Computers are rational after all. So it is our nature to deal with system failure by trying harder to control things. In technology this can often be counterproductive. More emphasis on governance for example can simply extend timelines and the risk absorbed with longer timelines is greater than the risk reduction you get from the governance. If we squeeze too hard, it just escapes...
So instead of control, let’s focus on facilitation. Gall’s Law gives us a hint about what to facilitate. Building on Gall’s law, since there is no guarantee that simple systems will work, the implication is to build lots of little systems and attempt to facilitate adoption of successful work. Or… “let a thousand flowers bloom.”
If that Gall’s law idea is true, then what we want is lots of starts with an ecosystem of support that leads to adoption. On the web this is enabled with cheap hosting, open source software, ready venture capital, ubiquitous web distribution… Is it possible to replicate a similar ecosystem inside a company in order to achieve a cognitive force multiplier?
Just to be clear, we aren’t talking about eliminating the head of the traditional IT project distribution, but we are talking about intentionally enabling a long tail. A single person with modest skills should be able to start the process of solving his or her own problems in code, and then if the project is adopted and useful it can grow back towards the head.
And how do we do this? Generativity is a nice term used by Jonathan Zittrain to describe a set of attributes that encourage wide contribution.
Leverage: How strongly a system or technology leverages a set of possible tasks, meaning it makes our difficult tasks easier.Adaptability: adaptability to a range of tasks. The ease with which it can be modified to broaden its range of usesEase of Mastery: Compare the mastery that is needed to use an airplane and to use a paperAccessibility / Transferability:The easier it is to obtain the technology, tools and information necessary to achieve mastery - and convey changes to others - the more generative a system is
They aren’t just (or even mostly) about technology. Lots of things from technology to policy and organization contribute. Even something as simple as how time tracking is implemented can have a big impact on how generative a system is. In technology, the “approachability” is critical – for example a runtime mapping api (e.g. openmaps or google maps) is much more generative than a disk with ESRI server code on it.
The thing is, innovation will always short to ground. Tell Carlos and Paul story (abridged version of this http://radar.oreilly.com/2010/03/apps-for-army-launches---the-h.html). Less generativity just means that the spark, when it happens, is bigger and further from where you want it.
Finding inspiration in unusual places. Scratch is inspirational because it has some interesting properties – the IP sharing approach is built in to the IDE (every project goes to a gallery), it is highly social in a way that supports co-learning, the tools are nicely sandboxed and designed for the capability curve…
With Carlos and Paul in mind, we proposed to the U.S. Army that instead of always building end use case systems, that they build a platform called the “battle command innovation platform”, whose end uses would not be known in advance, and provide tools, runtimes, and content to build usable systems in the field. We were looking to do something much more than just a PaaS runtime. There were strong community and content components to the vision that were targeted at the kind of users we expected to find in that “skunk works of one.”
The idea is to complete the long tail of IT in the enterprise. To make it one single continuum from “large systems” down to the simplest one person projects and provide each with tools, facilitation, and policy appropriate to their complexity and degree of mission criticality. From governance to facilitation. Create an Anode for innovation to jump to, that is on the reservation.
Ultimately all of these threads led to General Sorenson supporting Apps for Army and the follow on “Army Transformation Architecture.” Briefly tell the story.
This isn’t just for big federal enterprises. There are plenty of places that have a reasonable number of 4 and 5 sigma geeks who would build stuff if given permission, facilitation, and tools.
We started with a question. Let’s finish with a couple. I didn’t talk much about the technologies because I wanted to focus on the ideas. The ideas lead to an invisible hand that can operate in your environment.
Transcript of "Philly Emerging Tech 2011 Intentional Emergence"
Intentional Emergence<br />Innovation at the Edge<br />Jim Stogdill<br />
“It is no exaggeration to say that if we had had to rely on conscious central planning for the growth of our industrial system, it would never have reached the degree of differentiation, complexity, and flexibility it has attained. <br />…<br />Any further growth of its complexity, therefore, far from making central direction more necessary, makes it more important than ever that we should use a technique which does not depend on conscious control.”<br />Friedrich Hayak, The Road to Serfdom<br />
Internal characteristics are becoming “post-bureaucratic” with more permeable boundaries.<br />$<br />$<br />$<br />$<br />$<br />$<br />$<br />
Leverage: How strongly a system or technology leverages a set of possible tasks, meaning it makes our difficult tasks easier.<br />Adaptability: adaptability to a range of tasks. The ease with which it can be modified to broaden its range of uses<br />Ease of Mastery: Compare the mastery that is needed to use an airplane and to use a paper<br />Accessibility / Transferability:The easier it is to obtain the technology, tools and information necessary to achieve mastery - and convey changes to others - the more generative a system is <br />Make choices that increase generativity.<br />Generativity<br />
Some things that contribute to generativity<br />Open Data<br />Open Standards<br />Open Source Software<br />Open API’s<br />Runtime Platforms<br />Low hurdles<br />Simple rules<br />20% time<br />Variable Cost<br />Small world networks<br />Community<br />…<br />