Breathing Data, Competing
on Code                    ^mostly this


         Venkatesh Rao
        ribbonfarm.com
      vgr@ribbonfarm.com

       February 22, 2012
competing on coders
band-aid
people over process
software is eating everything

                         -- Marc Andreessen



every company is now a software company

                          -- David Kirkpatrick
most software is crap
Release!

     Expand                   Refactor, lower
  Feature Set,                entropy, pay off
increase entropy               technical debt




      Version n                 Version n+1
99%




1%
software is eating everything
#OccupySoftware!
Software is eating the world and every industry is being
dissolved, disaggregated, disrupted and de-engineered in
an acid rain downpour of Big Data and Free Agency
coming down from CLOUDS that are forming via DATA
evaporation from a world that is an ocean of sensors,
                      and Windows Phone
iPhone and Android apps and untamed social streams of
                     ^
Facebook life, and when the Internet of Things happens
and it is all connected up into a single giant software
application a Chinese virus will mutate into a self-
improving AI and the Internet will become SENTIENT and
TAKE OVER ALL THE DRONES the whole pile of crap will
collapse under its own weight and KILL US ALL.
technical debt is growing faster
worldwide than it is being paid off
Everyone knows that debugging is twice as
hard as writing a program in the first place.

                            -- Brian Kernighan
Action Item #1
            listen to this talk
        by Alan Kay immediately

   http://tele-task.de/archive/lecture/overview/5819/
people over process

this principle is not
generally wise outside of
software engineering
one size does not fit all
enlightened governance with a
wise policy of exceptions and
customization of processes to
specific situations
rational governance with a wise
policy of exception management
and customization of processes
to specific situations
     principles of forgiveness
great engineers ask for forgiveness,
not permission

   This is a meaningless bromide if you
   always forgive or never forgive

         principles of forgiveness are about when
         you forgive

              if you get them wrong something far
              worse than rule-breaking will happen
perfect rule following
Work-to-rule: an industrial action in which employees do no more than
the minimum required by the rules of their contract, and follow safety
or other regulations to the letter to cause a slowdown rather than to
serve their purpose.


http://en.wikipedia.org/wiki/Work-to-rule
sprint tempo          burndown charts
daily standups        code commenting
test-driven           forking norms
development           working code over
  who decides when it is okay to
usability testing     documentation
standrds              communication norms
  break which rules and why?
version control       dflss certifications
retrospectives        code review
user story semantics  meeting rituals
story points tracking patent applications
requirements          open source policies
pair programming      spiral development
governance




culture

narrative | archetypes
what you forgive depends on
where you are in the story
Gustav Freytag, 1863
Venkatesh Rao, 2011
good judgment emerges from a rich
sense of narrative situation awareness,
not formulaic applications of narrative
structure theory
narrative | archetypes
what you forgive depends on
who you are dealing with
tinker, tailor soldier, sailor
richman, poorman, beggarman, thief
        after software eats
            everything



     hacker, hustler, coder, admin
     moneyman, dataman, uxman, thief
the best guide to software archetypes can
be found within software culture itself
Contemporary culture is a two-tiered system, like the
Morlocks and the Eloi in H.G. Wells's The Time
Machine, except that it's been turned upside down. In
The Time Machine the Eloi were an effete upper class,
supported by lots of subterranean Morlocks who kept the
technological wheels turning. But in our world it's the
other way round. The Morlocks are in the minority, and
they are running the show, because they understand
how everything works.

-- Neal Stephenson, In the Beginning was the Command
                                                 Line
dwarves seek beauty

     don’t force them to do ugly
a good wizard makes you feel
in charge

       a good wizard-wrangler
       realizes he is not
ninjas understand your ideas
better than you do


           defer to their judgment
samurai prize honor and loyalty


          never stop fighting
          for them
elves understand magical realities
that are invisible to you


                   do not disturb
the exceptional nature of software
culture arises from the fact that software
engineers build their own tools

software culture is guild-like
Action Item #2


                 read this book
                 immediately

                 http://tempobook.com
// end flattery and pandering

// begin evil manipulation
data
When art critics get together they talk
about form and structure and
meaning. When artists get together
they talk about where you can buy
cheap turpentine.
                         -Pablo Picasso
the turpentine effect
form and structure and meaning
Action Item #3
             read this book
             immediately
“Thin
Simplification”
the kind of knowledge required in such
endeavors is not deductive knowledge
from first principles but…mētis…the kind
of knowledge that can be acquired only
by long practice at similar but not
identical tasks

      -- James Scott, Seeing Like a State
great programmers form a
subculture based on mētis


                   respect it
but not too much
self-congratulation is not strategy


   mētis    insularity   complacency
...organizations which design systems
... are constrained to produce designs
which are copies of the communication
structures of these organizations.

               -- Melvin Conway, 1968
structure follows strategy

                -- Alfred Chandler, 1962
when software eats everything
and every company is a software
company strategy follows data
managing mētis means
mastering data
breathing data makes better strategy
better strategy makes better structure
better structure means richer mētis
richer mētis means better code
better code means better company
better company lets you eat rather than be eaten
#1 http://tele-task.de/archive/lecture/overview/5819/




#2                      #3

Breathing Data, Competing on Code

  • 1.
    Breathing Data, Competing onCode ^mostly this Venkatesh Rao ribbonfarm.com vgr@ribbonfarm.com February 22, 2012
  • 2.
  • 3.
  • 4.
  • 5.
    software is eatingeverything -- Marc Andreessen every company is now a software company -- David Kirkpatrick
  • 6.
  • 7.
    Release! Expand Refactor, lower Feature Set, entropy, pay off increase entropy technical debt Version n Version n+1
  • 8.
  • 9.
  • 10.
  • 11.
    Software is eatingthe world and every industry is being dissolved, disaggregated, disrupted and de-engineered in an acid rain downpour of Big Data and Free Agency coming down from CLOUDS that are forming via DATA evaporation from a world that is an ocean of sensors, and Windows Phone iPhone and Android apps and untamed social streams of ^ Facebook life, and when the Internet of Things happens and it is all connected up into a single giant software application a Chinese virus will mutate into a self- improving AI and the Internet will become SENTIENT and TAKE OVER ALL THE DRONES the whole pile of crap will collapse under its own weight and KILL US ALL.
  • 12.
    technical debt isgrowing faster worldwide than it is being paid off
  • 13.
    Everyone knows thatdebugging is twice as hard as writing a program in the first place. -- Brian Kernighan
  • 14.
    Action Item #1 listen to this talk by Alan Kay immediately http://tele-task.de/archive/lecture/overview/5819/
  • 15.
    people over process thisprinciple is not generally wise outside of software engineering
  • 16.
    one size doesnot fit all
  • 17.
    enlightened governance witha wise policy of exceptions and customization of processes to specific situations
  • 18.
    rational governance witha wise policy of exception management and customization of processes to specific situations principles of forgiveness
  • 19.
    great engineers askfor forgiveness, not permission This is a meaningless bromide if you always forgive or never forgive principles of forgiveness are about when you forgive if you get them wrong something far worse than rule-breaking will happen
  • 20.
    perfect rule following Work-to-rule:an industrial action in which employees do no more than the minimum required by the rules of their contract, and follow safety or other regulations to the letter to cause a slowdown rather than to serve their purpose. http://en.wikipedia.org/wiki/Work-to-rule
  • 21.
    sprint tempo burndown charts daily standups code commenting test-driven forking norms development working code over who decides when it is okay to usability testing documentation standrds communication norms break which rules and why? version control dflss certifications retrospectives code review user story semantics meeting rituals story points tracking patent applications requirements open source policies pair programming spiral development
  • 22.
  • 23.
    what you forgivedepends on where you are in the story
  • 24.
  • 25.
  • 27.
    good judgment emergesfrom a rich sense of narrative situation awareness, not formulaic applications of narrative structure theory
  • 28.
  • 29.
    what you forgivedepends on who you are dealing with
  • 30.
    tinker, tailor soldier,sailor richman, poorman, beggarman, thief after software eats everything hacker, hustler, coder, admin moneyman, dataman, uxman, thief
  • 31.
    the best guideto software archetypes can be found within software culture itself
  • 32.
    Contemporary culture isa two-tiered system, like the Morlocks and the Eloi in H.G. Wells's The Time Machine, except that it's been turned upside down. In The Time Machine the Eloi were an effete upper class, supported by lots of subterranean Morlocks who kept the technological wheels turning. But in our world it's the other way round. The Morlocks are in the minority, and they are running the show, because they understand how everything works. -- Neal Stephenson, In the Beginning was the Command Line
  • 33.
    dwarves seek beauty don’t force them to do ugly
  • 34.
    a good wizardmakes you feel in charge a good wizard-wrangler realizes he is not
  • 35.
    ninjas understand yourideas better than you do defer to their judgment
  • 36.
    samurai prize honorand loyalty never stop fighting for them
  • 37.
    elves understand magicalrealities that are invisible to you do not disturb
  • 38.
    the exceptional natureof software culture arises from the fact that software engineers build their own tools software culture is guild-like
  • 39.
    Action Item #2 read this book immediately http://tempobook.com
  • 40.
    // end flatteryand pandering // begin evil manipulation
  • 41.
  • 42.
    When art criticsget together they talk about form and structure and meaning. When artists get together they talk about where you can buy cheap turpentine. -Pablo Picasso
  • 43.
  • 44.
    form and structureand meaning
  • 45.
    Action Item #3 read this book immediately
  • 46.
  • 47.
    the kind ofknowledge required in such endeavors is not deductive knowledge from first principles but…mētis…the kind of knowledge that can be acquired only by long practice at similar but not identical tasks -- James Scott, Seeing Like a State
  • 48.
    great programmers forma subculture based on mētis respect it
  • 49.
  • 50.
    self-congratulation is notstrategy mētis insularity complacency
  • 51.
    ...organizations which designsystems ... are constrained to produce designs which are copies of the communication structures of these organizations. -- Melvin Conway, 1968
  • 52.
    structure follows strategy -- Alfred Chandler, 1962
  • 53.
    when software eatseverything and every company is a software company strategy follows data
  • 54.
  • 55.
    breathing data makesbetter strategy better strategy makes better structure better structure means richer mētis richer mētis means better code better code means better company better company lets you eat rather than be eaten
  • 56.

Editor's Notes

  • #2 Scale: how America competed on precision and interchangeability in manufacturing in the late 1800s. Created a new kind of industrial worker and a new kind of ingenious designer-entrepreneur. Colt, Singer, Eli Whitney are some of the famous names from that era. It reached a peak with Taylor in 1919. It created a class hierarchy: between interchangeable and non-interchangeable workers. Craftsmen turned into Design Engineers and Technicians.
  • #3 Homeless person looking for CTO phenomenon. There are interchangeable and non-interchangeable programmers.
  • #5 Put a pin in this thought that everybody provides lip service to.
  • #6 Similar quotes: quash one bug, two new ones are created. Reading code is harder than writing code.
  • #7 In manufacturing, if a process isn’t being hacked and improvised constantly, it will actually not work at all. Same holds in s/w but 10x more.
  • #8 It comes down to culture being about 10x more importance than governance, and I am only going to talk about a small part of culture today: narrative and archetypes.
  • #9 Every age has its archetypes of work. This one is apparently
  • #10 Artists are process improvement mavens, not process bureaucrats. Which means they often don’t care about their subject. If an artist finds no subject, he will paint his cup of brushes or turpentine bottle.
  • #11 Torvalds: Git, Knuth: TeX, Ritchie/Thompson: Unix/C, Joel Spolsky,
  • #12 What are we building FOR? How should it be structured in relation to what it deals with? What does the stuff it processes MEAN? Many s/w development philosophies assume that software is being built for the stated purpose in whatever management scheme you use. NO, software engineering is like a forest that can yield solutions to your goals as one of the outcomes, but there is a good chance it will be a minor outcome. This is like a machine tool where every machined part results in the machine retooling itself to be better.
  • #13 If you are managing a development effort to produce a spreadsheet say, and you assume that the jungle of code and programmers is there just to serve that one purpose, you are going to destroy things. It will produce that spreadsheet, but don’t make it do so too efficiently. It also does other things you may not be aware of. Explore new programming models, produce new patterns and tools.