Habitual thinking - Patterns for IT disappointment and failure
Presentation in 3 parts that uncovers the problems with conventional thinking in IT solutions and how to break the habit and thereby creating the foundation IT-driven growth. Finally, we share lessons learned from implementation of our suggestions in connection with the Danish Electronic Land Registration project.
IntroFocus on thatyoucanchangeyour approach 3 parts in the presentation:1: Somethingabouthabitualthinking and whywe must thinkdifferently.Extract of whatweusuallymeet in real life.2: Whatwecan do about it. Manythingswhichweaddress. Focus on 2 technicalareas.Low hangingfruit3: How wesolved the tasks in ETL
Horror Scenario:Earth goes under, the Mayan calendarendsMost important: economiccrisis and it does not seem to getbetter
There must thinkdifferentlyWeareforced to do somethingdifferentthanweusually doFinancially problem is not only small companies, including largeMoney is no longer hanging on trees
Spørgsmål som alle IT viksomheder bør stille sig selv og have svar påHvor mange kan ærligt svare ærligt til alt ovenstående?Kan også bruges som eksempel på dårlig (ledende) spørge teknik
From Men in Black - erasesyourmemory X number of hours / days back
Manyprojects start from scratch every time. Reinvent the wheel in everyprojectThe problem with a projectorganizationFailure to follow up
Examples from reallifeBusiness wishes have a tendency to beconvertedintotechnicalrequirementsWeforget the real needs and spendtoomuch time on techniquepettiness
For many: Technologies = SolutionsPeople oftenthink of technologiesbeforethey have made clear what problem to solveTechnology becomes the same as the solution: The solution must be service-oriented so weneed a bus service and it must be WebSphereToo manytechnologiesrequiretoomany specialistsToo manytechnologiesmeansgreaterprojectrisk
Explanation of whatcausescanbeGartner Group is helping to do things for silverbullets. Good to learn from others, but oneshould not turn to it just because the gardenersays it.Gartner says Service Bus, so everyone must have it.The problem is that management chooses the technical solutions, not the architects or developers
People are still trying to find the toolthatcan handle everythingOne sizefits all
This meansthatwemake the worldtoocomplexIt costs and contributes to everythingtakingtoo long
There is nothing new happeningPeople do the same every timePeople dare not pursue new ways of doingthingsPeople dare not tryanything newDo the same eachyear, although all the participants aredead
Public sector view of the world: “We need more money”7 habbits of highly effective people: The 7.habbit“Sharpen the saw”. “We’re so busy driving that we haven’t got time to stop and fill up the car with gasoline”
You have to be open for new ideas
But whatif the real answer is wirelessnetworking?Thatansweryouwillonlygetwhenyou have an expert at your fingertipsOutsourcing is the same as cutting the engineer from the smith and thenyouloose out on all the goodideasWhat do I need?The requirementsshouldspecify the need, not the solution
Ways to ask a questionMany times the answer is implicit in the questionHere youcanonlyanswerwrong. No matterifyousayyes or no, youare a violent man.Remember to relate to the questions on the previous
Detervigtigt at heletidenudfordre sig selvSørge for at stille de rigtigespørgsmål
People onlychangestheirhabitualthinkingifthey’removed out of theircomfort zone and shaken upWhatproject doyouremember? The oneswhereyou have succeedagainst all oddsAs seen in manycompanies: Whyareyouallowed to bypassstrictprocesses whensomething is veryimportant , but you’re never allowedwhenyou have time and thingsaren’t as important
We forget to look at what it is were doingWe’re good at asking What and HowBut we forget to ask WHY
Low pricesreducesfocus on eliminatingwasteWhichtypicallyresults in a more messy system Whichagainresults in more expensivemaintenance
This is commonknowledge
Manychangesrequire major externalinfluences - desperation is often a muchgreatermotivatorthan inspirationWith everythingthat is happening nowthenweshouldbeready to do thingsdifferently
In the first part weheardabout the challenges.Therearemany solutions and wecan not go intothem all.Wewilltake a look at 2 solutions thatwebelievecancreate fast results with leasteffort+ wecankeep the changeswithinIT'sownenvironment (involvement of the business is minimal)
SCRUM and LEAN as theyareappliedtoday cover onlysoftvaluesThe worstthey did for LEAN was to give it a name – becausethen management expect it to come in a boxthatyoucaninstallWith XP and SCRUM/LEAN there’s still a tendency to have a productview of the worldImplementing agile/leantakes longer and it’s a processthatinvolves the entirecompany
Webuild on unstableground – due to a weak and fragmented system landscapeHow many put a lifetime up for it whencommissioning a new system?Knowledge transfer/ accumulation of experience, they all die together with the project
Alignment = Business AlignmentEfficacy: IT developmentefficacy
The layersaretechnically de-coupled but not in terms of data or temporal.Eachlayer has a technical integration layerthatmakes it harder to understand the correlationsDebugging is made more difficultSingle point of failureLatency is high and scalability is low
En samlet model for virksomheden er ikke længere muligt.Verden er blevet langt mere kompleks end tidligereI dag skal man indsamle information fra langt flere datakilder end tidligereEn samlet model betyder langt større sårbarhed
Alsoknown as a canonical model in SOA terms
It is not possible to create a context-free modelDoing so youwillonlygetsomethingthatdoes not work in anycontextToo manyunintended side effectsExample. Life insurance with lifecoverage. My companyassuresme, mywife and childrenarebeneficiaries. Who is the customer? It depends on what point in time I look at it.
Use Cases: There is a difference between the static and the dynamic part of a systemThe dynamic part contributes to provides an precisecontext
Loose coupling - data and temporalWeb shop can run without the otherbeing online
Keeptrack of where the different system in your landscape isIn the "Invest and differentiate" Time to market is important - makedemands on IT efficacy
Cleanup of the System Landscape cantake time, herearesomeapproachesthatarecheap and easy ...
ETL = Electronic LandRegistration (Elektronisk TingLysning in Danish)
Therearemanysimilarities with the previousarchitecturewetalkedaboutwas not a goodidea ...
Well-meaningarchitects, data modelers, analystsrepresent the initial data / domain model afterwhich it willbehanded over to developers whoimplement it by hand.After a fewiterations the codedoesn’t in anywayrepresent the original models. The original model becomes obsolete and willbethrown out. Ie. Documentation is nowonly in code.Analysts and business experts do not understand the code and canthus not beincluded in a develpmentpartnership in a goodway
Denmark has a long tradition of workingefficiently and automate tedious and repetitivetasksWe have a long tradition of wanting to work for commongoodWeshould do the same in IT tooThe positive side effect of highwages and highqualityIf you send it willbe, it is difficult to automate afterwardsThe offshoring partner gets all the possibilities of becoming more efficient and earnmoneyWearecheatingourselves of the benefitAgain, the goodideasemergewhen the smithwork with the engineer
Everythingrevolvesaround the model, in an agile (and iterative setup) that is run throughTigerTeamsTRIMM (code generator) and therebyproducing bi-products such as Java code with Hibernate / JPA annotations, WSDL (was not used at ETL, but is an option youcanselect), SQL and automated test code (JUnit-based) that tests the codethat has beenproduced and thusensuresthat the database codeworkswithout data loss.
Note:Wasn’tused at ETL
Note: Wasn’tused at ETL
Severalweeks of programmingcan save a fewdays of forethoughtWith the short deadline on ETL, a laterfreeze date ensured a muchbettertechnical and business solution
What is important in thisprocess is to ensure a stable interface between the generatedschematiccode and the interestingcodewhich in one form or otherwiseuse the schematiccode.This is typically not difficult.
Higherlevel of abstraction:Closer to the business worldviewLesstechnicalfocus and more focus on solving business challengesTechnical decisions canbepostponedThe model decorated with information for generatorsOne candescribethat "here must behistory" and thenlater on decidehowPoint of No Return canbepostponedChanges do not costextrathough the code base getsbiggerDocumentation is alwaysupdate to dateThe model is keptfresh, since all code is generatedbased on itHigherQuality and UniformityEverythingfollows the same patternsCode standards arekeptEithereverythingworks, or everything has the same mistakes/errorsRefactoring is both faster and cheaperGeneral changescostlessbecause it canberegenerated, ie. Doesn’trequirerewritingEven basicchangescanbe made fairlylateIt does not costmuch to makemistakes.Youcantryseveralthings at onceYouget abetter systemShort developmentcycleYoucan sit with your business and change the model in the morning and have somethingrunning the dayafter, so theycan test it sooner
Final points:Wewant to be Lean and AgileXP, SCRUM, etc. are the softer points but takes longerA simplertechnicalarchitecturecanhelp to makeyou Agile and Lean, but canalso stand on itsownAutomation canbeimplementedbottom up. And is againbetter on a simple technical platformEverythingcontributes to makingus more efficientPostulate: Wecan NOT be more effectivethanourtechnical platform and the degree of automation allowsus to use
Habitual thinking - Patterns for IT disappointment and failure
Habitual thinking Patterns for IT disappointment and failureHenrik Wivel & Jeppe Cramon
Corporate growth slowed dueto lack of funding Berlingske Finans6 out of 10 public projectsgoing over time and budget Version2
How well is your company doing?• Are your IT projects delivered on time, on budget and with the right functionality?• Are your IT systems running smoothly and stable?• Does your system landscape contain more applications performing the same or similar tasks?• Do you Outsource / Offshore the right tasks (or are you doing the right tasks in-house)?
How well is your company doing?• Are your IT projects aligned with business needs?• Is the business actively involved in IT projects and investments?• Do you balance the entire organizations needs with the individual business needs in terms of architecture and infrastructure?
PRODUCT VIEWWe need a combined Implement Siebel CRM in thecustomer view Call centerWe need to better Buy and install HP Qualitymanage our CenterrequirementsWe need a Service Build a solution using WebOrienteret Architecture services in Oracle Service Bus / SeeBeyond / ...
Typical management thinking• We choose the tools and architecture that Gartner recommends• More resources on a project = The sooner we will be finished• 100 foreign developers are better than 10 local, even if the total price is the same
All this leads to • High complexity and a fragmented system landscape • Strong need for product specialists who can not always participate across projects • Danger of hiring habitual thinkers rather than technical domain experts This creates the need for many resourceson projects and therefore result in high costs
Same procedure as last year?Same procedure as every year!!!
Because!!!• “We haven’t got time to look at new technologies”• “It requires more money and time”• “We have tried it before and failed”• “We realize we have challenges”• ”We _______________________________” (Insert an excuse)
Open for new ideas? Poul, you idiot, whats that!?You should only bring things we can eat!
If it takes 1 man 10 days to dig a telephone cable , how many men do I need if I want it done in 2 days? 5 men in Denmark 10 men in Indien. Half time and even half priceHow about using: Wireless network?
This is how we often ask• Where can I get 200 developers for my project?• Where can I get them cheapest?• How do I scale my technical platform to 200 developers?• How do I manage large teams over a long project cycle?
This is how we should ask• Why do I need 200 developers?• How would I do if I "only" had: – 30 developers – 3 months and not 3 years for the project• Do I need distributed teams?
Is the solution Offshoring? • 10 techinical domain experts in Denmark at $150 an hour = $1500 an hour • 100 developer in China/India at $30 an hour = $3000 an hourIt’s a bit like bying app on the Apple AppStore They only cost $1
We need a strong foundation for development 11% 7% Highly aligned ”Alignment Trap” ”IT enabled Growth” +13 +35 Source: Bain Analysis -14 -6Alignment 74% 8% ”Maintenance Zone” ”Well-Oiled IT” +0 +11 Less aligned -2 -15 Less effective Effciency Highly effective % of the 504 Combined yearly growth- respondents % difference compared to the overall averages IT spending rate over a 3 year period
A strong foundation is!• Simple architecture• Assignments and data owned by one system• Automation• Tests
NOT a Simple architecture Client Client Client Client Proces Service Proces Service Remote Facade Application Service Activity Activity Service Service Domain Domain Object Object Data Access Layer Data Data Data Data Service Service Service Service Data Storage Data Data Data Storage Storage Storage
A big unified model is every architects pibe dream But as they say
ONE MODEL TO RULE THEM ALL ONE MODEL TO FIND THEM ONE MODEL TO BRING THEM ALL AND IN THE DARKNESS BIND THEM
Still too big a model Online Ordering System Product Pricing Description Unit Price Promotional Price Promotion End Date Stock Keeping Unit (SKU) Quantity On Hand (QOH) Location Code InventoryCustomer Price Quantity Ordered Sales
There’s always more than onemodel at play in any big project But when code based on multiple models is combined
… then the code becomes filled with errors, unreliable and difficult to understand Communication between team members becomes confusing. It is often unclear in what context a model should NOT be used!
Smaller models and ”service reuse” Enterprise Pricing Service Product SKU Unit Price Promotional Price … Web Shop Product SKU Description Pricing Price Quantity OrderedCustomers … Inventory Service (SAP) Product SKU Description QOH Location Code … Inventory Sales
Now even better – with an ESB Enterprise Watch out! Pricing Service Its so easy to see a Product product as the SKU Unit Price solution to all your Promotional Price … problems! Web Shop Product SKU Description Pricing Price Quantity OrderedCustomers … Inventory Service (SAP) Product SKU Description QOH Location Code Enterprise … Inventory Service Bus Sales Before we had 1 problem. With the ESB we now have 2 problemes
Business focus is still lacking Processes and state changes are important components
A much better recipy Enterprise Pricing Service Product SKU New SKU Unit Price Promotional Price Event … Web Shop Product New SKU Price SKU Description New SKU Event Pricing Price Price Quantity Ordered EventCustomers … Inventory Service (SAP) Product New SKU SKU Description Event QOH Location Code … Inventory New SKU Message Bus Event Sales
What is a strong foundation?• Simple architecture• Assignments and data owned by one system• Automation• Tests
In brief…• Clean up the system landscape ...• Remove redundant systems• Remove systems that no longer have a reasonable cost-benefit• Modernize systems which can still be make a difference• Build new systems that can differentiate you from the rest• ... And do it based on Neil Nickolaisens Purpose Alignment Model
Neil Nickolaisen’s Purpose Alignment Model Invest High Partnership and Differentiate MarketDifferentiation Grade Who cares? Keep on a par with Low competitors Low High Criticality
What is a strong foundation?• Simple architecture• Assignments and data owned by one system• Automation• Tests
Automation• Continuous Integration – The code is automatically built and necessary tests are conducted for verification• Setting up environments – Server setup, firewalls, deployment• Integration testing• there is much more to get here ...
Electronic LandregistrationBackground Information on Electronic Landregistration for those who have not heard about it ...
ETL – Suggested Architecture Internal WebLogic Portal External Portal Portal Proces Service Proces Service IntegrationsOracle Service Bus Service Activity Activity Service Service Integrations Service Data Data Data Service Service Service Database
Suggested ETL Architecture• Advantages – Logically a reasonable decoupiling (Single Responsibility Principle) – Oracle makes good money on it• Disadvantages – Logical layers are turned into physical layer – High complexity – Difficult to find resources that can work with the OSB – Stability – Performance (both runtime and for developers) – Deployment
ETL – Simplified Architecture Internal Java applications External Portal PortalWebLogic Server Portal Facade Integrations Service Business Service Business Service Integrations Rich domain object Rich domain object Service Data Access Layer Database
Simplified ETL Architecture• Advantages – Physical and logical architecture are closer to each other – Low complexity – Stability – Easier to find resources that can code Java – Performance (both runtime and for developers) – Deployment• Disadvantages – The CV will not be boosted with Enterprise Service Bus – Oracle do not earn nearly as well ...
ETL - Automation Context• Extremely complex domain area• The rules are determined by Law (not logic)• Covering registrations there are several hundred years old• Complex logic required to automate law• Very short deadline for the third attempt to build ETL
Electronic Landregistration• Technical choices – Programming language: Java – Database: Oracle• First challenge – How do we integrate Java and Oracle database?
Sounds hard, but it’s quite easy Rule Language Meta Model + Rule Grammar (using Xtext) Text Editor +Data Language Meta Model Editor & IDE
Point of No Return, traditionallyCost ofRework Time Freeze V 1.0
Point of No Return, Model DrivenCost ofRework Time Freeze V 1.0
The end of writing tedious code by hand Model Generator Interesting code Frameworks (Written by hand) (Hibernate/JP Schematic code A, Entity Written by hand Framework) Libraries (f.eks. Java/.NET)
What do we get out of it?• Higher level of abstraction• Technical decisions can be postponed• "Point of No Return" can be postponed• Documentation is always update to date• Higher Quality and Uniformity• Refactoring is both faster and cheaper• Short development cycle
The real short versionWe can achieve the same with fewer resources We can achieve the same in less timeWe can achieve more with the the same resources and at the same time
Electronic Land registration Questions?http://tigerteam.dk @TigerTeamDK on Twitter