1. Architecting Large Systems
Simon Farrell
Enterprise Architect
Colt Technology Services
simon.farrell@colt.net
http://www.slideshare.net/SimonFarrell1/architecting-large-systems

2. History
• Analyst/Programmer for British Telecom in late 1980s
• Mercury Comms / Cable & Wireless 1992 - 2008
– Developer, Oracle DBA, MIS manager, Data architect, System
architect, Enterprise architect, IT strategist
• UCL Chief Enterprise Architect 2008 – 2012
• Colt Technology Services 2012 -
• I have been involved in commissioning & implementing
more software systems than I care to remember:
- CRM
- Trouble Ticketing
- Fulfilment
- Billing
- Telecom network management
- Marketing campaign
management
- Data warehouses / MIS
- EAI
- Workgroup apps
- Web apps
- Mobile apps
- Too many technologies to
mention
2

3. Obligatory Architecture Slide
• Blah blah … Loosely Coupled … Resilient … ROI
… SOA … ESB … blah blah … Source Code
Control … Continuous Build … Agile … Waterfall
… Outsource … Cloud … blah blah … Metrics …
Javascript … Web Sockets … HTML 5 … MVC …
Patterns … Domain Specific Languages …
Behaviour Driven Development … Test Driven
Design … blah blah … Tiered … Asynchronous …
Load Balancing … Data Model … NOSQL …
Relational … XML … RDF … Open Source …
Standards …blah blah …Critical Path …
• Now let’s talk about real IT projects…
3: 1

4. REQUIREMENTS
4http://goo.gl/75vpR

5. Common Themes
• Every IT project starts out with:
– Some users (“the business”)
– Who have some needs
– That they (or someone else) think can be met by a (usually) new
computer system
• Often, they are wrong
– Many business problems are process problems
– Often, a mechanism already exists to solve the problem
– Aphorism #1: Not every nail needs a hammer
• Good analysts are required to keep users sane &
requirements in check
– Understand ROI, business cases, opportunity costs
– Look outside the immediate scope of the problem to assess
business impact, costs and benefits
5: 2
Requirements

6. Examples #1
• International Trouble Ticket Exchange
– International customers with offices in many cities
– Bought network services with 2-hour response, 4 hour-fix SLAs
– Three follow-the sun helpdesks on different fault logging systems,
with different engineering teams watching local ticket queues
– Requirement was for > 200 tickets a day to be exchanged between
helpdesks, action taken and updates sent back
– Plan was to integrate helpdesks using a “service bus”, mapping
tickets from one format to another, adopting common SLAs,
converging on common definitions of “response”, etc
– Project spent $1m over 9 months
– Fix was to make all 3 systems visible everywhere and change the
local work practises to allow helpdesk staff to raise tickets in other
geographies
– Satisfied the real requirement: responsive customer service
6: 1
Requirements

7. STAKEHOLDERS
7http://goo.gl/8tL0j

8. Stakeholders and Funding
• Stakeholders are not budget owners
• Rarely is a big IT project funded by the problem owners
• The bigger the projected cost, the more tenuous its
relationship to reality
• Exceptions:
– Finance systems, funded by CFOs
– Billing systems, funded by CFOs
– Local systems, costing less than $100k
• If only CFOs funded software there would be a lot less of it
– It would still be pretty unusable, though…
• There is no problem so large that it cannot be made larger
by throwing lots of money and a team of IT people at it
– Sometimes, repeatedly
– See Mythical Man Month, Fred Brooks
8
Stakeholders

9. Examples #2
• Sales pipeline tool requested by sales director
– CIO #1: “Let’s implement Clarify CRM. We used it in my last company.”
– CIO #2 (12 months later, when project is stalled): “So-and-so are using
Siebel. And we can get a good deal…”
– Last remaining user on implementation team: “What is the difference
between a contact and an opportunity and why should I care?”
• High IT costs: outsource
– CEO: “IT is not our core business. XXX say they can do it cheaper.”
– CEO (12 months later): “Hire someone local to get my Macbook working”
– CIO (18 months later): “We need to hire a team of outsourcing managers”
– 24 months in: cessation of outsource
• Design
– Senior User: “…and it should be as easy to use as Amazon…”
– Developer: <sigh>
– Consultancy client exec: <rubs hands together>
9
Stakeholders

10. 10
People and Influence
Influence
Engagement
http://goo.gl/mVPE8
http://goo.gl/EhHOP
http://goo.gl/2yMQa
http://goo.gl/uDqOR
http://goo.gl/qAUCs
http://goo.gl/UZLBT
http://goo.gl/w7uXe
http://goo.gl/NKCOn
Board
CIO
Project Manager
Architect
Business
Analyst
End User
Developer
User
representative
: 8
Size of dot = degree of
understanding of the
business process
Stakeholders

11. Justifying Software Projects
• Very few IT project managers have financial training
• Very few project board members think of the money as
theirs
• The worst kept secrets in IT:
– Anyone with enough smarts to write a school sick note can make a
business case stack up
– If anyone tries to measure ROI, the author will be long gone and
the scope will have changed anyway
– The bigger the project, the greater the momentum
– CIOs are people too
• The average CIO’s tenure is 4.6 years (Forrester, 2010)
• This is just long enough to finish one or two big projects and start three
others – which the next CIO could easily cancel
• Difficult to keep your eye on building sustainable value unless you’re in
it for the long run
11: 4
Stakeholders

12. PICKING A SOLUTION
12http://goo.gl/LKnbG

13. The “It feels right” approach
• Everyone has a hobby-horse
– Buy vs build, Windows vs Linux, Open Source vs Proprietary
– Not Invented Here syndrome
• Nobody is good at estimating
– Consultants over-charge, in-house teams underestimate
– Plans bear no relation to reality
– Customers don’t have a clue but rapidly become disillusioned
• Justification follows decision
– Solutions chosen by pseudo-scientific “evaluation” exercises
– Planning follows end-date: “we need it by…”
• Decisions are not made in the right place
– Often collective, to spread the blame, protect the decision makers
– Often deferred to senior managers who don’t have all the facts
13: 1
Solution Selection

14. Examples #3: Network monitoring
Buy
• Cost/implementation: $2m, 2
year rollout
• Partial success: ~ 70% (2000 /
3000) devices discovered &
monitored; others added by
hand
• ROI based on selling “value-
add” ability for customers to
monitor the services they had
bought
• Ongoing costs: $350k p.a.
license, 2 administrators
• 3 customers bought it
Build
• Cost/implementation: 2 people,
6 months, say $100k
• 95%+ coverage of ~10,000
cellphone base stations
• No up-front ROI but belief that
this would be “a good thing”
• On-costs: 2 people full-time
• Within 12 months, all mobile
phone network
investment/expansion
decisions were based on traffic
stats from this system
14
In both cases the solution was chosen ahead of the requirements and based
on a “feels right” approach. It works just often enough to be common.
Solution Selection

15. Buy vs Build: Thoughts- 1
• Buy COTS packages like ERP, CRM, HR…
– Implementing these types of package is an enterprise-wide deal,
impacting everybody
– Process changes are going to be hard enough…
• Except in cases of competitive advantage there is no
reason to build these yourself
– Increasingly, even then it’s not worth it. Compete on process
efficiency, not software choice
• You might build a go-cart; you would always buy a bus
• Use SaaS and accept you are going to be a little bit
screwed on price
– Seriously, you are buying simplicity! (Good, fast…)
– SaaS modular pricing means you should only pay for what you use
– ROI becomes a bit more meaningful when you are paying real
money every quarter
15
Solution Selection

16. Buy vs Build: Thoughts- 2
• Sometimes you can’t buy it, or change your business
process to fit a bought solution
• Pick the smallest scope possible
– Creeping featurism / sportscar vs station wagon
– Beyond a certain size, homegrown is almost never a good idea
• Oracle Fusion: Started 2005
• Lots of advice out there
– Plan to throw one away
– Release early, release often
– End-to-end user engagement. User does demos. User answers functionality questions. Anything
else is a cop-out
– The poorest paid but most valuable members of a software team are the analysts
– UI/UX designers run a close second
• Accept no IT specialists. You can get what you want and it
should not be expensive
– But keep the scope small!
• excuses from 16
Solution Selection

17. IMPLEMENTATION
17
http://goo.gl/eqMMf
http://goo.gl/qKvKY
http://goo.gl/OduqQ

18. 18
It’s not about the technology…
LDAP JSON HTML API XML RDBMS HTTP TCP/IP Proxy DNS DHCP
PKI X509 H264 MVC DSL RDFa SQL LINQ .NET CLR Apache
Asynchronous URL URI VDI RDP SSL SSH USB SATA PUE LTE COMET
Latency Bandwidth VLAN VPN HCI UX TDD CI SOA ESB Javascript
XPath XQuery Java Rails VM SAS HBA NIC UDDI SOAP REST Git
Subversion Rack PoE VoIP QoS L2TP RADIUS LDAP SPARQL DOM
JVM PHP HSRP IBM Oracle Microsoft Linux NFS NTLM Kerberos SAML
RTMP XMPP iOS Android Windows VNC SCP Python Perl NTFS
WebDAV Proxy
Implementation

19. It’s all about the people…
• Project Managers
– Grand Designs, anyone?
– Projects have momentum related to size
• Makes it difficult to (re)prioritise
• Even harder to know when to stop
• Consultants / Contractors
– IT salaries & jobs rising, especially contractors http://www.itjobswatch.co.uk/
• In general, contractors are a good thing: focused, expensive, visible costs
• Internal IT staff have domain knowledge but often lack focus
– Consultants love excessive requirements and don't care about ROI
• Except their own: each engagement is “on the job training”
• Prefer bored professionals, not “excited” or “enthusiastic” ones
• Customers
– Most business users don’t know how to negotiate with IT suppliers
and are not educated to understand IT cant
– Buy your IT like you buy a car? Or like you buy a house?
19: 2
http://en.wikipedia.org/wiki/Project
_Management_Triangle
Implementation

20. Ingredients for success
Clear
problem
• Validate it by all means but the general is the enemy of the specific
• Pick one (ideally) or two (but never more) things that need changing
Committed
owner
• Willing to evangelise and create a fuss to get things changed
• Or willing to JFDI
Trusted by
managers
& workers
• Not with a blank cheque, but with the ability to distinguish good from
bad progress and the power to say “stop” if not delivering value
Excellent
communi-
cator
• To sell the idea of change and the idea that it will be for the better
• Fearless Change, Linda Rising http://www.lindarising.com/
In control
from start
to finish
• IT is not a priesthood. Project management is not a black art.
• Small team, accountable, clear measures of progress
20
Implementation

21. Examples 4: Billing System
Buy
• Cost: $5m
• Implementation: 18 months by
specialist ISV
• Minimal business process
change
• Championed by COO
• Business case based on
- flexibility of new tariffs
- large homogeneous corporate
customer base
- opportunity to rationalise
services
• 18 month ROI
Build
• Cost/implementation: 30 IT
staff, 10 years: conservatively
$40m
• Business case based on
“resale” to 20+ business units
• Made sense in business units
– Cheaper than buying COTS
– Analysts with “skin in the
game” championed rollout
• 18 month ROI in BUs, ongoing
break-even centrally
– $1m - $2m per
implementation, $200k
p.a.”license”
21
Common to both: focused, committed business owner with clear, simple goals
Implementation

22. FINAL MESSAGES
22http://goo.gl/jyHbc

23. It’s mainly about the money
• “Doing IT” is still more art than science
– We can’t afford to commission software like we commission art, but
there is no scientific method either
– “IT professionals” are often making it up as they go along
• Filter carefully for bad PMs, lazy analysts, neophyte consultants
• “Show me the money” is the only science I know
– Demand an ROI
– Demand early value
– Cut your losses
– Don’t spend other people’s money
• In 90%+ of cases, the end users should commission, manage and pay for IT
• Make domestic analogies as often as you can: “it’s my money, give me what I
want, or help me understand why not.”
• Methodologies are like standards: plenty to choose from
– Don’t expect a methodology to replace common sense
23: 2

24. QUESTIONS?
24
http://goo.gl/AQDRP