Re-architecting on the Fly #OReillySACon

Rafﬁ Krikorian / raﬃ.krikorian@gmail.com / 9 March 2015 
Software Architecture Conference 2015
RE-ARCHITECTING
ON THE FLY

SO… MY APOLOGIES

I BELIEVE IN “FULL-CONTACT”
PRESENTATIONS

Re-architecting on the ﬂy
The decision and process behind rewriting or re-architecting
a system is often plagued with a series of problems: people
always underestimate the complexity, people never fully
understand the customers, system requirements constantly
change out from under them, and, in almost all cases, it takes
much longer than anybody can predict. As part of this workshop,
we’ll look at a couple of case studies of re-architecture to
gleam strategies of success from them as well as common
pitfalls to avoid. This workshop should arm you with a
framework by which to approach your own decisions around how
to manage, maintain, and evolve your own systems.

@raffi
Raffi Krikorian
VP, Platform Engineering
Twitter ,Inc.
raffi@twitter.com
EXPIRED

2009
0050ish people

2014
3000ish people

2014
0450ish people

About
Tweets sent per second
5800

THE EXERCISE

The Rules
Break up into teams of a few people

Take the following case study
discuss

dissect

ask questions

Play the role of architects and also engineering leads
Volunteers to present when we’re done

STORAGE &
RETRIEVAL
LOGICPRESENTATIONROUTING
Monorail MySQL

Twitter already had some availability issues
earlier this week, leading to a complete
outage of about 90 minutes on Wednesday
and more fail whales popping up yesterday.

What we didn’t anticipate was some of the
complexities that have been inherent in
ﬁxing and optimizing our systems before
and during the event.

…and the team?
We can’t ship things fast enough!
‘Velocity’ is going down! - Management

This code looks like crap! - Engineering

Some hints as to what to do
You all are not just an architects, you are engineering leaders

How do you get a plan in place?

How do you figure out your first step?

What’s the execution of the plan look like?

Then…. what?

(Yes, this is vague. I’ll be floating around the room - ask
me questions!)

DO MY FORMER JOB

LET’S TALK

WHY ARE WE DOING THIS?

Not servicing customers
Not scaling to incoming traﬃc, usage, users, etc.

But also not matching in features that users want

HipChat architected for Amazon Web Services

customer demand to move it to Enterprise

lacking higher level services

lots smaller machines vs small number of bigger machines

It’s easier to write code than it’s to read code
We're programmers. Programmers are, in their
hearts, architects, and the first thing they want to do
when they get to a site is to bulldoze the place flat
and build something grand. We're not excited by
incremental renovation: tinkering, improving,
planting flower beds.
- Joel Spolsky

It’s easier to write code than it’s to read code
Unless properly tended, real world code becomes
complicated over time
energy goes into bug ﬁxes

every week in production means the code is gathering
bandaids, & experiences

Code reviews should be (partially) used to structure
software for “readability”

Technical debt
/** 
* For the brave souls who get this far: You are the chosen ones, 
* the valiant knights of programming who toil away, without rest, 
* fixing our most awful code. To you, true saviors, kings of men, 
* I say this: never gonna give you up, never gonna let you down, 
* never gonna run around and desert you. Never gonna make you cry, 
* never gonna say goodbye. Never gonna tell a lie and hurt you. 
*/
http://stackoverflow.com/questions/184618/ 
what-is-the-best-comment-in-source-code-you-have-ever-encountered

Technical debt
//  
// Dear maintainer: 
//  
// Once you are done trying to 'optimize' this routine, 
// and have realized what a terrible mistake that was, 
// please increment the following counter as a warning 
// to the next guy: 
//  
// total_hours_wasted_here = 42 
//
http://stackoverflow.com/questions/184618/ 
what-is-the-best-comment-in-source-code-you-have-ever-encountered

You haven’t paid it down
Technical debt (also known as design debt or code debt) is a
recent metaphor referring to the eventual consequences of poor
system design, software architecture or software development
within a codebase. The debt can be thought of as work that
needs to be done before a particular job can be considered
complete or proper. If the debt is not repaid, then it will keep
on accumulating interest, making it hard to implement
changes later on. Unaddressed technical debt increases software
entropy.
- Wikipedia

The causes
BUSINESS PRESSURES - the business considers getting something released sooner
before all of the necessary changes are complete. That code is shipped “unfinished”.

LACK OF PROCESS - development is not well managed. Big, long-lived, & isolated
changes are in flight & merging those changes becomes really difficult. Usually, there
is a lack of test suite, which encourages quick & risky band-aids to fix bugs.

ARCHITECTURE - modularity hasn’t been managed, & the software is not flexible enough
to adapt to changes in business needs. As the requirements evolve, the code proves
unwieldy & must be refactored in order to support future requirements. The longer
that refactoring is delayed, & the more code is written to use the current form, the
more debt that piles up that must be paid at the time the refactoring is finally done.

LACK OF ENGINEERING MENTORSHIP - knowledge isn't shared around the organization,
business efficiency suffers, or junior developers are not properly mentored. Possibly,
the wrong engineers have been hired as well.

More, & more, & more
Time seems to dilate

each change is taking longer than we think

each change is turning out to be harder than we think

Each change results in a cascade of new defects

This causes your team to enter a vicious spiral

SOME EXAMPLE STORIES
WAR

Textmate

JUNE
2012
TODAY

History of Word
The time is October 5, 1991

shipped award winning Mac Word 5.0

Win Word 2.0 shipped in 1991 as well with more features

Business problem is emerging - WordPerfect is eating up part
of the market
Technical problem - Mac Word and Win Word are different
codebases
Proposed solution? Project Pyramid.

Pyramid was cancelled
Microsoft decided that “taking a few steps back” for Pyramid would
cause them to lose to WordPerfect

New strategy, target Mac Word from the Win Word codebase

Huge problems

Windowing is handled fundamentally diﬀerently between the operating
systems

68K Mac OS doesn’t do memory management

compiled with a beta version of Visual C++ 2.0 (so beta that most
optimizations had to be turned oﬀ)

In the end - Mac Word 6.0 wasn’t “Mac-like”

Apache HTTP
1996
1997
1998
1999
2000
2001
2002
Whispers of
Apache 2
Apache Week talks about
ﬁltering, & diﬃculty in
implementation. Need
multi-threading
Huge rewrite
planned for post 1.2
Decide same
codebase for all
releases in 2.0
Halt features
in pre 2.0
2.0 alpha
SSL in 1.3 & 2.0
2.0 beta
2.0 GA
Planning in
place

EC2 performance is sad
Customers demand exceptional performance & always-on
availability
experienced networking issues

hanging VM instances

unpredictable performance degradation (probably due to
noisy neighbors)

Spending more time working around these issues rather
than features

Migrating SwiftType
Nothing fundamentally wrong with the architecture, just not
suitable for a zero downtime migration
Need connectivity between EC2 and the new datacenter

Migrate data

stateless services are not a problem

choose whether the backend services are replicated or deal with
inter-datacenter latency
build custom replication services for backends that can’t do it
natively (search)

What happens during a rewrite?
Business stays still or business moves on

Have to implement accrued business logic in a fraction of time

old code is used

old code is tested

lots of bugs have been found and fixed

The old system can’t be turned off until the new system is at
100% parity (or the company agrees to turn off the old)

This will ABSOLUTELY take a lot longer than anybody thinks!
BAD!
BAD!

What about non-code issues?
You’re going to gather seriously unhappy customers
external - no features!

internal - things are being held up

Political battles within engineering

Excessive frustrations because deadlines  
will almost certainly be missed
NOBODY EVER TAKES INTO
ACCOUNT THIS STUFF

STORAGE &
RETRIEVAL
Monorail MySQL

STORAGE &
RETRIEVAL
Redis
Memcache
Flock
T-Bird
MySQLTweet
User
Timeline
Social
Graph
DMs
API
Web
Monorail
TFE

NEVER to do one if you can get away with it
It will take longer than you think
The market, product, & business will change while your
development is in ﬂight

Existing customers will become frustrated
YOU DON’T CONTROL THE REWRITE,  
IT CONTROLS YOU

BREAK!

SO, I HAVEN’T TALKED YOU
OUT OF IT YET

Deep philosophical question
Are you better than those that came before you?

If you have the same team, there is NO GUARANTEE that
your team now will do better the second time through

If you have a diﬀerent team, there is NO GUARANTEE that
your team now will do better than the last team

This isn’t big, right?
How good is your documentation?

Do you even know all the corner cases?

Systems get stronger by being in production

they accumulate bug ﬁxes, bandaids, operational
knowledge

how well do you know all those?

can you reproduce all those?

@timoreilly testing, testing. can you hear this?

@rafﬁ i’m trying to reach you!

ALL THE THINGS TO PLAN FOR

Functionality
Do you REALLY know what the system does?

all the corner cases of the product?

“make it do what it already does” is harder than you think

Most programmers don’t know what to ask!

doubly true if they weren’t the original designers of the system

even if they are the original designers, no way they remember
all the corner cases

Can code serve as the spec?
PROBABLY 
NOT!

Feature creep
This is incredibly tempting, especially if feature
development is halting on the old system

It will potentially kill you
YOU TRY NOT TO DO THIS DURING REGULAR
DEVELOPMENT, SO, WHY START NOW?

SIMPLICITY COMPLEXITY
FLEXIBILITY

Flexibility breeds complexity
If you try to build a system
that is ﬂexible, you’ll
probably get a system that is
complicated

If you build a system that is
simple, you may be able to
build something that is
ﬂexible onto of it

Aside: The Unix Pipeline
program1 | program2 | program3
A set of processes chained by their standard streams.
The output of each process (STDOUT) feeds directly as
input (STDIN) to the next one

Very simple idea evolves into component based software
engineering
STDOUT
STDIN

The different stages of software
IRRELEVANT
“OLD SCHOOL”
MAINSTREAM
EARLY ADOPTER
IDEA TODAY

Nothing works as expected
1. TECHNOLOGY TRIGGER - proof-of-concept stories &
media interest trigger significant publicity. No usable
products exist & commercial viability is unproven.

2. PEAK OF INFLATED EXPECTATIONS - some success & lots
of failures. Few companies take action; many do not.

3. TROUGH OF DISILLUSIONMENT - experiments &
implementations fail. Investments continue only if
the surviving providers improve their products.

4. SLOPE OF ENLIGHTENMENT - instances of the
technology’s benefits start to crystallize & become
more widely understood. 2nd- & 3rd-gen products
appear from providers. More pilots funded;
conservative companies remain cautious.

5. PLATEAU OF PRODUCTIVITY - mainstream adoption
starts to take off. The technology's broad market
applicability & relevance are clearly paying off.
1 2 3 4 5

The 5 stages of grief
1. ELATION - yay! we get to throw out that crappy product! we
get to start it over, & do it the right way!

2. BUCKLE DOWN - wow, there is a lot of work here.

3. OH SHIT - there really is a lot of work here. We’re never going
to get it done in time.

4. EXHAUSTION - we missed our deadline. Again.

5. RELIEF - we’re out the door! But… it’s crappy. We cut
corners, but, we’re done. Next time, we’re totally going to do
it right.
http://www.jamesshore.com/Blog/How-to-Survive-a-Rewrite.html

ONE LAST TRY, BECAUSE I DON’T
WANT YOU TO DO THE REWRITE!

Ammunition against the rewrite
One goal of a rewrite is usually to create a cleaner
codebase
remember how learning all the edge cases is hard?

remember how writing code is easier than reading code?

to save time, a lot of developers cut and paste code from the
old system to the new system!

It’s hard to create a cleaner codebase during the rewrite!

Can we quantify this somehow?
Can we make a more data driven decision?

Gives more conﬁdence to the decision, and helps sell it
better to others

A rewrite is a massive investment, and its only fair to be
really sure before going down the road

If you don’t do anything
pstarting + (vcurrent x t)

If you are on a new system
pstarting + (vnew x (t - trewrite))

What we’re betting on
pstarting + (vcurrent x t)

<


For some value of t where t > trewrite

Variables to play with

vnew = that the new system has to be faster to iterate on

trewrite = that we need to min(time to write the new system)
vcurrent x t < vnew x (t - trewrite)

How do we estimate vnew
Take the new tech for a spin
Have a small team build something non-trivial

This will not be a hard science

gives you a sense of the extent of the software

gives you a sense of the capabilities of the team

Have to understand maintenance costs, scalability costs

How do we estimate trewrite
Given that you estimated vnew, use that to try to scope
out the rewrite

Given an estimate of trewrite, try to understand whether
that t is too long

what is the state of the product now?

how much time will you lose in the business?

OK, I STILL HAVEN’T TALKED
YOU OUT OF THIS

Hold the line
Product managers are going to get
anxious

you’ve sold them on a great vision

they feel like you are taking too
long and blocking them

You have to manage them (or,
ignore them)

Do the smallest thing you can
each step of the way

Define “done”
100% feature matching is
difficult
Implementing features is hard

remember how hard it was to
write it the first time?

you’re at a different part of
the software cycle

Try to do things incrementally

Aside: The Saturn V
Von Braun says “German
engineering dictates that you
should test each part”

Worst case situation: a ﬁreball
of 1,400’ that will burn for 35s
at 2500F

Will we make the “end of the
decade”?

George Mueller, pulled rank,
and pushed for an “all up test”

Be incremental
All-up almost never works - the
Saturn V is an exception
Don’t be “lazy” and assume all-up is
easier

do an integrated approach & ready
to release at every step of the way

TFE
(HTTP Proxy)
Monorail Woodstar

TFE
(HTTP Proxy)
Monorail Woodstar
100% Monorail 0% Woodstar

TFE
(HTTP Proxy)
Monorail Woodstar

Find the starting line
If you're going with feature parity, then do the biggest
bang part of the system ﬁrst

If you can, spend a bit of time extending your runway a bit
too

Drive with data, & instrument the old system to gather
that data

Twitter API
Other
Photos
Proﬁle
Activity
Home

Don’t ignore data!
Your implementations may change, but your
underlying data will change very slowly

Fake or synthetic data gives you a false sense of
security
Test with real data as soon as possible

get real data piped through the system

ﬁgure out how you are going to do reconciliations

Manage tech debt better
Try to reduce scheduled pressure
Establish a culture of design quality

Start refactoring, continuous
design, & other code-quality
practices

Partition out a portion of time to
always work on technical debt

Shy away from vanity stuff
Don’t let the sirens drag you to a “hot new language”

Stay away from the trap of “we could do a better job
recruiting if our stack looked like…”

Do what’s right for your team and what your team can
execute on

Prepare for mounting tension
We all know that fixing bugs and firefighting is stressful

imagine one team who has to do this, & another team
who doesn’t have to worry about it

which would you like to be on?

Tension is going to build & people are going to be angry

Make friends, communicate, & be transparent

Know the business
Figure out who are all the stakeholders
in the rewrite

Figure out who are the decision makers
& the ones who control your fate

they are usually not those who
complain the most

they are those who have “the
power”

Understand all the diﬀerent non-
technical motivations of the
company

Find your inner salesperson
You have to sell it to the business

Gather all the data

cost savings

feature iteration

reliability

performance

stability

Focus on the data. Don't use
anecdotes, but, instead, show results
of experiments.

Get ready for the politics
You are in a precarious position

You are exposed

taking up engineering resources

not delivering new features for the
company

in an “open transaction”

Keep the in-ﬂight work small and
well integrated

Keep an eye on code quality!
On repeat - getting feature
parity is really hard!

To keep parity, developers,
inevitably, cut & paste code
from a previous system!

Have code quality metrics
APPROVED

Get the team ready
Organizations which design systems ... are
constrained to produce designs which are copies of
the communication structures of these organizations
- Conway’s Law

I still haven’t talked you out of this
1. Hold the line

2. Deﬁne “done”

3. Incrementalism

4. Find the starting line

5. Don’t ignore data

6. Manage tech debt better

7. Stay away from vanity stuﬀ

8. Prepare for mounting
tension

9. Know the business

10.Get ready for politics

11.Key an eye on code quality

12.Get the team ready

LET’S RECAP!

The 5 stages ofgrief
1. ELATION - yay! we get to throw out that crappy product! we
get to start it over, & do it the right way!

2. BUCKLE DOWN - wow, there is a lot of work here.

3. OH SHIT - there really is a lot of work here. We’re never going
to get it done in time.

4. EXHAUSTION - we missed our deadline. Again.

5. RELIEF - we’re out the door! But… it’s crappy. We cut
corners, but, we’re done. Next time, we’re totally going to do
it right.
http://www.jamesshore.com/Blog/How-to-Survive-a-Rewrite.html

vcurrent x t < vnew x (t - trewrite)

THANKS! (WHEW)

Re-architecting on the Fly #OReillySACon

More Related Content

Viewers also liked

Similar to Re-architecting on the Fly #OReillySACon

More from Raffi Krikorian

Recently uploaded

Re-architecting on the Fly #OReillySACon