For startups and enterprises alike, the value of speed, the ability to pivot rapidly and to orient around customer behavior has increased– these abilities are now essential, some would argue - existential. Yet, growth brings an asymmetry: you now have to be stable, cost efficient, and keep customer promises, while disrupting yourself - building the next propositions for customer delight. In this session, Sheldon will distill the true foundations of scalability - people, processes, and technology – and how to organize all three for increasing customer and business impact

1. Sure You’re Growing, but
Are You Scaling?
February, 2019
@sheldon_tm @PublicisSapient

3. Circa 1995, B.G.
C++ Developer
Employee 196

5. Omni-channel Platform for a Global Hospitality Major
with 30 brands
• Retired 10+ year old systems, launched in 9 months
• 12 scrum teams, 2 continents, 5+ locations
• 100% traffic in 2018
• 20+MM daily page views
• 6 week release cycle -> weekly/release on demand

6. CHANNEL
SOURCE
SYSTEMS
OF RECORD
Mainframe
• Availability
• Products
Content
• Labels
• Photos/Videos
Search Engine
• Property Index
• Content Index
Property DB
• Property Info
• Reviews & Ratings
SHOPPING
PRESEN-
TATION
SERVICES
Un-dated Search
Keyword Search
Dated Search
Dated Search
Dated Search
Dated Search
Dated Search
City Search Group Search
Dated Search
Dated Search
Alternate Date Search
Dated Search
Dated Search
Map Based Search
Caching
Poi Search
Suggestion Search
Deals/Offers Search
SHOPPING
DOMAIN
SERVICES
Policy Management Api ManagementService Discovery
Request Validation Security
Caching
Properties
Dated Search
Dated Search
Products
Dated Search
Dated Search
Availability
Dated Search
Dated Search
Pricing
Content
Gateway

7. Scale
Shopping
• 150 billion requests/year
• 10 million shop calls/hour
• 3 million rate changes/week
• 30% YoY growth
• 10% YoY third party growth
Tactics
• Features – Microservices
• Architectural layers – Caching
• Infrastructure – Cloud Agnostic

8. “A service is said to be scalable if when we
increase the resources in a system, it results
in increased performance in a manner
proportional to resources added.”
Werner Vogels

9. The Universal Scalability Law
Source: Neil Gunther, https://arxiv.org/abs/0808.1431v2
C = Usable Capacity
N = Workers
α = Contention (queuing)
β = Coherency (cross-talk)
C(N) =
γN
1 + α(N − 1) + βN(N − 1)

10. The Universal Scalability Law
C = Usable Capacity
N = Workers
α = Contention (queuing)
β = Coherency (cross-talk)
C(N) =
γN
1 + α(N − 1) + βN(N − 1)
Workers
Output
Source: Neil Gunther, https://arxiv.org/abs/0808.1431v2

11. Common Scaling Tactics
1. Optimize repeated processing
2. Reduce contention via replication
3. Minimize use of shared resources
4. Reuse resources and results (caching)
5. Partition and parallelize
6. Scale out, not up
7. Asynchronous, stateless, reactive patterns
8. Relax transactional consistency (CAP theorem
optimizations)
9. Prioritize work/degrade gracefully
10.Automation, rollback, observability, failover
11.…

12. How do organizations scale?

13. General
Principles of
Management
Daniel McCallum (1856)

14. All that is required to render the efforts of railroad companies in every respect equal to that
of individuals, is a rigid system of personal accountability through every grade of service.
1. A proper division of responsibilities.
2. Sufficient power conferred to enable the same to be fully carried
out, that such responsibilities may be real in their character.
3. The means of knowing whether such responsibilities are
faithfully executed.
4. Great promptness in the report of all derelictions of duty, that
evils may at once be corrected.
5. Such information to be obtained through a system of daily reports
and checks that will not embarrass principal officers nor lessen
their influence with their subordinates.
6. The adoption of a system, as a whole, which will not only enable
the general superintendent to detect errors immediately, but will
also point out the delinquent.
Source: https://howlingpixel.com/i-en/Daniel_McCallum

16. Burgernomics:
The Big Mac Index
Image: Antti Vuorela [CC BY-SA 3.0
(https://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia Commons

17. 1949
Fries
1949
Car Hop
1968
Big Mac
1975
Drive Thru
2015
All Day Breakfast
2014
Delivery

18. “In the last five years the world has
moved faster outside the business
than inside…
The business cannot ignore what
customers are saying when the
message is clear: We're not on
our game.”
Steve Easterbrook, CEO
May 2015

19. Exponential Growth of Computing for 120 Years
Source: https://commons.wikimedia.org/wiki/File:Moore%27s_Law_over_120_Years.png

20. Time
Change
Technology
changes at an
exponential rate
This change gap widens
over time, eventually
requiring a Ctrl+Alt+Del
Organizations
change at a
linear rate

21. 2006
“Don’t talk to strangers
on the internet.”
“Never get in a stranger’s car.”

22. 2006
“Don’t talk to strangers
on the internet.”
“Never get in a stranger’s car.”
2016
Literally use the Internet
to call strangers
and get in their car

23. 2
Learn/Pivot
Rapidly
1
Speed
3
Orient Around
Customer
Behavior

24. Source: World Economic Forum/Bain & Company

25. General
Principles of
Management
Jeff Bezos (2002)

26. If you can’t feed a team with two pizzas, it’s too large. And…
1. All teams will expose their data…
2. Teams must communicate through
interfaces
3. …no other form of interprocess
communication allowed
4. It doesn’t matter what technology they
use
5. Interfaces, without exception, must be
externalizable.
6. Anyone who doesn’t do this will be firedSource: https://gist.github.com/chitchcock/1281611

27. Revisiting The Universal Scalability Law
C = Usable Capacity
N = Workers
α = Contention (queuing)
β = Coherency (cross-talk)
C(N) =
γN
1 + α(N − 1) + βN(N − 1)
Workers
Output
Source: Neil Gunther, https://arxiv.org/abs/0808.1431v2

29. 250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250

30. 250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
65,000
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250
50 lines of
Fortran
250

31. 31

34. to begin, begin.

35. +
Product
owner
Business
architect
ops
CX design
lead
Complianc
e
Branch
Digital
technologis
t
Legal
Telephony
center
Feature
team
Business
roll-in team
Model
office team
LAB TEAM MAKE - UP SCALING TEAMS
Business or IT led Customer led
Top-down governance
Fast delegated
decision making
Release only
when perfect
Release when there’s
customer value
Project-specific
business case
VC-style funding allocation
IT and business
”co-located”
Single multi-functional
end-to-end team
To
To
To
To
To
1. Orient Around Customer Behavior

36. 1. Orient
Around
Customer
Behavior

37. Continuous JourneyFixed Destination
Scope
“What are we delivering?”
Time
“When will we deliver?”
Cost
“How much effort will this take?”
Quality
“How do we continually ensure
quality?”
Productivity
“How can we go as fast as possible?”
Value
“What will maximize customer
value?”
2. Change your steering system

38. 3. Optimize your Value Streams
Customer
Request
Customer
Receipt

39. 3. Optimize your Value Streams
Catalog of 85+ systemic and engineering interventions sourced from XP, 12 Factor App, Lean, Industry Experts & Publicis Sapient Experience
Systemic Engineering
Service
Introduction
Individual
Productivity
Highly Targeted
Op Model &
Org Design
Changes
Reduced
Documentation
& Approvals
Highly Targeted
Architecture
Intervention
Engineering
Practices &
Automation
Automated
Deployments
& Environment
Provisioning

40. Code review Builds
Code
compliance Deployment on ci
environmentDevelopment
Junit & code
coverage
Cobertura
or Karma
Functional
automation test
Pa11y
Security scan
Performance
automation test
Automated deployment / environment on
demand
Publish reports / update dashboard
/pass/fail status
Business
owner
Service
Developer
Service
Operations
Service
Tester
Developer idea
Development
Unit
Test
Code
Compliance
Test
Commit
Rapid feedback
4. Tooling and automation

41. Performance
automation test
Code review Builds
Code
compliance
Deployment on ci
environmentDevelopment
Junit & code
coverage
Cobertura
or Karma
Functional
automation test
Pa11y
Security scan
Automated deployment / environment on
demand
Publish reports / update dashboard
/pass/fail status
Business
owner
Service
Developer
Service
Operations
Service
Tester
Developer idea
Development
Unit
Test
Code
Compliance
Test
Commit
Rapid feedback
1.
Develop code
2.
Automated end-to-end
testing across systems
3.
Automated
reporting &
deployment
90 days to 2 minute
DevOps
Testing Cycles
4. Tooling and automation

42. 4. Tooling and automation
Enterprise EnablerCBS / Unisys
Bundled Raw
Events
CBS /
UNISYS
Raw
Events
Data
Processor Curated Event
Stream
Enabler
Batch Jobs
Data Ingestion
CRUD
Operations
EAADS
Consumers
Request-response
Micro-services
Event-driven
Micro-services
SOURCE
SYSTEMS
CBS
SODS
TODS
Source
System
Snapshot
& Historical
Data LoadData Load
Enabler
Data
Load to
EAADS
Snapshot
Load
Platform and Infrastructure team

43. Namespace:Development
Namespace:Staging
Namespace:Production
Developer Portal
Battle tested
Catalogue of Repositories
New Project
Choose Appropriate
templateDEVELOPER
GitHub
New / Updated template
Business Logic
Orchestration File (ex : Jenkinsfile)
packaging (ex. :Dockerfile)
Environment Orchestration (ex : Helmcharts)
Infra
Account
Infra Billing
AWS
Account,
Keys
Create Cluster
Create Disk
Create Pod
Create Service -1
Create Service -N
Health
checks
DISK
LIST OF
SLAVE
IMAGES
Infrastructure
Template with Default Business Logic
Test Cases & dummy Test Data
3
1
2
6
74
9
8 5
Images on-
demand
4. Tooling and automation
1
2
3
4
5
7
6
9
8
Choose a template from the
Market Place
Template copied to your
Github account
Submit updated or Brand New
Template to Template repo
Developer enhances the business
logic/test cases/test data and
checks in the code
Orchestrator instantiates the
Environments Dev/Staging/
Prod and executes tests with
test data
Orchestrator picks up the
template and creates pipeline
Provide the Cloud Account ID
linked to Cost Centre/billing
DevelopmentProcess

44. 4. Tooling and automation
amplifies personal
networks to build
meaningful relationships
The power of
connection
personalises information
and provides actionable
intelligence
The power of
knowledge
recognises ambitions
and overcomes barriers
to augment growth and
evolution
The power of
opportunity

45. 5. Make Goals Visible
Culture
Modern DevOps Toolchain
Done in sprintWeeks
2 minutes 48 seconds1-2 days
Accessibility testing duration
Duration of initial code quality test
Time to execute test scripts 3 hoursWeeks
Lead time from “Idea to live” 4 - 6 months12 - 18 months
Number of operational Labs 80Multidisciplinary Labs
Route-to-live time 5 days (for presentation layer)90 daysAgile Method & Mindset
Time to create new development environments 10 minutesWeeksTest & Learn On Cloud
Customer Impact
5 days40 days
20 min45 min
Micro-service-led Architecture
And App Consolidation
Prototypes/experimentsDiagrams/boxesArchitectural communication approach
Architectural design Modern architecture with Reactive programming3-tier architecture
Number of shared components 200Shared Components
Management parameters Productivity, value, qualityTime, cost, scopeData-centric Management Model
To…From…
Time to open a commercial account
Time to open a retail account in branch
E2E processing time for pension contributions
Time for corp. pension customers to update files 21 days
30 days
24 hours
1 day

46. 5. Make Goals Visible
Quality Panel: Automated test
% collected as an excel input
Quality Panel: Defect
injection into production
collected from Jira
Quality Panel: Code Quality
rating collected from Sonar
Quality Panel: Unit test
coverage collected from Sonar
Quality Panel: Count of
completed stories demonstrated
to Product Owner collected
from Jira
People Panel: Kaizen mindset –
retrospectives are happening and
actions are taken, collected from
Jira or excel input
People Panel: Happiness Index –
are people engaged and happy,
collected as excel input
Speed Panel: Do we have
enough stories in the backlog to
keep the flow of work going –
collected from Jira
Speed Panel: Code build and
broken build time durations,
collected from Jenkins
Speed Panel: Time to create
a new environment,
collected as excel input
Speed Panel: Number of
check-ins done per day in master,
collected from Jenkins
Speed Panel: Key cycle time
components for end to end
lead time (DOR-DOD, DOD-
Live), collected from Jira
Speed Panel: Flow of work
through various states and
bottleneck identification -
information collected from Jira
Speed Panel: Is team
velocity stabilized and is
the team delivering to the
forecast?
Speed Panel: Stories
marked as tech debt,
collected from Jira
Speed Panel: Functional test
execution time, collected as
excel input
Engineering Team Dashboard

48. thank you
copyright publicis sapient | confidential
@sheldon_tm @PublicisSapient