The document discusses 7 habits of data effective companies. It describes how companies have evolved through different digital maturity phases from analog to born-digital. The key differences observed between phases include impact on cost, value extraction, and capabilities. The 7 habits discussed are: treating data processing as an industrial process, focusing on latency and waste reduction, being use case driven and value stream aligned, initially centralizing data, architecting for failure and sharing, treating it as a software engineering problem, and following the Unix philosophy of building specialized components. The document provides examples and illustrations for each habit.

1. www.scling.com
The 7 habits of
data effective companies
Lars Albertsson, Founder, Scling
2022-11-08
1

2. www.scling.com
Digital maturity phases
Analog Software enabled Information-based Born-digital
Cafés Elderly care Construction Cars Banks Telecom Spotify FAANG
Digital information
foundation exists
Disruptive value
from data
● More value from data (+ AI) to the right
● What differs between information stage vs born-digital?
● Presentation scope: Observed differences & underlying dynamics
2
Analog Software-enabled Information-based Born-digital

3. www.scling.com
Observed differences
● Impact
○ Cost
○ Value extraction
○ Capability
● 7 habits
○ Data factory - from craft to industrial process
○ Lean process - focus on latency and waste
○ Use case driven - value stream aligned
○ Centralise first
○ Architect for failure and sharing
○ It's a software engineering problem
○ Unix philosophy
3

4. www.scling.com
● From raw material to refined artifact that is appreciated by someone
○ Raw, collected data → data artifact (report, recommendation, …)
○ Customer with a need → sales transaction
○ Employee with an idea → product making a user happy
● Intermediate artifacts produced as side-effect
○ Indirect, partial value
Value creation streams
4

5. www.scling.com
Efficiency gap, data cost & value
● Data processing produces datasets
○ Each dataset has business value
● Proxy value/cost metric: datasets / day
○ S-M traditional: < 10
○ Bank, telecom, media: 100-1000
5
2014: 6500 datasets / day
2016: 20000 datasets / day
2018: 100000+ datasets / day,
25% of staff use BigQuery
2021: 500B events collected / day
2016: 1600 000 000
datasets / day
Disruptive value of data, machine learning
Financial, reporting
Insights, data-fed features
effort
value

6. www.scling.com
Data value cost - master data management example
MDM = single view of important entity
● Large bank
○ View of all customers
○ ~millions of records
○ Proprietary vendor MDM tool
○ 5 person team
● Spotify
○ View of all users
○ ~100 millions of records
○ Hadoop MapReduce pipeline
○ ~10% of full-time engineer
6

7. www.scling.com
What conclusion from this graph?
COVID-19 fatalities / day in Sweden
7

8. www.scling.com
What conclusion from this graph?
COVID-19 fatalities / day in Sweden
8
Fatalities collected during 2 day
Fatalities collected during 4 days
Fatalities collected during 10 days

9. www.scling.com
Forecast for analytics with fresh data
9
Graph by Adam Altmejd, @adamaltmejd

10. www.scling.com
and sometimes reach them.
Scandinavian companies reach for the stars
10

11. www.scling.com
Data not acted upon - value left on table
11
Changing to winter tyres at Bilia. When I arrived,
the staff could see that mechanics were currently
not fully booked, and offered me a front wheel
adjustment, which I accepted. Data value taken
from table.
Neighbour jumps in their Volvo, talks to husband
through window, and drives off. On arrival she
discovers that she has no car key - husband's key
was close enough to start car. An early warning
that key is no longer in proximity would have been
valuable.
I make the effort to submit a detailed bug report
regarding Volvo's known over-aggressive
automatic safety brake problem, in order to
provide more data for debugging. I receive no
feedback from the process, and lose interest.
Our Volvo incorrectly activates the automatic safety
brake again. This is a known problem since years
back, which affects our car occasionally. I have no
efficient channel to report time and circumstances
for this occasion, allowing Volvo to get more data
on the problem, and me to feel confident that the
issue is worked on.
After a repair at Bilia of the rear view mirror, the
car does not properly detect cars in front. Camera
near mirror is suspected. Bilia cannot obtain
camera measurements or car detection statistics
from Volvo, so a cycle of trial and error repairs
follows, taking me to the mechanic multiple times.
The Volvo phone app warns about
many things: interrupted charging, car
parked but not being charged, etc. But
not about the roof and windows left
open in the rain. Rear view mirror
electronics now need a repair.
I change the audio balance in the
Volvo with six screen clicks.
Measurements could indicate that
long click sequences at high speed
indicates poor user interface.
I send a design suggestion to Volvo with pics of a
hacked solution for a luggage problem. A simple
hook could solve it. I do not receive "We have
assigned your suggestion id X, and will get back if
it is implemented." I will not send another, and feel
less connected to the brand.
Wife has driven our Volvo. Mirrors and seats
automatically move to my position, except for the
right mirror, which I need to correct again, as I
have for years. Volvo could have measured and
detected irregular changes of settings, and found
this bug.
Wife unlocks Volvo. Driver seat switches to her
position. I take the driver's seat, pull it back, and
change to my profile. Car switches seat back to
her position, and I have to pull it back again.
When seat heating was automatically activated,
"Increase seat heating" audio command turns it
off. I make additional commands to get to the
desired state. Measurements could identify
repeated audio or UX commands.

12. www.scling.com
Enabling innovation
12
"The actual work that went into
Discover Weekly was very little,
because we're reusing things we
already had."
https://youtu.be/A259Yo8hBRs
https://youtu.be/ZcmJxli8WS8
https://musically.com/2018/08/08/daniel-ek-would-have-killed-discover-weekly-before-launch/
"Discover Weekly wasn't a great
strategic plan and 100 engineers.
It was 3 engineers that decided to
build something."
"I would have killed it. All of a sudden,
they shipped it. It’s one of the most
loved product features that we have."
- Daniel Ek, CEO

13. www.scling.com
Security Waterfall
1. Data factory - from craft to industrial process
13
Application
delivery
Traditional
operations
DevSecOps
Traditional
QA
Infrastructure
DB-oriented
architecture
Agile
Containers
DevOps CI/CD
Infrastructure
as code
Data factories,
data pipelines,
DataOps

14. www.scling.com
From craft to process
14

15. www.scling.com
From craft to process
15
Multiple time windows
Assess ingress data quality
Repair broken data from
complementary source
Forecast based on history,
multiple parameter settings
Assess outcome data quality
Assess forecast success,
adapt parameters

16. www.scling.com
Sustainable production machine learning
16
Multiple models,
parameters, features
Assess ingress data quality
Repair broken data from
complementary source
Choose model and parameters based
on performance and input data
Benchmark models
Try multiple models,
measure, A/B test

17. www.scling.com
2. Lean factories - focus on latency & waste
17
Value-stream mapping:
vs
Primary waste forms in data value streams: "Put first things first"
● Big design / modelling / architecture up front
● Redundant technical environments / components
● Human coordination
● Rules and processes no longer needed

18. www.scling.com
Eliminating delivery friction
18
● In theory simple - scrutinise everything
○ Positive engineering: writing code, tests, docs, refactor, improve
○ All else is negative
● You are limited by your assumptions
○ State of practice far from state of art
But the test suite
takes 3 hours.
We have this
checklist.
Security must
approve.
X must be
released before Y.
That is another
team's job.
We don't have
access.
We must test in
staging first.
We haven't
performance
tested yet.

19. www.scling.com
3. Use case driven, value stream aligned
● Stream-aligned teams
○ Direct business value impact
○ Few handoffs
● Platform & specialist teams serve stream-aligned teams
○ Mechanism for business value to set priority
"Begin with the end in mind"
19

20. www.scling.com
Value-driven teams
20
Raw
Commitment iff
onboarding
Waterfall, one team delivers to next
- Time to business value, low agility

21. www.scling.com
Cross functional teams
21
● Team has competence for end-to-end value creation
○ Data engineering (software, QA, DevOps)
○ Subject matter expertise
○ Product management
● Most value in technically simple things
○ Counting things
○ Data availability
○ Fancy things and machine learning - higher cost, lower ROI

22. www.scling.com
Push-driven vs pull-driven
22
● Technology capabilities
○ In-house
○ Vendors
● Ivory tower platforms
● Modelling up front
● Work driven by business stakeholder needs
● Teams self-sufficient to deliver value
● Common services driven by internal stakeholders

23. www.scling.com
Push-driven vs pull-driven
23
● Technology capabilities
○ In-house
○ Vendors
● Ivory tower platforms
● Modelling up front
● Work driven by business stakeholder needs
● Teams self-sufficient to deliver value
● Common services driven by internal stakeholders
Started 2002 / 2006,
launched 2010,
killed 2012
1000 person years,
cost $125M
Started 2009-05-10,
launched 2009-05-16
$80M revenue in 15 months
https://www.flickr.com/photos/downloadsourcefr/15944373702, CC BY 2.0

24. www.scling.com
4. Centralise first
● Decentralisation is a scaling decision
○ At a high cost in friction
● Most successful data lakes centralised
until very large
○ 1000+ node clusters
○ 10000+ jobs / day
○ 100000+ datasets / day
● Premature decentralisation harmful
○ Without strong homogeneity → silos
○ The data mesh is not for you
24

25. www.scling.com
Data-centric innovation
● Need data from teams
○ willing?
○ backlog?
○ collected?
○ useful?
○ quality?
○ extraction?
○ data governance?
○ history?
25

26. www.scling.com
Data platform
Big data - a collaboration paradigm
26
Stream storage?
Data lake
Data
democratised

27. www.scling.com
5. Architect for failure and sharing
● Data processing is fragile
○ Spans across domains
○ Spans across time
○ Immature, heterogeneous components
○ Few engineers / value flow
● Accept high failure probability, reduce impact
○ Functional architecture
○ Offline - online separation
○ Versioned data for fallback
27

28. www.scling.com
Data lake
Transformation
Cold
pond
Data factories
28
Mutation
Immutable,
shareable

29. www.scling.com
Online vs offline
29
Online Offline
ETL
Cold
pond
Online
Data feature
Dataset
Job
Pipeline
Service
Service

30. www.scling.com
OO vs functional architecture
REST DAO RDBMS Stream processing Data lake
OO λ
ORM Document DB Data warehouse Lakehouse Frozen lake
30

31. www.scling.com
● Forgiving environment
○ Machine errors
○ Human errors
● Friendly to experiments
○ "Dark pipelines" run in parallel
● Operationally efficient
○ Separate dev, test, staging environments not necessary
○ Self-healing
Raw, immutable data + transformation process
31
∆?

32. www.scling.com
Life of an error, microservices
32
● My service, bad code!
○ Bad data spreads instantly
● Different recovery strategies - ad-hoc
○ No standard patterns
○ No guaranteed recovery
● High cost of error
○ Proactive QA - staging environment
Service
Data
warehouse
ETL
Queue
DB

33. www.scling.com
Life of an error, batch pipelines
33
● My processing job, bad code!
1. Revert serving datasets to old
2. Fix bug
3. Remove faulty datasets
4. Deploy
5. Backfill is automatic (Luigi)
Done!
● Low cost of error
○ Reactive QA
○ Production environment sufficient

34. www.scling.com
Life of an error, frozen lake
34
● My processing job, bad code!
1. Revert serving datasets to old
2. Fix bug
3. Bump pipeline version
4. Deploy
5. Backfill is automatic (Luigi)
Done!
● Low cost of error
○ Reactive QA
○ Production environment sufficient

35. www.scling.com 35
Life of an error, streaming
● Works for a single job, not pipeline. :-(
Job
Stream
Stream Stream
Stream Stream Stream
Job
Job
Stream Stream Stream
Job
Job Job
Reprocessing in Kafka Streams

36. www.scling.com
6. It's a software engineering problem
● Data value can be tackled from different angles
○ Strategy
○ Data science
○ Technology
○ Engineering
● Things happen when it becomes a software engineering problem
○ Engineering = building internal artifacts for value extraction
○ New ways of working - manifest in processes
○ New ways of collaborating - manifest in shared structures
36

37. www.scling.com
Process engineering
● Idea - deploy - feedback loop
○ Find and understand code & data
■ Absence of walls
○ Continuous integration & deployment
■ Integration == integration
● Automate toil
○ Pull-driven
○ Carrots > whips - "Golden path" platform service
● Continuous improvement
○ Visualise friction
○ Feedback & priority loops between teams
○ Sharpen the saw
37

38. www.scling.com
● Functional QA
○ Feedback on code
○ Testing challenge
○ Guard rail for development speed
● Data product QA
○ Feedback on code + data
○ Monitoring & observability challenge
● High quality demands high code
○ Low/no-code incompatible with SWE processes for speed & quality
○ SQL lacks convenient support for data quality measurements and improvements
Quality assurance is a fundament for speed
38
15 minutes!

39. www.scling.com
Big vendor Few wide scope components Many small scope components
Tied to vendor evolution Perpetual integration work Amortised integration work
● Successful companies tend to own the mortar
7. Unix philosophy - do one thing well
39

40. www.scling.com
One-way vs two-way decisions
40
● Architect for two-way decisions
○ Small, replaceable components
○ Non-destructive computations
○ Sharing without risk
● One way: privacy architecture
Expensive
to reverse
Easy to
reverse

41. www.scling.com
One-way vs two-way decisions
41
● Architect for two-way decisions
○ Small, replaceable components
○ Non-destructive computations
○ Sharing without risk
● One way: privacy architecture
Expensive
to reverse
Easy to
reverse
● Data pipeline deployment evolution
○ Spotify 2014 - 2018
1. Self-contained jar file. Ad-hoc continuous deploy flow.
2. Docker container on VM pool
3. Docker container on Kubernetes
logs
CI
dev
env
o11y

42. www.scling.com
8th habit - find your voice
● Your business use cases define your needs
○ Not technology
○ Not vendors
○ Not trends
○ Not fear-of-missing-out on AI, blockchains, …
● How do non-big-tech companies find own path and data success?
○ Technology?
○ Consultants?
○ Grow organically?
■ Very little data success experience available on job market
○ New modes of collaboration?
42

43. www.scling.com
Scling - data-factory-as-a-service
43
Data value through collaboration
Customer
Data factory
Data platform & lake
data
domain
expertise
Value from data!
Rapid data
innovation
Learning by doing,
in collaboration

44. www.scling.com
Tech has massive impact on society
44
Product?
Supplier?
Employer?
Make an active
choice whether to
have an impact!
Cloud?