apidays LIVE Australia 2021 - Accelerating Digital September 15 & 16, 2021 Orbital Design and the Modern Data Stack Graeme Lewis, Chief Pipeline Officer & John Cosgrove, Chief Executive Officer at Lightfold

1. Lightfold
The Modern Data Stack
Apidays Presentation
September 2021

2. Lightfold looks at data differently
Not so long ago, we looked up and concluded the earth
was the centre of the universe. We’ve made the same
mistake with applications. But the truth is, its not how
you do things, its what you know about them…

3. Data is the new centre
Instead of functionality, Lightfold sees data as the centre
of the business system – especially your customer data.

4. The Customer Halo
We call this centre the ‘Customer Halo’. It encompasses all of what you know about your customers – who they are,
what they’ve purchased, their interactions, their engagement. It’s not just the core of the business… it is the business…

5. New York
JULY
Australia
SEPTEMBER
Singapore
APRIL
Helsinki & North
MARCH
Paris
DECEMBER
London
OCTOBER
Jakarta
FEBRUARY
Hong Kong
AUGUST
JUNE
India
MAY
Check out our API Conferences here
50+ events since 2012, 14 countries, 2,000+ speakers, 50,000+ attendees,
300k+ online community
Want to talk at one of our conferences?
Apply to speak here

6. Customer Halo types
Just like stars in the sky, individual Customer Halos are diverse; some businesses are just starting out, some have
powerful central engines that push data outwards. Others have become huge and sprawling; some are even multiple
cores in a constant state of interaction and tight orbit…

7. Orbiting this core is an application space
A diverse range of functional applications orbit the central Customer Halo – each distinct but each relying on the core
to remain anchored and synchronized. The cloud revolution has seen an explosion in this space – there are apps for
everything and at every scale.

8. But it’s the core that powers everything
Whatever shape the Customer Halo takes, that core of customer data is beaming out into the application space and
powering each part of the business. While each application consumes and reflects part of the Customer Halo, they are
each only a facet of the whole. This is why so many businesses struggle to find ’the Customer 360’…
It’s not just an application or an integration – it’s the data itself.

9. The power of data at light speed
Here’s the game-changer: the platforms that underpin this new model can be used to make replicating data between
the Customer Halo and your application layer trivial.
Data movements within cloud platforms are instant and cost-effective. Apps can be powered by up-to-the-minute data
from the Customer Halo and any changes the app is responsible for can be instantly reflected back. It’s not about
extracting, transforming and loading data anymore; it’s about sharing and replicating across a vast data mesh.
This is data architecture free of gravity and the heavy costs of launching into the cloud.
This is data at light speed.

10. Welcome to the new frontier
This new ‘orbital architecture’ is changing the rules of business
intelligence and customer analytics.
The agility layer has moved from apps to data and as a result, we’re
building solutions with clients that were science fiction only a year ago.
Welcome to the next generation of business intelligence.
Welcome to the Modern Data Stack.

11. ON-PREMISE
PUBLIC CLOUD
EXTRACT-LOAD
TRANSFORM
INSIGHTS
STORE
DISTRIBUTE
Welcome to the Modern Data Stack
Fivetran
ETL is dead. Long live ELT.
The new paradigm is to focus on highly efficient, set-and-forget pipelines that
can replicate data from all sources to the central repository.
Fivetran is supporting the biggest brands in the world by providing a pricing
model based on rows that change and pre-built connectors that deliver query-
ready data.
Snowflake
Revolutionary cloud data platform.
Dynamically scalable compute power separate from storage. Zero copy cloning
and no lock distribution. Built-in fail-safes, time-travel and enterprise grade
security. Snowflake is redefining what’s possible for data in the cloud.
We combine Snowflake with DBT to give you a toolbox to manage all your
transformations – including feature engineering – all within the same platform.
Tableau and more…
Pick your visual layer – Tableau, PowerBI – wherever your people work
Pick your data science tool.
Whatever your application, Fivetran and Snowflake are ready for it.

12. Who we are

13. Lightfold is a specialist cloud analytics consultancy.
We believe in harnessing the speed to value offered
by modern cloud platforms by providing fixed price
engagements that are laser focused on producing
meaningful and valuable outcomes.
Our team has experience working with the biggest
brands around the world as well as smaller businesses
doing cool things here at home. We work closely with
the Salesforce, Snowflake, and Fivetran product teams
to shape how their platforms evolve and keep close to
the latest developments.

14. Traditional consulting is broken
We think the traditional ‘time and materials’ model of consultancy is broken.
It incentivises land and expand, encourages absurd rigidity of action, and
puts adherence to the letter of the agreement over the successful delivery
of outcomes. We fundamentally disagree with it as the way to deliver the
best experience and outcomes for our customers.
That’s why everything we do is outcome based and fixed price. We get
things done.
This approach opens Lightfold up to risks that other consultancies and
implementation partners don’t have to face, but we stand behind this
methodology as the only way we’ve found to do consulting right, and our
experience has been that our customers love it.
We’d be happy to put you in touch with some of our other clients, if you’re
keen to hear from them directly.
Photo by Parrish Freeman on Unsplash

15. How we work
We work remotely, like everyone else! The difference is
we want your project to be a great experience from
contract sign on to build sign off.
Our entire team is onboarded onto every project, so
your assigned engineer works alongside each of our
leads in:
• Executive: Account management
• Technical: Architecture and data management
• Producer: Project governance and tactical
management
• Design: Tesseract methodology and asset design
We’ll create a dedicated project channel on Slack where
you can engage with us, unmetered and as required.

16. Tesseract is our design methodology that
accelerates speed to value. It brings
together human-centered design, agile
and value proposition design to create a
unique methodology focused on building
meaningful data applications for users.
It focuses on understanding the complex
and dynamic relationships between duties
(people), assets on hand (data) and how
people do their work (actions) ensures
that applications enrich and support
individuals as well as the business.
Introducing Tesseract
Greek tessares four + aktis ray

17. Tesseract toolkit
Each component of Tesseract is geared to unpick one
aspect of the relationship between data, people and
action in order to create a successful data application.
The composition of a Tesseract engagement is entirely
flexible, it responds to your capability and requirements.
All components share the same objectives and trace the
same inputs through to a final design – so you have
visibility and consistency in every decision and every
output.
© Copyright Lightfold 2020
Drivers
Personas
Propositions
People
3V Data Assessment
Definitional Framework
Security
Data
User Flow
Action Architecture
Action
Value Map
MVP
Path to Production
Studio Governance
Application

18. What we build

19. Modular. Accurate. Rugged. Orbital.
Lightfold believes that the goal of modern cloud data architecture is to
provide the benefits of traditional data warehousing through new
mediums, capabilities and scale.
But orbital architectures – structures that work without the limitations of
on-premise solutions – look different to the warehouses of the past two
decades. We don’t have to choose between size of data and speed of
data.
In zero-gravity, big doesn’t mean heavy and everything moves very fast.
With the move to ELT instead of ETL and the ability to perform any level
of data transformation within the platform on demand, there are logical
patterns that are emerging as the next evolution of data warehousing.
The risk is that clients just lift their legacy solution into orbit without re-
imagining how that solution could be structured to fully unlock the value
of modern cloud technology.
That means we need a plan to build a scalable, modular and robust
architecture in orbit.

20. It starts with Fivetran
Before we can properly build in orbit, we need a rock solid way to make sure
everything gets where it needs to go. That’s where Fivetran comes in.
Fivetran has over 150 out-of-the-box connectors for sources ranging from
social media and advertising through to on-premise databases, and new
connectors are being added all the time. These connectors are designed and
maintained using the very best practices and, as a result, Fivetran provide
excellent data SLAs.
If we apply the orbital development rubric of Modular, Accurate, and Rugged
to Fivetran, we can see that it fits extremely well into the Modern Data Stack.
• Modular – Adding new connectors is trivial, as is adding new tables within
existing connectors. As the data estate grows, Fivetran can handle vast
quantities of new data replication with virtually zero spin up time.
• Accurate – Delta only data replication powered by best practice
development techniques ensures that Fivetran connectors are
consistently behaving accurately and efficiently.
• Rugged – Automated schema change management means that every
change made to the source schema is immediately and losslessly
replicated in your data repository of record.

21. Why Schema Matters
Data warehousing has been evolving for thirty years. Bill
Inmon called out the need for a true warehouse – that is, a
logistics hub function that can serve as the exchange point for
incoming data and outgoing information – in the 90’s. The
problem is, solutions in the past have been predominantly
about the limits of tech on-premise.
In the cloud, intelligence stacks are liberated from the
performance constraints of traditional solutions and can far
exceed the traditional limitations of BI.
However, a mistake many businesses make when transitioning
to cloud intelligence is under-investment in the logical
structure – or schema – of each layer in the new stack. While
it’s absolutely possible to just dump raw data into Snowflake
and then run lengthy SQL queries against it ad-hoc to power a
Tableau dashboard, this approach lacks rigour, reliability and
auditability, which ultimately means it won’t scale.
We need to implement appropriately structured solutions at
each layer of the stack – from raw data to insights.
10101001010100101
10101001010100101
10101001010100101
STAGING LOADS
DATA VAULT
(WAREHOUSE SCHEMA)
ANALYTIC
SCHEMAS
(DATA MARTS)
STREAM/API
DIMENSIONAL
DENORMALISED
INSIGHTS
DASHBOARDS
𝑥𝑥 + 𝑎𝑎 𝑛𝑛 = �
𝑘𝑘=0
𝑛𝑛
𝑛𝑛
𝑘𝑘
𝑥𝑥𝑘𝑘𝑎𝑎𝑛𝑛−𝑘𝑘
MACHINE LEARNING
RAW DATA

22. Data Vault – All the data, All the time
One pattern which is now experiencing an explosive growth in
utilisation because of how well it performs in orbit is the Data
Vault.
This hyper-modular pattern only allows one type of DML
function – insert. Data isn’t deleted and it isn’t overwritten.
It’s stored forever and it enforces rigorous connectivity at the
business key and timestamp level. It’s an air-tight logical
model when implemented correctly and it’s designed to work
in massive parallel processing (MPP) exactly like Snowflake.
The model is based around three key table entities:
• Hubs – These identify business logical entities and store
their primary business keys, acting as both a data
integration point and a form of MDM
• Satellites – These attach to Hubs and store all of the
attribute data from every source system pertinent to that
Hub
• Links – These connect hubs together to track relationships
or junction objects over time
We believe it could change the way insight is managed at
Grill’d and provide a blueprint for growing your orbital
foothold into a full-fledged data frontier.
APPLICATION
SOURCE_1
CUSTOMER
SOURCE_2
CONFORMED
SAME
PRODUCT
LOAN
SOURCE_3
LINK
POOL
CONFORMED
SOURCE_1
SOURCE_3
CONFORMED
SOURCE_1
SOURCE_2
CONFORMED

23. Thank you
If you have any further questions please contact us at
customersuccess@lightfold.com.au

24. New York
JULY
Australia
SEPTEMBER
Singapore
APRIL
Helsinki & North
MARCH
Paris
DECEMBER
London
OCTOBER
Jakarta
FEBRUARY
Hong Kong
AUGUST
JUNE
India
MAY
Check out our API Conferences here
50+ events since 2012, 14 countries, 2,000+ speakers, 50,000+ attendees,
300k+ online community
Want to talk at one of our conferences?
Apply to speak here