Lighthouse is an open-source toolkit for building data lakes. It provides utilities for defining data sources, constructing data pipelines, and consistently reading and writing data to the data lake. It uses Spark and supports ORC file format by default. The toolkit aims to make it easy to identify, reuse, and test data sources and transformations.

1. Lighthouse
an open-source toolkit to build data lakes
Mathias Lavaert and Kris Peeters

3. DAS WAR EIN BEFEHL!

4. Your code is
really poor
quality sir
Your test
coverage is
low sir
Configuration
in json? What
is this? 1923?
You didn’t even
define clear
interfaces sir

5. Can I at least help
build the new
version?

6. No
No No
Nope nope nope
nope nope nope
nope nope nope

7. Lighthouse is an open-source toolkit
to build data lakes

8. Define
Data lake
Construct
Data pipelines
Batteries
Included*
*pull requests welcome

9. 90% of big data projects
Welcome to my swamp

10. We strongly-typed formats and schemas
- ORC by default, but you are free to change it
- Use either datasets (with case classes) or dataframes
We uniquely identifying and reusing data sources
- Define once which data sources are in your data lake
- Define once where they are located and what they look like
- Reuse them as you please
We consistent reads and writes to the data lake
- Snapshot reads, partitioned reads from the filesystem
- Sync with Hive / Athena / … any kind of metastore
- Sync to JDBC …. because we also looooooove SQL

11. Please don’t touch
any of the wires

12. SOURCE
SINK
Magical transform 1
Magical transform 2
Magical transform 3
SOURCE
SINK
Magical transform 4
Magical transform 5
Magical transform 6
SINK
Magical transform 7
Magical transform 8
Each individual transform
is easy to test
Separate how to get the
data from what you do
with it
You can use the data lake
as source and sink but
that’s optional

13. Parameter store
- Store credentials
and config parameters
- Use AWS Parameter Store
or Filesystem Vault
Testing functions
- Compare datasets
- Pretty Print data frames
- Compare columns
- sparkTest() unit tests
Demo project
- Simple setup
- Data lake & data pipelines
- Copy&paste for your needs
Batteries included

14. Define
Data lake
Construct
Data pipelines
Batteries
Included*
*pull requests welcome
Lighthouse is an open-source toolkit
to build data lakes

16. What is the demo use case?
Aggregated view
Combining
Cleaning

17. When can you use it
Here’s when Lighthouse is useful
- If you have more than one project in Spark
that you need to implement AND has to run
in production
- If you have business analysts and data
scientists who want to explore and play with
all the data that is available in the
organisation
- If you don’t have a lot of experience yet
Here’s when you shouldn’t use Lighthouse:
- If you only have to run one very particular use
case. Typically startups. Lighthouse will
generate more overhead
- If you don’t have a large volume or variety of
data. Python + Postgres will do just fine
- If you are in prototyping / experimentation
mode, and there is no need yet to put stuff in
production

18. How can you use it
Integration Scheduling/workflows
RAW
A staging area
where the data
enters the data
lake
BASE
Basic building
blocks for all use
cases
MASTER
Business objects
which are agreed
on and shared
across the
enterprise
EXPORT
Data exported to
an external
system like a
RDMS

19. Feedback from community
Support multiple clouds
Data anonymization
Data portal

20. Lighthouse
an open-source toolkit to build data lakes
Mathias Lavaert and Kris Peeters
https://github.com/datamindedbe/lighthouse