The world of data is ever-growing. There are ample technologies out in the market. Through the session, we will see some beautiful open source tools in action. Well knowing tools is often the first step, but indeed an important one. This session is best suited for beginners, keen to dive into data world. Typical data engineering actions – -1) Connecting with data sources ⚡️ 0) Pre-processing 🤫 1) Ingestion 🛠 2) Business logic 🙋🏻‍♀️ 3) Dashboarding 👓 4) Lift and shifts 💪🏽 5) Quality control and statistical checks ☑️ * Everything else follows… Often each step adds a special tooling need. Open source is a great way to get started eliminating steep learning curves.

1. Data
Pipelines
Powered by Open Source
and Fun
Sayantika Banik (she/her)

2. Recipe to build data pipelines
Short demo (time check)
Intro to open source data
engineering tools
Takeaways
Overview
Agenda for today

3. Data Engineer @Quansight
Organiser @PyLadies Bangalore,
India
Project Incubator founding member
@NumFOCUS
D&I workgroup member @PSF
Open source contributor @Scipy
& NumPy
Crazy about abstract art, nature &
notes
sayantikabanik.com
@sayabanik

4. Internet in few seconds
source : everysecond.io

5. In the background 🤫
find the sources
keep pinging the source
is it big enough, to be called big data?
Volume, Velocity, Variety and Veracity
load in batch
stream it
orchestration
add partitioning
let's use this, this, this and more of that
clean the data
ETL
Data Engineering
lakes and warehouses
It's broken again!!!!
We forgot to add tests
metadata
hadoop
dask
Note: The colours are chosen randomly
with no meaning attached
missing data
spark

7. Building data pipelines in five
simple steps
Ingestion from
source
Quality control Business logic
1 2 3
Orchestrate
4
update, break &
continue
5

8. Tools that exists
Image credits: Link

10. Handy open source tools

11. Package, dependency
and environment
management for any
language—Python, R,
Ruby, Lua, Scala,
Java, JavaScript, C/
C++, Fortran, and
more.
https://docs.conda.io/
en/latest/

12. Intake is a
lightweight package
for finding,
investigating, loading
and disseminating
data.
https://github.com/int
ake/intake

13. GitHub Actions makes
it easy to automate
all your software
workflows, now with
world-class CI/CD.

14. Data orchestration
platform, comes with a
nice UI : Dagit
https://dagster.io/

15. Two main components
(covered as part of
this session)
- Ops are the core unit
of computation in
Dagster.
An individual op should
perform relatively
simple tasks, such as:
Deriving a dataset from
other datasets
- Jobs are the main
unit of execution and
monitoring in Dagster.
The core of a job is a
graph of ops connected
via data dependencies.
Launching Dagster Run

17. Repositiory
https://github.com/sayantikabanik/DataJourney

18. @sayabanik sayantikabanik.com

19. Have fun and keep
exploring