We've updated our privacy policy. Click here to review the details. Tap here to review the details.
Activate your 30 day free trial to unlock unlimited reading.
Activate your 30 day free trial to continue reading.
Download to read offline
It is not fast enough! That is one of the more common responses to a data engineer when putting a data pipeline in production. It is easy to dig down into the code and try to optimize it. My experience as a data engineer shows me that it is often easier and more efficient, both in time spent and outcome, to focus on a more holistic view of the pipeline.
In this talk, we will look at a structured process to optimize our batch pipelines. We will introduce steps that make our process data-driven instead of a gut feeling. With examples from real-world cases where delivery time was reduced in order by magnitude, we will look at actions where taken.
The intended audience is a beginner to intermediate data engineers. After the talk, you will have a better understanding of how to optimize your pipeline and be able to explain the steps taken for a stakeholder. You will know:
* what metrics to look at
* how to visualize the metrics
* how to detect bottlenecks and other time thieves from the metrics
* what actions to take.
It is not fast enough! That is one of the more common responses to a data engineer when putting a data pipeline in production. It is easy to dig down into the code and try to optimize it. My experience as a data engineer shows me that it is often easier and more efficient, both in time spent and outcome, to focus on a more holistic view of the pipeline.
In this talk, we will look at a structured process to optimize our batch pipelines. We will introduce steps that make our process data-driven instead of a gut feeling. With examples from real-world cases where delivery time was reduced in order by magnitude, we will look at actions where taken.
The intended audience is a beginner to intermediate data engineers. After the talk, you will have a better understanding of how to optimize your pipeline and be able to explain the steps taken for a stakeholder. You will know:
* what metrics to look at
* how to visualize the metrics
* how to detect bottlenecks and other time thieves from the metrics
* what actions to take.
You just clipped your first slide!
Clipping is a handy way to collect important slides you want to go back to later. Now customize the name of a clipboard to store your clips.The SlideShare family just got bigger. Enjoy access to millions of ebooks, audiobooks, magazines, and more from Scribd.
Cancel anytime.Unlimited Reading
Learn faster and smarter from top experts
Unlimited Downloading
Download to take your learnings offline and on the go
You also get free access to Scribd!
Instant access to millions of ebooks, audiobooks, magazines, podcasts and more.
Read and listen offline with any device.
Free access to premium services like Tuneln, Mubi and more.
We’ve updated our privacy policy so that we are compliant with changing global privacy regulations and to provide you with insight into the limited ways in which we use your data.
You can read the details below. By accepting, you agree to the updated privacy policy.
Thank you!