Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Data ops in practice Slide 1

YouTube videos are no longer supported on SlideShare

View original on YouTube

Data ops in practice Slide 3 Data ops in practice Slide 4 Data ops in practice Slide 5 Data ops in practice Slide 6 Data ops in practice Slide 7 Data ops in practice Slide 8 Data ops in practice Slide 9 Data ops in practice Slide 10 Data ops in practice Slide 11 Data ops in practice Slide 12 Data ops in practice Slide 13 Data ops in practice Slide 14 Data ops in practice Slide 15 Data ops in practice Slide 16 Data ops in practice Slide 17 Data ops in practice Slide 18 Data ops in practice Slide 19 Data ops in practice Slide 20 Data ops in practice Slide 21 Data ops in practice Slide 22 Data ops in practice Slide 23 Data ops in practice Slide 24 Data ops in practice Slide 25 Data ops in practice Slide 26 Data ops in practice Slide 27
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

9

Share

Download to read offline

Data ops in practice

Download to read offline

DataOps is a methodology and culture shift that brings the successful combination of development and operations (DevOps) to data processing environments. It breaks down silos between developers, data scientists, and operators, resulting in lean data feature development processes with quick feedback. In this presentation, we will explain the methodology, and focus on practical aspects of DataOps.

Related Books

Free with a 30 day trial from Scribd

See all

Data ops in practice

  1. 1. www.mimeria.com DataOps in practice 2019-03-14 Lars Albertsson Mimeria 1
  2. 2. www.mimeria.com Friction to production 2 Which NoSQL database would you recommend for this use case? It won’t matter. Use something that you are used to. MySQL, Oracle? Well, I’d prefer something else. ? If we use an RDBMS, Ops have rules and opinions that slow us down.
  3. 3. www.mimeria.com Distance to production 3 What are your data scientists up to? They have received a data dump and built a model. We will hand it off to a developer team, who hands it off to operations when the model is translated to Java. Great, that is the first 1%. What’s next?
  4. 4. www.mimeria.com Risky operations 4 How to I test the pipeline? You temporarily change the output path and run manually. Don’t do that. What if I forget to change path?
  5. 5. www.mimeria.com Disrupted or disruptor 5 Operational risk Silos Friction
  6. 6. www.mimeria.com Digital revolution steam engines 6 Data ML AI Internet Web Cyber- Micro- processor PC AI
  7. 7. www.mimeria.com Properties of disruptors 7 Sustained ROI from machine learning Short time from idea to production Homogeneous data platform Data internally democratised Major cloud + open source Organisation aligned by use case Data processing pipelines Diverse teams: data science, dev, ops
  8. 8. www.mimeria.com DataOps 8 Sustained ROI from machine learning Short time from idea to production Homogeneous data platform Data internally democratised Major cloud + open source Organisation aligned by use case Data processing pipelines Diverse teams: data science, dev, ops Purpose Context Means Building data processing software Running data processing software
  9. 9. www.mimeria.com Big data - a collaboration paradigm 9 Stream storage Data lake Data democratised
  10. 10. www.mimeria.com Data pipelines 10 Data lake
  11. 11. www.mimeria.com Nearline ● Stream storage (Kafka) ● Asynchronous event processing ● 10 ms - 1 hour Data integration timescales 11 Job Stream Offline ● File storage (Hadoop) ● Asynchronous batch processing ● 10 minutes - Online ● SOA / microservices ● Synchronous RPC ● 1-100 ms Stream Job Stream
  12. 12. www.mimeria.com Upgrade ● Careful rollout ● Risk of user impact ● Proactive QA Operational manoeuvres - online 12 Service failure ● User impact ● Data loss ● Cascading outage Bug ● User impact ● Data corruption ● Cascading corruption
  13. 13. www.mimeria.com Data platform overview 13 Data lake Cold store Service Service Online services Offline data platform Batch processing
  14. 14. www.mimeria.com Data platform overview 14 Data lake Cold store Dataset Job Service Service Online services Offline data platform Batch processing
  15. 15. www.mimeria.com Data platform overview 15 Data lake Cold store Dataset Pipeline Service Service Online services Offline data platform Job Workflow orchestration (Luigi, Airflow) Online services Service Data feature
  16. 16. www.mimeria.com Operational manoeuvres - offline 16 Upgrade ● Instant rollout ● No user impact ● Reactive QA Service failure ● Pipeline delay ● No data loss ● No downstream impact Bug ● Temporary data corruption ● Downstream impact
  17. 17. www.mimeria.com Production critical upgrade 17 ● Dual datasets during transition ● Run downstream parallel pipelines ○ Cheap ○ Low risk ○ Easy rollback ● Easy to test end-to-end ○ Upstream team can do the change No dev & staging environment needed! ∆?
  18. 18. www.mimeria.com Life of an error, batch pipelines 18 ● Faulty job, emits bad data 1. Revert serving datasets to old 2. Fix bug 3. Remove faulty datasets 4. Backfill is automatic (Luigi) Done! ● Low cost of error ○ Reactive QA ○ Production environment sufficient
  19. 19. www.mimeria.com Operational manoeuvres - nearline 19 Upgrade ● Swift rollout ● Parallel pipelines ● User impact, QA? Service failure ● Pipeline delay ● No data loss ● Downstream impact? Bug ● Data corruption ● Downstream impact Job Stream Stream Job Stream Job Stream Stream Job Stream Job Stream Stream Job Stream
  20. 20. www.mimeria.com Life of an error, streaming 20 ● Works for a single job, not pipeline. :-( Job StreamStream Stream Stream Stream Stream Job Job Stream Stream Stream Job Job Job Reprocessing in Kafka Streams
  21. 21. www.mimeria.com Deployment example, cloud native 21 source repo Luigi DSL, jars, config my-pipe:7 Luigi daemon Worker Worker Worker Worker Worker Worker Worker Worker Redundant cron schedule, higher frequency kind: CronJob spec: schedule: "10 * * * *" command: "luigi --module mymodule MyDaily" Docker image Docker registry S3 / GCS Dataproc / EMR
  22. 22. www.mimeria.com Monitoring timeliness, examples ● Datamon - Spotify internal ● Twitter Ambrose (dead?) ● Airflow 22
  23. 23. www.mimeria.com 23 Measuring correctness: counters ● Processing tool (Spark/Hadoop) counters ○ Odd code path => bump counter ○ System metrics Hadoop / Spark counters DB Standard graphing tools Standard alerting service
  24. 24. www.mimeria.com 24 Measuring correctness: pipelines ● Processing tool (Spark/Hadoop) counters ○ Odd code path => bump counter ○ System metrics ● Dedicated quality assessment pipelines DB Quality assessment job Quality metadataset (tiny) Standard graphing tools Standard alerting service
  25. 25. www.mimeria.com 25 Machine learning operations ● Multiple trained models ○ Select at run time ● Measure user behaviour ○ E.g. session length, engagement, funnel ● Ready to revert to ○ old models ○ simpler models Measure interactionsRendez- vous DB Standard alerting service Stream Job
  26. 26. www.mimeria.com Nearline Data processing tradeoff 26 Job Stream OfflineOnline Stream Job Stream Data speed Innovation speed
  • RichieSiburian

    Sep. 10, 2021
  • bysimon

    Feb. 18, 2020
  • KapildevSingh7

    Jan. 10, 2020
  • RyanGross1

    May. 20, 2019
  • rpaskev

    May. 1, 2019
  • hmesha

    Apr. 16, 2019
  • jalbertbowden

    Apr. 11, 2019
  • AmaIbe

    Apr. 3, 2019
  • datakitchen99

    Mar. 15, 2019

DataOps is a methodology and culture shift that brings the successful combination of development and operations (DevOps) to data processing environments. It breaks down silos between developers, data scientists, and operators, resulting in lean data feature development processes with quick feedback. In this presentation, we will explain the methodology, and focus on practical aspects of DataOps.

Views

Total views

2,140

On Slideshare

0

From embeds

0

Number of embeds

30

Actions

Downloads

110

Shares

0

Comments

0

Likes

9

×