Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Semantic Image Logging Using Approximate Statistics & MLflow

Download to read offline

As organizations launch complex multi-modal models into human-facing applications, data governance becomes both increasingly important, and difficult. Specifically, monitoring the underlying ML models for accuracy and reliability becomes a critical component of any data governance system. When complex data, such as image, text and video, is involved, monitoring model performance is particularly problematic given the lack of semantic information. In industries such as health care and automotive, fail-safes are needed for compliant performance and safety but access to validation data is in short supply, or in some cases, completely absent. However, to date, there have been no widely accessible approaches for monitoring semantic information in a performant manner.
In this talk, we will provide an overview of approximate statistical methods, how they can be used for monitoring, along with debugging data pipelines for detecting concept drift and out-of-distribution data in semantic-full data, such as images. We will walk through an open source library, whylogs, which combines Apache Spark and novel approaches to semantic data sketching. We will conclude with practical examples equipping ML practitioners with monitoring tools for computer vision, and semantic-full models.

  • Be the first to like this

Semantic Image Logging Using Approximate Statistics & MLflow

  1. 1. Semantic image logging with approximate statistical methods & MLflow Leandro G. Almeida, PhD
  2. 2. Four steps to image logging • Scaling to real-world datasets with approximate statistics • Logging in ML applications • Logging semantic image data
  3. 3. Approximate Statistics • approximate distribution • Quantiles ( min, max, .. ) • Std-dev • Count • Type counts • Top k frequent items Constant memory footprint!
  4. 4. whylogs Minimal Setup Start logging in 4 lines of code github.com/whylabs/whylogs
  5. 5. Even easier with
  6. 6. 6 Spark-powered scaling
  7. 7. Three steps to image logging • Logging in ML applications • Logging semantic image data • Scaling to real-world datasets with approximate statistics • Why (to) Log ? • How (to) Log ? • What (to) Log ?
  8. 8. Why (to) Log ? Testing doesn’t stop at the test set.
  9. 9. Why (to) Log ? Monitoring Deployments • Data drift • Model drift • Concept drift • Domain shift • Head to Tail drift
  10. 10. Why (to) Log ? Monitoring Deployments • Data drift • Model drift • Concept drift • Domain shift • Head to Tail drift • Input Data is inherently different • Feedback Loop where model affects user behavior • Target Properties change over time • Biased Dataset • Tasks based on the relevance of outliers
  11. 11. What (to) Log ?
  12. 12. What (to) Log ? • Inputs/Outputs • Task Metrics • Perfomance Metrics
  13. 13. What (to) Log ? • Meta Data • Device • Encoding • Raw Resolution • Aspect Ratio • Features distributions • Quality Based • Engineered • Outputs • Semantic • Inputs/Outputs • Task Metrics • Perfomance Metrics
  14. 14. What (to) Log ? • File Meta Data • Device • Encoding • Raw Resolution • Aspect Ratio • Inputs/Outputs • Task Metrics • Perfomance Metrics
  15. 15. What (to) Log ? • Features distributions • IQA • Engineered • Learned • Outputs • Embeddings
  16. 16. What (to) Log ? • Features distributions • IQA • Engineered • Learned • Outputs • Embeddings Reference Set (Baseline) Current Image or Set
  17. 17. What (to) Log ? • Features distributions • IQA • Engineered • Learned • Outputs (image based) • Embeddings Current Image or Set Reference Set (Baseline)
  18. 18. What (to) Log ? Current Image or Set Reference Set (Baseline)
  19. 19. What (to) Log ? Current Image or Set Pair Distance dij: over entire dataset or per cluster Distance from each cluster center (closest concentre embedding) C1 C2 C3 Cn C4 …
  20. 20. What (to) Log ? • Features distributions • IQA • Engineered • Learned • Outputs (non images) • Embeddings Current Image or Set
  21. 21. Four Steps • Scaling to real-world datasets with approximate statistics • Approximate Statistics • Logging in ML applications • Logging semantic image data
  22. 22. 22 Spark-powered scaling
  23. 23. 23 Try today & contribute bit.ly/whylogs
  24. 24. Thank you! leandro@whylabs.ai @lalmei 24 bit.ly/whylogs

As organizations launch complex multi-modal models into human-facing applications, data governance becomes both increasingly important, and difficult. Specifically, monitoring the underlying ML models for accuracy and reliability becomes a critical component of any data governance system. When complex data, such as image, text and video, is involved, monitoring model performance is particularly problematic given the lack of semantic information. In industries such as health care and automotive, fail-safes are needed for compliant performance and safety but access to validation data is in short supply, or in some cases, completely absent. However, to date, there have been no widely accessible approaches for monitoring semantic information in a performant manner. In this talk, we will provide an overview of approximate statistical methods, how they can be used for monitoring, along with debugging data pipelines for detecting concept drift and out-of-distribution data in semantic-full data, such as images. We will walk through an open source library, whylogs, which combines Apache Spark and novel approaches to semantic data sketching. We will conclude with practical examples equipping ML practitioners with monitoring tools for computer vision, and semantic-full models.

Views

Total views

64

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

4

Shares

0

Comments

0

Likes

0

×