Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
Andreu Mora, Adyen
Time series forecasting
and monitoring with
Apache Spark and
ElasticSearch
#UnifiedDataAnalytics #Spark...
Adyen
Payments Processor
Tech company
International customers (aka merchants)
Omnichannel
Back in the day…
The legacy monitor was based on a SQL query that
would compute an average for the hour of the week
and co...
Doesn’t quite work:
• Generates loads of False Positives
• It was fairly trimmed down: top merchants.
Reduce False
Positives
Catch anomalies
Do that at scale
Harness the
detection
performance
Connect to a live
platform
OK, but
What is an anomaly?
No luxury of a labelled dataset, divergence 

of opinions.
Connecting to a live platform witho...
Considerations when dealing with Big Data
Big Technology
Leverage on mature Tech to
solve the problem (hello Spark).
Big d...
Big Data Platform
Volumes Predictions
Big Data Platform
Volumes Predictions
TimeSeries Ecosystem
Flint
Spark-ts
FB Prophet
Stats models
TimeSeries Ecosystem
Flint
Spark-ts
FB Prophet
Stats models
Data size
consideration
1 year @ 1 min @ double64 = 4.2 mb
Scoring in Java
While working on a fully functional engine to
deploy ML models based on MLflow.
Launch fast and iterate!
Tr...
Big Data Platform
Volumes Predictions
Big Data Platform
Volumes
Model
Coefficients
Fourier
components
Would not
optimise the
business cycles
ARIMA
Not perfect for
picking up
seasonality
Isolation Forests
G...
Ridge Regression
Makes scoring in Java nice and
kinda easy.
Residuals
Confidence intervals modelled
through quantile regres...
Ridge Regression
Makes scoring in Java nice and
kinda easy.
Residuals
Confidence intervals modelled
through quantile regres...
Ridge Regression
Makes scoring in Java nice and
kinda easy.
Residuals
Confidence intervals modelled
through quantile regres...
Ridge Regression
Makes scoring in Java nice and
kinda easy.
Residuals
Confidence intervals modelled
through quantile regres...
Ridge Regression
Makes scoring in Java nice and
kinda easy.
Residuals
Confidence intervals modelled
through quantile regres...
Ridge Regression
Makes scoring in Java nice and
kinda easy.
Residuals
Confidence intervals modelled
through quantile regres...
Ridge Regression
Makes scoring in Java nice and
kinda easy.
Residuals
Confidence intervals modelled
through quantile regres...
Ridge Regression
Makes scoring in Java nice and
kinda easy. easy
Residuals
Confidence intervals modelled
through quantile r...
Trendspotting
Estimating hinges and trends and offering it as
subproduct to Account Managers for evaluating
the low variat...
Train set: 90 days
Test set: 7 days
Real volume
Predicted volume
95% confidence
How do the predictions look like?
Missed event
The implementation
on Spark
How did we get there, on the Spark side.
Reusability
Overloads of scikit-learns and pandas all...
SeasonalEstimator(BaseEstimator,RegressorMixin)
Input daily time series —> {t:[…], v:[…]}
Collect to list —> [{t:[…], v:[…]}]
Hinges and Hyperparameters
Distribute UDF
Ma...
Cross-validation
F4-sampling score: favours higher sampling
considering classical precision and recall.
Custom cv folds sp...
The output
Harnessing 

the prediction
performance
Enabling canary
roll-out based on
scores
Overcoming
unsupervised
learning
Alarm rate and synthetic recall allow us to
know for each case how many alarms would
have...
Trade-off alarm
rates and recall
We provide a number of choices (95%, 97%,
99% probability and completely profile what to
e...
The model payload
Go Live
Houston? Houston? …
Grafana dashboard
So we saw this on the data
’You don’t call us, we call you’
Post on Medium
https://medium.com/adyen
DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT
Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye
Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye
Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye
Upcoming SlideShare
Loading in …5
×

of

Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 1 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 2 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 3 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 4 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 5 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 6 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 7 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 8 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 9 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 10 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 11 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 12 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 13 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 14 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 15 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 16 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 17 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 18 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 19 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 20 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 21 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 22 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 23 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 24 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 25 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 26 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 27 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 28 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 29 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 30 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 31 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 32 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 33 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 34 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 35 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 36 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 37 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 38 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 39 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 40 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 41 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 42 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 43 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 44 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 45 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 46 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 47 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 48 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 49 Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye Slide 50
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

3 Likes

Share

Download to read offline

Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye

Download to read offline

Adyen enables integrating companies to accept payments from their customers using any payment method over any sales channel. We have designed and implemented a time series forecasting algorithm that allows us to predict the volume for each integration with confidence and thus be able to flag anomalies such as traffic drop or abnormally low traffic. We are using Apache Spark as our computational engine both to make this data available to the training process as well as to train over years of data in a scalable way. The prediction performances are benchmarked and the models are served in production through custom real-time monitoring and alerting infrastructure that uses ElasticSearch as hot storage. With this state-of-the-art solution, Adyen knows whether a problem happened and can alert the operational teams accordingly in a record time.

‘This presentation will cover the journey we took with focus on the mathematical concepts, the present time constraints, the prediction performances, and the architecture needed to make this happen. We’ll go over lessons learned, pitfalls, and best practices discovered on modeling time series datasets with Apache Spark. Data Scientists would be able to gain insights on applying effective and real-life seasonality modeling techniques. We’ll share our approaches used for sub-millisecond model serving that would inspire Data Engineers who work on related problems.

Scalable Time Series Forecasting and Monitoring using Apache Spark and ElasticSearch at Adye

  1. 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
  2. 2. Andreu Mora, Adyen Time series forecasting and monitoring with Apache Spark and ElasticSearch #UnifiedDataAnalytics #SparkAISummit
  3. 3. Adyen Payments Processor Tech company International customers (aka merchants) Omnichannel
  4. 4. Back in the day… The legacy monitor was based on a SQL query that would compute an average for the hour of the week and compare to a threshold.
  5. 5. Doesn’t quite work: • Generates loads of False Positives • It was fairly trimmed down: top merchants.
  6. 6. Reduce False Positives
  7. 7. Catch anomalies
  8. 8. Do that at scale
  9. 9. Harness the detection performance
  10. 10. Connect to a live platform
  11. 11. OK, but What is an anomaly? No luxury of a labelled dataset, divergence 
 of opinions. Connecting to a live platform without 
 ML deployment hooks ready. We were working on MLflow but not there yet. No standard for timeseries forecasting at scale With spark, several choices.
  12. 12. Considerations when dealing with Big Data Big Technology Leverage on mature Tech to solve the problem (hello Spark). Big diversity Many different topologies for our merchants and yet one algorithm to track them all. Big consequences 1000 merchants * 10 min * 95% accuracy = 50400 emails/week
  13. 13. Big Data Platform Volumes Predictions
  14. 14. Big Data Platform Volumes Predictions
  15. 15. TimeSeries Ecosystem Flint Spark-ts FB Prophet Stats models
  16. 16. TimeSeries Ecosystem Flint Spark-ts FB Prophet Stats models Data size consideration 1 year @ 1 min @ double64 = 4.2 mb
  17. 17. Scoring in Java While working on a fully functional engine to deploy ML models based on MLflow. Launch fast and iterate! Transporting the model The model transported for tens of thousands of accounts needs to be lightweight. Harness the maths No using blackboxed models, equations need to be understood and replicated in Java. Needs to perform fast Score and decide whether our seen traffic form ElasticSearch is actually anomalous on the ms scale.
  18. 18. Big Data Platform Volumes Predictions
  19. 19. Big Data Platform Volumes Model Coefficients
  20. 20. Fourier components Would not optimise the business cycles ARIMA Not perfect for picking up seasonality Isolation Forests Great for multidimensional data, not so much for time series. Autoencoders Good luck transporting the model for each merchant. XGBM Noice, but score that in Java. Research stage Understand a problem and build a solution, decide what’s best.
  21. 21. Ridge Regression Makes scoring in Java nice and kinda easy. Residuals Confidence intervals modelled through quantile regression of observed values. Events Recurrent or one-off events are shown to the model. Piece-wise linear trends Breaks down the signal into pieces and learn the last trends. Gaussian Basis Functions Allow us to teach the model to understand business cycles The model Discover anomalous behaviour based on a probability p. Pre-sampling Allow us to sample and bucketize the merchants to adequate intervals.
  22. 22. Ridge Regression Makes scoring in Java nice and kinda easy. Residuals Confidence intervals modelled through quantile regression of observed values. Events Recurrent or one-off events are shown to the model. Piece-wise linear trends Breaks down the signal into pieces and learn the last trends. Gaussian Basis Functions Allow us to teach the model to understand business cycles The model Discover anomalous behaviour based on a probability p. Pre-sampling Allow us to sample and bucketize the merchants to adequate intervals.
  23. 23. Ridge Regression Makes scoring in Java nice and kinda easy. Residuals Confidence intervals modelled through quantile regression of observed values. Events Recurrent or one-off events are shown to the model. Piece-wise linear trends Breaks down the signal into pieces and learn the last trends. Gaussian Basis Functions Allow us to teach the model to understand business cycles The model Discover anomalous behaviour based on a probability p. Pre-sampling Allow us to sample and bucketize the merchants to adequate intervals.
  24. 24. Ridge Regression Makes scoring in Java nice and kinda easy. Residuals Confidence intervals modelled through quantile regression of observed values. Events Recurrent or one-off events are shown to the model. Piece-wise linear trends Breaks down the signal into pieces and learn the last trends. Gaussian Basis Functions Allow us to teach the model to understand business cycles The model Discover anomalous behaviour based on a probability p. Pre-sampling Allow us to sample and bucketize the merchants to adequate intervals.
  25. 25. Ridge Regression Makes scoring in Java nice and kinda easy. Residuals Confidence intervals modelled through quantile regression of observed values. Events Recurrent or one-off events are shown to the model. Piece-wise linear trends Breaks down the signal into pieces and learn the last trends. Gaussian Basis Functions Allow us to teach the model to understand business cycles The model Discover anomalous behaviour based on a probability p. Pre-sampling Allow us to sample and bucketize the merchants to adequate intervals.
  26. 26. Ridge Regression Makes scoring in Java nice and kinda easy. Residuals Confidence intervals modelled through quantile regression of observed values. Events Recurrent or one-off events are shown to the model. Piece-wise linear trends Breaks down the signal into pieces and learn the last trends. Gaussian Basis Functions Allow us to teach the model to understand business cycles The model Discover anomalous behaviour based on a probability p. Pre-sampling Allow us to sample and bucketize the merchants to adequate intervals.
  27. 27. Ridge Regression Makes scoring in Java nice and kinda easy. Residuals Confidence intervals modelled through quantile regression of observed values. Events Recurrent or one-off events are shown to the model. Piece-wise linear trends Breaks down the signal into pieces and learn the last trends. Gaussian Basis Functions Allow us to teach the model to understand business cycles The model Discover anomalous behaviour based on a probability p. Pre-sampling Allow us to sample and bucketize the merchants to adequate intervals.
  28. 28. Ridge Regression Makes scoring in Java nice and kinda easy. easy Residuals Confidence intervals modelled through quantile regression of observed values. Events Recurrent or one-off events are shown to the model. Piece-wise linear trends Breaks down the signal into pieces and learn the last trends. Gaussian Basis Functions Allow us to teach the model to understand business cycles The model Discover anomalous behaviour based on a probability p. Pre-sampling Allow us to sample and bucketize the merchants to adequate intervals.
  29. 29. Trendspotting Estimating hinges and trends and offering it as subproduct to Account Managers for evaluating the low variations of volume.
  30. 30. Train set: 90 days Test set: 7 days Real volume Predicted volume 95% confidence How do the predictions look like?
  31. 31. Missed event
  32. 32. The implementation on Spark How did we get there, on the Spark side. Reusability Overloads of scikit-learns and pandas allow us to ensure reusability Cross-validation Ensure the best tuning through tuning of hyperparameters. Scalability Using Spark’s map-reduce paradigm we totally control the computational performances.
  33. 33. SeasonalEstimator(BaseEstimator,RegressorMixin)
  34. 34. Input daily time series —> {t:[…], v:[…]} Collect to list —> [{t:[…], v:[…]}] Hinges and Hyperparameters Distribute UDF Making it happen at scale
  35. 35. Cross-validation F4-sampling score: favours higher sampling considering classical precision and recall. Custom cv folds split TimeSeriesWeekSplit get the sense of the business cycle
  36. 36. The output
  37. 37. Harnessing 
 the prediction performance
  38. 38. Enabling canary roll-out based on scores
  39. 39. Overcoming unsupervised learning Alarm rate and synthetic recall allow us to know for each case how many alarms would have been captured and raised, even without having a labelled dataset.
  40. 40. Trade-off alarm rates and recall We provide a number of choices (95%, 97%, 99% probability and completely profile what to expect in terms of anomalies.
  41. 41. The model payload
  42. 42. Go Live Houston? Houston? …
  43. 43. Grafana dashboard
  44. 44. So we saw this on the data
  45. 45. ’You don’t call us, we call you’
  46. 46. Post on Medium https://medium.com/adyen
  47. 47. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT
  • VijayGharge1

    Oct. 4, 2020
  • dalgual

    Apr. 1, 2020
  • keitaonabuta

    Nov. 22, 2019

Adyen enables integrating companies to accept payments from their customers using any payment method over any sales channel. We have designed and implemented a time series forecasting algorithm that allows us to predict the volume for each integration with confidence and thus be able to flag anomalies such as traffic drop or abnormally low traffic. We are using Apache Spark as our computational engine both to make this data available to the training process as well as to train over years of data in a scalable way. The prediction performances are benchmarked and the models are served in production through custom real-time monitoring and alerting infrastructure that uses ElasticSearch as hot storage. With this state-of-the-art solution, Adyen knows whether a problem happened and can alert the operational teams accordingly in a record time. ‘This presentation will cover the journey we took with focus on the mathematical concepts, the present time constraints, the prediction performances, and the architecture needed to make this happen. We’ll go over lessons learned, pitfalls, and best practices discovered on modeling time series datasets with Apache Spark. Data Scientists would be able to gain insights on applying effective and real-life seasonality modeling techniques. We’ll share our approaches used for sub-millisecond model serving that would inspire Data Engineers who work on related problems.

Views

Total views

1,793

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

23

Shares

0

Comments

0

Likes

3

×