SlideShare a Scribd company logo

AWS Forcecast: DeepAR Predictor Time-series

•Download as PPTX, PDF•
•
PolarSeven Pty Ltd
PolarSeven Pty Ltd

Slides from AWS meetup presentation 4th September 2019.

Technology
1 of 12
Amplifying OrganisationalIntelligence Intellify Pty Ltd IntellifyAI Intellify_AISydney Level 8 11York Street Sydney, NSW 2000 T. (02) 8089 4073 www.intellify.com.au Melbourne Level 28 303 Collins Street Melbourne,VIC 3000 T. (03) 9132 9846 info@intellify.com.au 20 Bridge Street AWS Forecast: DeepAR Predictor Time-series
Amplifying OrganisationalIntelligence Agenda 1. Introduction 2. Why time-series and how are they different? 3. Classical (Statistical) and Modern time-series methods 4. AWS Forecast: Modern Time-series 5. AWS Predictor: DeepAR 6. Demo on DeepAR
Amplifying OrganisationalIntelligence Why are time-series methods important? 1 2 3 Time series are everywhere! Most methods were designed for use on cross-sectional data We can drive better business outcomes through the use of time-series methods
Amplifying OrganisationalIntelligence How are time series problems different? • Different states in a time series can make the problem harder to model. • There could be multiple forecasting horizons; short, medium, long term. • Typically you care about the prediction as well as the confidence in the prediction. • Model testing and validation must be conducted in a different way to avoid data leakage and select the best model.
Amplifying OrganisationalIntelligence What are desirable properties of time series methods? Multi-step multivariate prediction Shares information across time-series Leverages meta-information Works on sparse data Handles non-linearities/interactions Works with high dimensional data Models autocorrelation structure implicitly Minimal feature pre-processing and engineering
Amplifying OrganisationalIntelligence Traditional Models Autoregressive models are remarkably flexible at handling a wide range of different time series patterns, but … How about ability to learn and generalized from similar series (to learn more complex models without overfitting) Benefits Challenges • Interpretable • Implicitly models auto- correlation structure • Works well when there is little exogenous information • Doesn’t share information across time-series • Forecasting a large number of individual or grouped time- series • Struggles with sparsity and special events Benefits Challenges • Shares information across time series • Uses meta-information • Models non-linearities as well as interactions • Some works with missing values • Struggles if little meta- information • Requires larger volumes of data • Larger amounts of data preprocessing needed. • Tend to average predictions too much across time series Based on neural networks with a modified architecture. Implicitly models interactions, non-linearities as well as time-series features. LSTM’s (vs RNN) do a better job of modelling long term time dependencies. ML Models Random Forest. Prophet. LSTM. AWS ForecastAutoregressive ARIMA. ETS Classic vs Modern Models: Benefit & Challenges
Amplifying OrganisationalIntelligence Benefits • 50% more accurate forecasts with machine learning • Reduce forecasting time from months to hours Use cases • Product Demand Planning • Retail product demand • Supply chain demand • Operational metrics • Business metrics • Financial planning • Resource planning Statistical Machine Learning Volume of data Works well with little information Needs data from several series or several features Can share meta- information No (ARIMAX exception) Yes Can handle sparse data No Yes Can handle non- linearities/interaction s No or only explicitly Yes Can leverage shared information between time-series No (VAR exception) Yes, but tends to average too much Can work with high dimensional data Limited Yes AWS Forecast Amazon Forecast is a fully managed service that uses machine learning to deliver highly accurate forecasts.
Amplifying OrganisationalIntelligence Datasets and Dataset Groups Predictors Forecasts AWS Forecast HowThis Works? Setting Up: • Sign Up for AWS • Set Up the AWSCLI • Set Up Permissions for Amazon Forecast • Autoregressive Integrated Moving Average(ARIMA) • arn:aws:forecast:::algorithm/ARIMA • DeepAR • arn:aws:forecast:::algorithm/Deep_AR • Prophet • arn:aws:forecast:::algorithm/Prophet Predictor:
Amplifying OrganisationalIntelligence AWS Predictor: DeepAr DeepAR is a forecasting model based on autoregressive RNNs, which learns a global model from historical data of all time series in all datasets DeepAr is Multi-step multivariate time series: • Given observed values of a series i for t time-steps, estimating probability distribution of the next T steps Pros Cons • Shares information across groups of time series • Models non-linearities as well as interactions • Minimal manual feature engineering • Ability to incorporate a wide range of likelihood models, including probabilistic forecasts in the form of Monte Carlo samples • Struggles if little meta- information • Requires larger volumes of data • Tend to average predictions too much across time series
Amplifying OrganisationalIntelligence Best Practices for using the DeepAR Algorithm • Input/Output interface: • Supports two data channels (Train and Test for evaluation) • Format: JSON, gzip, and Parquet • Best practice: • Except for when splitting your dataset for train and test, always provide the entire time series. Why: the lagged value features • Test points should start immediately after the last time point of training • Avoid using very large values (>400) for the prediction length because it makes the model slow and less accurate. Solution: consider aggregating your data at a higher frequency. • ARIMA or ETS, might provide more accurate results on on a single time series. The DeepAR algorithm starts to outperform the standard methods when your dataset contains hundreds of related time series. • Train: on both GPU and CPU instances. Inference: only CPU • Use small number for context_length, prediction_length, num_cells, num_layers, or mini_batch_size, in case of small instances
Amplifying OrganisationalIntelligence DeepAR: Demo SageMaker/DeepAR demo on electricity dataset
Amplifying OrganisationalIntelligence References • https://docs.aws.amazon.com/forecast/latest/dg/forecast.dg.pdf • https://aws.amazon.com/blogs/aws/amazon-forecast-time-series-forecasting-made-easy/ • https://docs.aws.amazon.com/sagemaker/latest/dg/deepar.html

Recommended

Deep ar presentation
Deep ar presentationDeep ar presentation
Deep ar presentationCyrus Moazami-Vahid
 
Time series deep learning
Time series deep learningTime series deep learning
Time series deep learningAlberto Arrigoni
 
Time Series Forecasting Project Presentation.
Time Series Forecasting Project Presentation.Time Series Forecasting Project Presentation.
Time Series Forecasting Project Presentation.Anupama Kate
 
Borderline Smote
Borderline SmoteBorderline Smote
Borderline SmoteTrector Rancor
 
Recurrent Neural Networks, LSTM and GRU
Recurrent Neural Networks, LSTM and GRURecurrent Neural Networks, LSTM and GRU
Recurrent Neural Networks, LSTM and GRUananth
 
Cross-validation Tutorial: What, how and which?
Cross-validation Tutorial: What, how and which?Cross-validation Tutorial: What, how and which?
Cross-validation Tutorial: What, how and which?Pradeep Redddy Raamana
 
LSTM Basics
LSTM BasicsLSTM Basics
LSTM BasicsAkshay Sehgal
 
Time series forecasting with machine learning
Time series forecasting with machine learningTime series forecasting with machine learning
Time series forecasting with machine learningDr Wei Liu
 

Recommended

Deep ar presentation
Deep ar presentationDeep ar presentation
Deep ar presentationCyrus Moazami-Vahid
 
Time series deep learning
Time series deep learningTime series deep learning
Time series deep learningAlberto Arrigoni
 
Time Series Forecasting Project Presentation.
Time Series Forecasting Project Presentation.Time Series Forecasting Project Presentation.
Time Series Forecasting Project Presentation.Anupama Kate
 
Borderline Smote
Borderline SmoteBorderline Smote
Borderline SmoteTrector Rancor
 
Recurrent Neural Networks, LSTM and GRU
Recurrent Neural Networks, LSTM and GRURecurrent Neural Networks, LSTM and GRU
Recurrent Neural Networks, LSTM and GRUananth
 
Cross-validation Tutorial: What, how and which?
Cross-validation Tutorial: What, how and which?Cross-validation Tutorial: What, how and which?
Cross-validation Tutorial: What, how and which?Pradeep Redddy Raamana
 
LSTM Basics
LSTM BasicsLSTM Basics
LSTM BasicsAkshay Sehgal
 
Time series forecasting with machine learning
Time series forecasting with machine learningTime series forecasting with machine learning
Time series forecasting with machine learningDr Wei Liu
 
Intro to Feature Selection
Intro to Feature SelectionIntro to Feature Selection
Intro to Feature Selectionchenhm
 
LSTM
LSTMLSTM
LSTM佳蓉 倪
 
Vanishing & Exploding Gradients
Vanishing & Exploding GradientsVanishing & Exploding Gradients
Vanishing & Exploding GradientsSiddharth Vij
 
Anomaly detection (Unsupervised Learning) in Machine Learning
Anomaly detection (Unsupervised Learning) in Machine LearningAnomaly detection (Unsupervised Learning) in Machine Learning
Anomaly detection (Unsupervised Learning) in Machine LearningKuppusamy P
 
Feature selection
Feature selectionFeature selection
Feature selectiondkpawar
 
An Introduction to Anomaly Detection
An Introduction to Anomaly DetectionAn Introduction to Anomaly Detection
An Introduction to Anomaly DetectionKenneth Graham
 
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...Edureka!
 
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...Praxitelis Nikolaos Kouroupetroglou
 
Anomaly Detection in Seasonal Time Series
Anomaly Detection in Seasonal Time SeriesAnomaly Detection in Seasonal Time Series
Anomaly Detection in Seasonal Time SeriesHumberto Marchezi
 
Autoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time SeriesAutoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time SeriesYiqun Hu
 
Feature Engineering
Feature EngineeringFeature Engineering
Feature EngineeringHJ van Veen
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural NetworksDatabricks
 
Time Delayed Recurrent Neural Network for Multi-Step Prediction
Time Delayed Recurrent Neural Network for Multi-Step PredictionTime Delayed Recurrent Neural Network for Multi-Step Prediction
Time Delayed Recurrent Neural Network for Multi-Step PredictionKostas Hatalis, PhD
 
Recurrent neural networks rnn
Recurrent neural networks rnnRecurrent neural networks rnn
Recurrent neural networks rnnKuppusamy P
 
LSTM Tutorial
LSTM TutorialLSTM Tutorial
LSTM TutorialRalph Schlosser
 
18 Data Streams
18 Data Streams18 Data Streams
18 Data StreamsPier Luca Lanzi
 
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...Mahan Hosseinzadeh
 
Long Short Term Memory
Long Short Term MemoryLong Short Term Memory
Long Short Term MemoryYan Xu
 
Lstm
LstmLstm
LstmMehrnaz Faraz
 
Model selection and cross validation techniques
Model selection and cross validation techniquesModel selection and cross validation techniques
Model selection and cross validation techniquesVenkata Reddy Konasani
 
What is Data as a Service by T-Mobile Principle Technical PM
What is Data as a Service by T-Mobile Principle Technical PMWhat is Data as a Service by T-Mobile Principle Technical PM
What is Data as a Service by T-Mobile Principle Technical PMProduct School
 
Machine learning systems for engineers
Machine learning systems for engineersMachine learning systems for engineers
Machine learning systems for engineersCameron Joannidis
 

More Related Content

What's hot

Intro to Feature Selection
Intro to Feature SelectionIntro to Feature Selection
Intro to Feature Selectionchenhm
 
Vanishing & Exploding Gradients
Vanishing & Exploding GradientsVanishing & Exploding Gradients
Vanishing & Exploding GradientsSiddharth Vij
 
Anomaly detection (Unsupervised Learning) in Machine Learning
Anomaly detection (Unsupervised Learning) in Machine LearningAnomaly detection (Unsupervised Learning) in Machine Learning
Anomaly detection (Unsupervised Learning) in Machine LearningKuppusamy P
 
Feature selection
Feature selectionFeature selection
Feature selectiondkpawar
 
An Introduction to Anomaly Detection
An Introduction to Anomaly DetectionAn Introduction to Anomaly Detection
An Introduction to Anomaly DetectionKenneth Graham
 
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...Edureka!
 
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...Praxitelis Nikolaos Kouroupetroglou
 
Anomaly Detection in Seasonal Time Series
Anomaly Detection in Seasonal Time SeriesAnomaly Detection in Seasonal Time Series
Anomaly Detection in Seasonal Time SeriesHumberto Marchezi
 
Autoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time SeriesAutoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time SeriesYiqun Hu
 
Feature Engineering
Feature EngineeringFeature Engineering
Feature EngineeringHJ van Veen
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural NetworksDatabricks
 
Time Delayed Recurrent Neural Network for Multi-Step Prediction
Time Delayed Recurrent Neural Network for Multi-Step PredictionTime Delayed Recurrent Neural Network for Multi-Step Prediction
Time Delayed Recurrent Neural Network for Multi-Step PredictionKostas Hatalis, PhD
 
Recurrent neural networks rnn
Recurrent neural networks rnnRecurrent neural networks rnn
Recurrent neural networks rnnKuppusamy P
 
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...Mahan Hosseinzadeh
 
Long Short Term Memory
Long Short Term MemoryLong Short Term Memory
Long Short Term MemoryYan Xu
 
Model selection and cross validation techniques
Model selection and cross validation techniquesModel selection and cross validation techniques
Model selection and cross validation techniquesVenkata Reddy Konasani
 

What's hot (20)

Intro to Feature Selection
Intro to Feature SelectionIntro to Feature Selection
Intro to Feature Selection
 
LSTM
LSTMLSTM
LSTM
 
Vanishing & Exploding Gradients
Vanishing & Exploding GradientsVanishing & Exploding Gradients
Vanishing & Exploding Gradients
 
Anomaly detection (Unsupervised Learning) in Machine Learning
Anomaly detection (Unsupervised Learning) in Machine LearningAnomaly detection (Unsupervised Learning) in Machine Learning
Anomaly detection (Unsupervised Learning) in Machine Learning
 
Feature selection
Feature selectionFeature selection
Feature selection
 
An Introduction to Anomaly Detection
An Introduction to Anomaly DetectionAn Introduction to Anomaly Detection
An Introduction to Anomaly Detection
 
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
 
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...
Presentation - Msc Thesis - Machine Learning Techniques for Short-Term Electr...
 
Anomaly Detection in Seasonal Time Series
Anomaly Detection in Seasonal Time SeriesAnomaly Detection in Seasonal Time Series
Anomaly Detection in Seasonal Time Series
 
Autoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time SeriesAutoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time Series
 
Feature Engineering
Feature EngineeringFeature Engineering
Feature Engineering
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural Networks
 
Time Delayed Recurrent Neural Network for Multi-Step Prediction
Time Delayed Recurrent Neural Network for Multi-Step PredictionTime Delayed Recurrent Neural Network for Multi-Step Prediction
Time Delayed Recurrent Neural Network for Multi-Step Prediction
 
Recurrent neural networks rnn
Recurrent neural networks rnnRecurrent neural networks rnn
Recurrent neural networks rnn
 
LSTM Tutorial
LSTM TutorialLSTM Tutorial
LSTM Tutorial
 
18 Data Streams
18 Data Streams18 Data Streams
18 Data Streams
 
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...
Prophet at Scale: Using Prophet at scale to tune and forecast time series at ...
 
Long Short Term Memory
Long Short Term MemoryLong Short Term Memory
Long Short Term Memory
 
Lstm
LstmLstm
Lstm
 
Model selection and cross validation techniques
Model selection and cross validation techniquesModel selection and cross validation techniques
Model selection and cross validation techniques
 

Similar to AWS Forcecast: DeepAR Predictor Time-series

What is Data as a Service by T-Mobile Principle Technical PM
What is Data as a Service by T-Mobile Principle Technical PMWhat is Data as a Service by T-Mobile Principle Technical PM
What is Data as a Service by T-Mobile Principle Technical PMProduct School
 
Machine learning systems for engineers
Machine learning systems for engineersMachine learning systems for engineers
Machine learning systems for engineersCameron Joannidis
 
FlorenceAI: Reinventing Data Science at Humana
FlorenceAI: Reinventing Data Science at HumanaFlorenceAI: Reinventing Data Science at Humana
FlorenceAI: Reinventing Data Science at HumanaDatabricks
 
Challenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in ProductionChallenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in Productioniguazio
 
Get the most out of your AWS Redshift investment while keeping cost down
Get the most out of your AWS Redshift investment while keeping cost downGet the most out of your AWS Redshift investment while keeping cost down
Get the most out of your AWS Redshift investment while keeping cost downAgilisium Consulting
 
Mind Map Test Data Management Overview
Mind Map Test Data Management OverviewMind Map Test Data Management Overview
Mind Map Test Data Management Overviewdublinx
 
Observability – the good, the bad, and the ugly
Observability – the good, the bad, and the uglyObservability – the good, the bad, and the ugly
Observability – the good, the bad, and the uglyTimetrix
 
Using Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformUsing Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformDevOps.com
 
Improving AI Development - Dave Litwiller - Jan 11 2022 - Public
Improving AI Development - Dave Litwiller - Jan 11 2022 - PublicImproving AI Development - Dave Litwiller - Jan 11 2022 - Public
Improving AI Development - Dave Litwiller - Jan 11 2022 - PublicDave Litwiller
 
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...InfluxData
 
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine Aleksandr Tavgen
 
Observability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyObservability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyAleksandr Tavgen
 
Data science neural network project life cycle
Data science neural network project life cycleData science neural network project life cycle
Data science neural network project life cycleVincent Pommier
 
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...inside-BigData.com
 
BsidesLVPresso2016_JZeditsv6
BsidesLVPresso2016_JZeditsv6BsidesLVPresso2016_JZeditsv6
BsidesLVPresso2016_JZeditsv6Rod Soto
 
The Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it WorkThe Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it WorkIvo Andreev
 
Modernising the data warehouse - January 2019
Modernising the data warehouse - January 2019Modernising the data warehouse - January 2019
Modernising the data warehouse - January 2019Phil Watt
 
Data preprocessing using Machine Learning
Data preprocessing using Machine Learning Data preprocessing using Machine Learning
Data preprocessing using Machine Learning Gopal Sakarkar
 
Machine Learning in the Financial Industry
Machine Learning in the Financial IndustryMachine Learning in the Financial Industry
Machine Learning in the Financial IndustrySubrat Panda, PhD
 

Similar to AWS Forcecast: DeepAR Predictor Time-series (20)

What is Data as a Service by T-Mobile Principle Technical PM
What is Data as a Service by T-Mobile Principle Technical PMWhat is Data as a Service by T-Mobile Principle Technical PM
What is Data as a Service by T-Mobile Principle Technical PM
 
Machine learning systems for engineers
Machine learning systems for engineersMachine learning systems for engineers
Machine learning systems for engineers
 
FlorenceAI: Reinventing Data Science at Humana
FlorenceAI: Reinventing Data Science at HumanaFlorenceAI: Reinventing Data Science at Humana
FlorenceAI: Reinventing Data Science at Humana
 
Automated Analytics at Scale
Automated Analytics at ScaleAutomated Analytics at Scale
Automated Analytics at Scale
 
Challenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in ProductionChallenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in Production
 
Get the most out of your AWS Redshift investment while keeping cost down
Get the most out of your AWS Redshift investment while keeping cost downGet the most out of your AWS Redshift investment while keeping cost down
Get the most out of your AWS Redshift investment while keeping cost down
 
Mind Map Test Data Management Overview
Mind Map Test Data Management OverviewMind Map Test Data Management Overview
Mind Map Test Data Management Overview
 
Observability – the good, the bad, and the ugly
Observability – the good, the bad, and the uglyObservability – the good, the bad, and the ugly
Observability – the good, the bad, and the ugly
 
Using Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformUsing Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS Platform
 
Improving AI Development - Dave Litwiller - Jan 11 2022 - Public
Improving AI Development - Dave Litwiller - Jan 11 2022 - PublicImproving AI Development - Dave Litwiller - Jan 11 2022 - Public
Improving AI Development - Dave Litwiller - Jan 11 2022 - Public
 
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
 
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
 
Observability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyObservability - the good, the bad, and the ugly
Observability - the good, the bad, and the ugly
 
Data science neural network project life cycle
Data science neural network project life cycleData science neural network project life cycle
Data science neural network project life cycle
 
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
 
BsidesLVPresso2016_JZeditsv6
BsidesLVPresso2016_JZeditsv6BsidesLVPresso2016_JZeditsv6
BsidesLVPresso2016_JZeditsv6
 
The Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it WorkThe Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it Work
 
Modernising the data warehouse - January 2019
Modernising the data warehouse - January 2019Modernising the data warehouse - January 2019
Modernising the data warehouse - January 2019
 
Data preprocessing using Machine Learning
Data preprocessing using Machine Learning Data preprocessing using Machine Learning
Data preprocessing using Machine Learning
 
Machine Learning in the Financial Industry
Machine Learning in the Financial IndustryMachine Learning in the Financial Industry
Machine Learning in the Financial Industry
 

More from PolarSeven Pty Ltd

Aws user group #04 landing zones
Aws user group #04 landing zonesAws user group #04 landing zones
Aws user group #04 landing zonesPolarSeven Pty Ltd
 
Aws user group #03 - All things Iot
Aws user group #03 - All things IotAws user group #03 - All things Iot
Aws user group #03 - All things IotPolarSeven Pty Ltd
 
Aws user group #01 lets talk serverless
Aws user group #01 lets talk serverlessAws user group #01 lets talk serverless
Aws user group #01 lets talk serverlessPolarSeven Pty Ltd
 
Amazon Web Services User Group Sydney - March 2018
Amazon Web Services User Group Sydney - March 2018Amazon Web Services User Group Sydney - March 2018
Amazon Web Services User Group Sydney - March 2018PolarSeven Pty Ltd
 
Amazon Web Services User Group Sydney - February 2018
Amazon Web Services User Group Sydney - February 2018Amazon Web Services User Group Sydney - February 2018
Amazon Web Services User Group Sydney - February 2018PolarSeven Pty Ltd
 
Deep Dive on Cloud Policies and Automation
Deep Dive on Cloud Policies and AutomationDeep Dive on Cloud Policies and Automation
Deep Dive on Cloud Policies and AutomationPolarSeven Pty Ltd
 
Securing Traffic Leaving A VPC
Securing Traffic Leaving A VPCSecuring Traffic Leaving A VPC
Securing Traffic Leaving A VPCPolarSeven Pty Ltd
 
Telstra Programmable Networks & Scaling a Serverless Team with Automation
Telstra Programmable Networks & Scaling a Serverless Team with Automation Telstra Programmable Networks & Scaling a Serverless Team with Automation
Telstra Programmable Networks & Scaling a Serverless Team with AutomationPolarSeven Pty Ltd
 
AWS User Group Sydney - Meetup #60
AWS User Group Sydney - Meetup #60AWS User Group Sydney - Meetup #60
AWS User Group Sydney - Meetup #60PolarSeven Pty Ltd
 
Visibility, Optimization & Governance for Cloud Services
Visibility, Optimization & Governance for Cloud ServicesVisibility, Optimization & Governance for Cloud Services
Visibility, Optimization & Governance for Cloud ServicesPolarSeven Pty Ltd
 
AWS OpsWorks for Chef Automate
AWS OpsWorks for Chef AutomateAWS OpsWorks for Chef Automate
AWS OpsWorks for Chef AutomatePolarSeven Pty Ltd
 
AWS CloudFormation Automation, TrafficScript, and Serverless architecture wit...
AWS CloudFormation Automation, TrafficScript, and Serverless architecture wit...AWS CloudFormation Automation, TrafficScript, and Serverless architecture wit...
AWS CloudFormation Automation, TrafficScript, and Serverless architecture wit...PolarSeven Pty Ltd
 
AWS User Group Sydney - Atlassian 5-10-16
AWS User Group Sydney - Atlassian 5-10-16AWS User Group Sydney - Atlassian 5-10-16
AWS User Group Sydney - Atlassian 5-10-16PolarSeven Pty Ltd
 

More from PolarSeven Pty Ltd (20)

Aws user group #04 landing zones
Aws user group #04 landing zonesAws user group #04 landing zones
Aws user group #04 landing zones
 
Aws user group #03 - All things Iot
Aws user group #03 - All things IotAws user group #03 - All things Iot
Aws user group #03 - All things Iot
 
Aws user group #01 lets talk serverless
Aws user group #01 lets talk serverlessAws user group #01 lets talk serverless
Aws user group #01 lets talk serverless
 
AWS Reinvent Recap 2018
AWS Reinvent Recap 2018 AWS Reinvent Recap 2018
AWS Reinvent Recap 2018
 
AWS User Group October
AWS User Group OctoberAWS User Group October
AWS User Group October
 
AWS User Group August
AWS User Group AugustAWS User Group August
AWS User Group August
 
AWS User Group November
AWS User Group NovemberAWS User Group November
AWS User Group November
 
AWS User Group September
AWS User Group September AWS User Group September
AWS User Group September
 
Amazon Web Services User Group Sydney - March 2018
Amazon Web Services User Group Sydney - March 2018Amazon Web Services User Group Sydney - March 2018
Amazon Web Services User Group Sydney - March 2018
 
Amazon Web Services User Group Sydney - February 2018
Amazon Web Services User Group Sydney - February 2018Amazon Web Services User Group Sydney - February 2018
Amazon Web Services User Group Sydney - February 2018
 
Deep Dive on Cloud Policies and Automation
Deep Dive on Cloud Policies and AutomationDeep Dive on Cloud Policies and Automation
Deep Dive on Cloud Policies and Automation
 
Securing Traffic Leaving A VPC
Securing Traffic Leaving A VPCSecuring Traffic Leaving A VPC
Securing Traffic Leaving A VPC
 
Telstra Programmable Networks & Scaling a Serverless Team with Automation
Telstra Programmable Networks & Scaling a Serverless Team with Automation Telstra Programmable Networks & Scaling a Serverless Team with Automation
Telstra Programmable Networks & Scaling a Serverless Team with Automation
 
AWS User Group Sydney - Meetup #60
AWS User Group Sydney - Meetup #60AWS User Group Sydney - Meetup #60
AWS User Group Sydney - Meetup #60
 
Shared Security in AWS
Shared Security in AWSShared Security in AWS
Shared Security in AWS
 
Visibility, Optimization & Governance for Cloud Services
Visibility, Optimization & Governance for Cloud ServicesVisibility, Optimization & Governance for Cloud Services
Visibility, Optimization & Governance for Cloud Services
 
AWS OpsWorks for Chef Automate
AWS OpsWorks for Chef AutomateAWS OpsWorks for Chef Automate
AWS OpsWorks for Chef Automate
 
AWS CloudFormation Automation, TrafficScript, and Serverless architecture wit...
AWS CloudFormation Automation, TrafficScript, and Serverless architecture wit...AWS CloudFormation Automation, TrafficScript, and Serverless architecture wit...
AWS CloudFormation Automation, TrafficScript, and Serverless architecture wit...
 
AWS User Group December 2016
AWS User Group December 2016AWS User Group December 2016
AWS User Group December 2016
 
AWS User Group Sydney - Atlassian 5-10-16
AWS User Group Sydney - Atlassian 5-10-16AWS User Group Sydney - Atlassian 5-10-16
AWS User Group Sydney - Atlassian 5-10-16
 

Recently uploaded

AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentationphoebematthew05
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphSIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxMaking_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxnull - The Open Security Community
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
APIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April Automation LPDGAPIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April Automation LPDGMarianaLemus7
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 

Recently uploaded (20)

AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentation
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphSIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping Elbows
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxMaking_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
APIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April Automation LPDGAPIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April Automation LPDG
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 

AWS Forcecast: DeepAR Predictor Time-series

  • 1. Amplifying OrganisationalIntelligence Intellify Pty Ltd IntellifyAI Intellify_AISydney Level 8 11York Street Sydney, NSW 2000 T. (02) 8089 4073 www.intellify.com.au Melbourne Level 28 303 Collins Street Melbourne,VIC 3000 T. (03) 9132 9846 info@intellify.com.au 20 Bridge Street AWS Forecast: DeepAR Predictor Time-series
  • 2. Amplifying OrganisationalIntelligence Agenda 1. Introduction 2. Why time-series and how are they different? 3. Classical (Statistical) and Modern time-series methods 4. AWS Forecast: Modern Time-series 5. AWS Predictor: DeepAR 6. Demo on DeepAR
  • 3. Amplifying OrganisationalIntelligence Why are time-series methods important? 1 2 3 Time series are everywhere! Most methods were designed for use on cross-sectional data We can drive better business outcomes through the use of time-series methods
  • 4. Amplifying OrganisationalIntelligence How are time series problems different? • Different states in a time series can make the problem harder to model. • There could be multiple forecasting horizons; short, medium, long term. • Typically you care about the prediction as well as the confidence in the prediction. • Model testing and validation must be conducted in a different way to avoid data leakage and select the best model.
  • 5. Amplifying OrganisationalIntelligence What are desirable properties of time series methods? Multi-step multivariate prediction Shares information across time-series Leverages meta-information Works on sparse data Handles non-linearities/interactions Works with high dimensional data Models autocorrelation structure implicitly Minimal feature pre-processing and engineering
  • 6. Amplifying OrganisationalIntelligence Traditional Models Autoregressive models are remarkably flexible at handling a wide range of different time series patterns, but … How about ability to learn and generalized from similar series (to learn more complex models without overfitting) Benefits Challenges • Interpretable • Implicitly models auto- correlation structure • Works well when there is little exogenous information • Doesn’t share information across time-series • Forecasting a large number of individual or grouped time- series • Struggles with sparsity and special events Benefits Challenges • Shares information across time series • Uses meta-information • Models non-linearities as well as interactions • Some works with missing values • Struggles if little meta- information • Requires larger volumes of data • Larger amounts of data preprocessing needed. • Tend to average predictions too much across time series Based on neural networks with a modified architecture. Implicitly models interactions, non-linearities as well as time-series features. LSTM’s (vs RNN) do a better job of modelling long term time dependencies. ML Models Random Forest. Prophet. LSTM. AWS ForecastAutoregressive ARIMA. ETS Classic vs Modern Models: Benefit & Challenges
  • 7. Amplifying OrganisationalIntelligence Benefits • 50% more accurate forecasts with machine learning • Reduce forecasting time from months to hours Use cases • Product Demand Planning • Retail product demand • Supply chain demand • Operational metrics • Business metrics • Financial planning • Resource planning Statistical Machine Learning Volume of data Works well with little information Needs data from several series or several features Can share meta- information No (ARIMAX exception) Yes Can handle sparse data No Yes Can handle non- linearities/interaction s No or only explicitly Yes Can leverage shared information between time-series No (VAR exception) Yes, but tends to average too much Can work with high dimensional data Limited Yes AWS Forecast Amazon Forecast is a fully managed service that uses machine learning to deliver highly accurate forecasts.
  • 8. Amplifying OrganisationalIntelligence Datasets and Dataset Groups Predictors Forecasts AWS Forecast HowThis Works? Setting Up: • Sign Up for AWS • Set Up the AWSCLI • Set Up Permissions for Amazon Forecast • Autoregressive Integrated Moving Average(ARIMA) • arn:aws:forecast:::algorithm/ARIMA • DeepAR • arn:aws:forecast:::algorithm/Deep_AR • Prophet • arn:aws:forecast:::algorithm/Prophet Predictor:
  • 9. Amplifying OrganisationalIntelligence AWS Predictor: DeepAr DeepAR is a forecasting model based on autoregressive RNNs, which learns a global model from historical data of all time series in all datasets DeepAr is Multi-step multivariate time series: • Given observed values of a series i for t time-steps, estimating probability distribution of the next T steps Pros Cons • Shares information across groups of time series • Models non-linearities as well as interactions • Minimal manual feature engineering • Ability to incorporate a wide range of likelihood models, including probabilistic forecasts in the form of Monte Carlo samples • Struggles if little meta- information • Requires larger volumes of data • Tend to average predictions too much across time series
  • 10. Amplifying OrganisationalIntelligence Best Practices for using the DeepAR Algorithm • Input/Output interface: • Supports two data channels (Train and Test for evaluation) • Format: JSON, gzip, and Parquet • Best practice: • Except for when splitting your dataset for train and test, always provide the entire time series. Why: the lagged value features • Test points should start immediately after the last time point of training • Avoid using very large values (>400) for the prediction length because it makes the model slow and less accurate. Solution: consider aggregating your data at a higher frequency. • ARIMA or ETS, might provide more accurate results on on a single time series. The DeepAR algorithm starts to outperform the standard methods when your dataset contains hundreds of related time series. • Train: on both GPU and CPU instances. Inference: only CPU • Use small number for context_length, prediction_length, num_cells, num_layers, or mini_batch_size, in case of small instances
  • 12. Amplifying OrganisationalIntelligence References • https://docs.aws.amazon.com/forecast/latest/dg/forecast.dg.pdf • https://aws.amazon.com/blogs/aws/amazon-forecast-time-series-forecasting-made-easy/ • https://docs.aws.amazon.com/sagemaker/latest/dg/deepar.html

Editor's Notes

  1. Lets first start with why time-series series methods are important. The first reasons is that time series problems are everywhere; they appear in financial data, customer behavior data, property data and engineering problems. In fact, in our experience, we have that around 70% of our consulting projects have some time-series component or consideration that needs to be incorporated into the solution. The second reason is that most methods, especially the standard ones inside statistics and machine learning are built for cross-sectional problems. If you haven’t heard of this terminology before, cross sectional problems are where we take many observations at a point in time from many individuals.
  2. Time–series data is a chronological sequence of observations on a particular variable.
  3. Time–series data is a chronological sequence of observations on a particular variable.
  4. Exponential smoothing (ETS methods) Classical methods typically work through: Decomposition of time-series into each of its components Find average historical affects for each component Aggregate average historical affects and forecast one step ahead Modern time series methods follow the same patterns as traditional machine learning approaches with 3 major modifications: Time-series features are manually created by the user (time-series feature engineering) if the algorithm cannot implicitly model them. Specific Machine learning methods are applied that give us the point estimate as well as the distribution. Traditional time series validation (not random sampling) is used with specific metrics.
  5. Developers with no machine learning expertise can use the Amazon Forecast APIs, AWS Command Line Interface (AWS CLI), or Amazon Forecast console to import training data into one or more Amazon Forecast datasets, train predictors, and generate forecasts.
  6. When creating forecasting projects in Amazon Forecast, you work with the following resources: Before using Amazon Forecast to evaluate or forecast time-series data, create an AWS account, configure access permissions, and set up the AWS Command Line Interface (AWS CLI).
  7. Autoregression is a time series model that uses observations from previous time steps as input to a regression equation to predict the value at the next time step.  A recurrent neural network (RNN) is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence
  8. https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/deepar_electricity/DeepAR-Electricity.ipynb