If you have any device or source that generates values over time (also a log from a service), you want to determine if in a time frame, the time serie is correct or you can detect some anomalies. What can you do as a developer (not a Data Scientist) with .NET o Azure? Let's see how in this session.

1. Time Series Anomaly Detection
with .net and Azure
Marco Parenzan
Solution Sales Specialist @ Insight // Microsoft Azure MVP // 1nn0va Community Lead
Marco Parenzan

2. Platinum Sponsor
Gold Sponsor
Technical Sponsor

3. Marco Parenzan
Linkedin: https://www.linkedin.com/in/marcoparenzan/
Slideshare: https://www.slideshare.net/marco.parenzan
GitHub: https://github.com/marcoparenzan

4. Agenda
• Scenario
• Anomaly Detection in Time Series
• Data Science for the .NET developer
• How Data Scientists work
• Bring ML.NET to Azure
• Anomaly Detection As A Service in Azure
• Conclusions

5. Scenario

6. Scenario
From real industrial fridges 
In an industrial fridge, you monitor temperatures to check
not the temperature «per se», but to check the healthy of
the plant
• Opening a door
• Condenser
• Evaporator
You can considering each of these events as anomalies
that alter the temperature you measure in different part of
the fridge

7. Simple Cloud Architecture (Smart City*)
Storage
Account
IoT Hub
Devices
Events
Ingest

8. Current data analysis
Data collecting
Data Analysis
Data Report...?
«What happens in this scenario?»

9. What happens in production?
Efficiency Anomalies
Batch Streaming

10. Where we can evolve...
Storage
Account
Function App
IoT Hub
Devices
Events Notification
Logic App
Ingest Process

11. Anomaly Detection in Time Series

12. Anomaly Detection
Anomaly detection is the process of identifying unexpected items or events in data sets, which differ
from the norm.
And anomaly detection is often applied on unlabeled data which is known as unsupervised anomaly
detection.

13. Time Series
Definition
• Time series is a sequence of data points recorded in time order, often taken at successive
equally paced points in time.
Examples
• Stock prices, Sales demand, website traffic, daily temperatures, quarterly sales
Time series is different from regression analysis because of its time-dependent nature.
• Auto-correlation: Regression analysis requires that there is little or no autocorrelation in the data.
It occurs when the observations are not independent of each other. For example, in stock prices,
the current price is not independent of the previous price. [The observations have to be
dependent on time]
• Seasonality, a characteristic which we will discuss below.

14. Components of a Time Series
Trend
• is a general direction in which something is developing or changing. A trend can be upward(uptrend) or
downward(downtrend). It is not always necessary that the increase or decrease is consistently in the same direction in a
given period.
Seasonality
• Predictable pattern that recurs or repeats over regular intervals. Seasonality is often observed within a year or less.
Irregular fluctuation
• These are variations that occur due to sudden causes and are unpredictable. For example the rise in prices of food due
to war, flood, earthquakes, farmers striking etc.

15. Anomaly Detection in Time Series
In time series data, an anomaly or outlier can be termed as a data point which is not following the
common collective trend or seasonal or cyclic pattern of the entire data and is significantly distinct
from rest of the data. By significant, most data scientists mean statistical significance, which in order
words, signify that the statistical properties of the data point is not in alignment with the rest of the
series.
Anomaly detection has two basic assumptions:
• Anomalies only occur very rarely in the data.
• Their features differ from the normal instances significantly.

16. How to do Time Series Anomaly Detection?
Statistical Profiling Approach
• This can be done by calculating statistical values like mean or median moving average of the
historical data and using a standard deviation to come up with a band of statistical values which
can define the uppermost bound and the lower most bound and anything falling beyond these
ranges can be an anomaly.
• this approach is very handy and can always be the baseline approach, instead of going with any
sophisticated and complex methods which require a lot of fine tuning and may not be explainable.
• This is very effective for highly volatile time series as well, as most of the time series predictive
model algorithms fail when the data is highly volatile.
• But the main drawback of this approach is detecting the local outliers.

17. How to do Time Series Anomaly Detection?
By Predictive Confidence Level Approach
• One way of doing anomaly detection with time series data is by building a predictive model using
the historical data to estimate and get a sense of the overall common trend, seasonal or cyclic
pattern of the time series data.
• we can come up with a confidence interval, or a confidence band for the predicted values and
any actual data point which is falling beyond this confidence band is an anomaly
• The main advantage of this approach is finding local outlier but the main disadvantage is, this
approach highly depends on the efficiency of the predictive model. Any loop hole in the predictive
model can give false positives and false negatives.

18. How to do Time Series Anomaly Detection?
Clustering Based Unsupervised Approach
• Unsupervised approaches are extremely useful for anomaly detection as it does not require any
labelled data, mentioning that a particular data point is an anomaly.

19. Data Science for the .NET developer

20. Data Science and AI for the .NET developer
ML.NET is first and foremost a framework that you can use
to create your own custom ML models. This custom
approach contrasts with “pre-built AI,” where you use pre-
designed general AI services from the cloud (like many of
the offerings from Azure Cognitive Services). This can work
great for many scenarios, but it might not always fit your
specific business needs due to the nature of the machine
learning problem or to the deployment context (cloud vs. on-
premises).
ML.NET enables developers to use their existing .NET skills
to easily integrate machine learning into almost any .NET
application. This means that if C# (or F# or VB) is your
programming language of choice, you no longer have to
learn a new programming language, like Python or R, in
order to develop your own ML models and infuse custom
machine learning into your .NET apps.

21. ML.NET Components
Anomaly Detection

22. Classic Process

23. Helping no-data scientits developers (all! )
Unsupervised Machine LearningNo labelling
Auto(mated) MLfind the best tuning for you with parameters and algorithms
Automated Training Set for Anomaly Detection Algorithms
• the algorithms automatically generates a simulated training set based non your input data
https://docs.microsoft.com/en-us/azure/machine-learning/algorithm-cheat-sheet

24. Independent Identically Distributed (iid)
Data points collected in the time series are independently sampled from the same distribution
(independent identically distributed). Thus, the value at the current timestamp can be viewed as the
value at the next timestamp in expectation.

25. Singular Spectrum Analysis (SSA)
This class implements the general anomaly detection transform based on Singular Spectrum
Analysis (SSA). SSA is a powerful framework for decomposing the time-series into trend,
seasonality and noise components as well as forecasting the future values of the time-series.
In principle, SSA performs spectral analysis on the input time-series where each component in the
spectrum corresponds to a trend, seasonal or noise component in the time-series

26. Spectrum Residual Cnn (SrCnn)
to monitor the time-series continuously and alert for potential incidents on time
The algorithm first computes the Fourier Transform of the original data. Then it computes
the spectral residual of the log amplitude of the transformed signal before applying the Inverse
Fourier Transform to map the sequence back from the frequency to the time domain. This sequence
is called the saliency map. The anomaly score is then computed as the relative difference between
the saliency map values and their moving averages. If the score is above a threshold, the value at a
specific timestep is flagged as an outlier.
There are several parameters for SR algorithm. To obtain a model with good performance, we
suggest to tune windowSize and threshold at first, these are the most important parameters to SR.
Then you could search for an appropriate judgementWindowSize which is no larger than
windowSize. And for the remaining parameters, you could use the default value directly.
Time-Series Anomaly Detection Service at Microsof [https://arxiv.org/pdf/1906.03821.pdf]

27. Some tools required
.NET 5 + WPF + ML.NET
• Mandatory , the platform where we try to make experiments
Xplot.Ploty (soon you will understand I use this) https://fslab.org/XPlot/
• XPlot is a cross-platform data visualization package for the F# programming language powered by
popular JavaScript charting libraries Plotly and Google Charts. The library provides a complete
mapping for the configuration options of the underlying libraries and so you get a nice F# interface
that gives you access to the full power of Plotly and Google Charts. The XPlot library can be used
interactively from F# Interactive, but charts can equally easy be embedded in F# applications and in
HTML reports.
WebView2 https://docs.microsoft.com/en-us/microsoft-edge/webview2/gettingstarted/wpf
• The Microsoft Edge WebView2 control enables you to embed web technologies (HTML, CSS, and
JavaScript) in your native apps. The WebView2 control uses Microsoft Edge (Chromium) as the
rendering engine to display the web content in native apps. With WebView2, you may embed web
code in different parts of your native app. Build all of the native app within a single WebView instance.

28. Demo
Experimenting ML.NET with WPF

29. How Data Scientists work

30. Jupyter
Evolution and generalization of the seminal role of Mathematica
In web standards way
• Web (HTTP+Markdown)
• Python adoption (ipynb)
Written in Java
Python has an interop bridge...not native (if ever important)Python is a kernel for Jupyter

31. .NET Interactive and Jupyter
and Visual Studio Code
.NET Interactive gives C# and F# kernels to Jupyter
.NET Interactive gives all tools to create your hosting application independently from Jupyter
In Visual Studio Code, you have two different notebooks (looking similar but developed in parallel by
different teams)
.NET Interactive Notebook (by the .NET Interactive Team) that can run also Python
Jupyter Notebook (by the Azure Data Studio Team – probably) that can run also C# and F#
There is a little confusion on that 
.NET Interactive has a strong C#/F# Kernel...
...a less mature infrastructure (compared to Jupiter)

32. Demo
Experimenting ML.NET
with .NET Interactive

33. Bring ML.NET to Azure

34. .NET (5) hosting in Azure
Existing apps
.NET web apps (on-premises)
Cloud-Optimized
PaaS
Cloud-Native
PaaS for microservices and serverless
Monolithic / N-Tier
architectures
Monolithic / N-Tier
architectures
Microservices and serverless architectures
Cloud
Infrastructure-Ready
Monolithic / N-Tier
architectures
Relational
Database
VMs
Managed services
On-premises Azure
PaaS for containerized microservices
+ Serverless computing
+ Managed services
And Windows Containers
IaaS
(Infrastructure as a Service)
Azure Azure

35. Functions everywhere
Platform
App delivery
OS
On-premises
Code
App Service on Azure Stack
Windows
●●●
Non-Azure hosts
●●●
●●●
+
Azure Functions
host runtime
Azure Functions
Core Tools
Azure Functions
base Docker image
Azure Functions
.NET Docker image
Azure Functions
Node Docker image
●●●

36. Logic Apps
Visually design workflows in the cloud
Express logic through powerful control flow
Connect disparate functions and APIs
Utilize declarative definition to work with CI/CD

37. Demo
Bring ML.NET to Azure

38. Anomaly Detection As A Service in Azure

39. Azure Cognitive Services
Cognitive Services brings AI within reach of every developer—without requiring machine-learning
expertise. All it takes is an API call to embed the ability to see, hear, speak, search, understand, and
accelerate decision-making into your apps. Enable developers of all skill levels to easily add AI
capabilities to their apps.
Five areas:
• Decision
• Language
• Speech
• Vision
• Web search
Anomaly Detector
Identify potential problems early on.
Content Moderator
Detect potentially offensive or unwanted
content.
Metrics Advisor PREVIEW
Monitor metrics and diagnose issues.
Personalizer
Create rich, personalized experiences for
every user.

40. Anomaly Detector
Through an API, Anomaly Detector ingests time-series data of all types and selects the best-fitting
detection model for your data to ensure high accuracy. Customize the service to detect any level of
anomaly and deploy it where you need it most -- from the cloud to the intelligent edge with
containers. Azure is the only major cloud provider that offers anomaly detection as an AI service.

41. Demo
Anomaly Detection As A Service

42. Anomaly Detector
Through an API, Anomaly Detector ingests time-series data of all types and selects the best-fitting
detection model for your data to ensure high accuracy. Customize the service to detect any level of
anomaly and deploy it where you need it most -- from the cloud to the intelligent edge with
containers. Azure is the only major cloud provider that offers anomaly detection as an AI service.
It seems too much simple…

43. Business metrics
monitoring
AIOps Predictive maintenance
Metrics Advisor(preview)
REST APIs (SDKs)
Data ingestion, detection/alert
configuration, anomaly query
APIs
Web-based workspace
Model customization & incident
diagnostics

44. Collect time-series data Detect anomalies Send incident alerts Analyze root cause
Metrics Monitoring Pipeline
A simple light Time-Series Insights-like service

45. Conclusions

46. Conclusions
Start simple and bulk: you already have data
If you have daily data, you need to aggregate (a month?) to have training
• take time for a correct Data Lake strategy
• there is time for realtime
The right algorithm is the one that gives you what you want to see
• Also professionals make the same (besides REAL data scientists)
• But if you know statistics, if better for you 
Azure Cognitive Services will become more important
• New Metrics Advisor Service!

48. Platinum Sponsor
Gold Sponsor
Technical Sponsor