If you have any device or source that generates values over time (also a log from a service), you want to determine if in a time frame, the time serie is correct or you can detect some anomalies. What can you do as a developer (not a Data Scientist) with .NET and Azure?

1. TIME SERIES ANOMALY DETECTION
WITH .NET AND AZURE
 Marco Parenzan
 Solution Sales Specialist @ Insight // Microsoft Azure MVP //
1nn0va Community Lead

2. MARCO PARENZAN
 Solution Sales Specialist @ Insight
 1nn0va Community Lead (Pordenone)
 Microsoft Azure MVP
 Profiles
 Linkedin: https://www.linkedin.com/in/marcoparenzan/
 Slideshare: https://www.slideshare.net/marco.parenzan
 GitHub: https://github.com/marcoparenzan

3. AGENDA
• Scenario
• Anomaly Detection in Time Series
• Data Science for the .NET developer
• How Data Scientists work
• Bring ML.NET to Azure
• Anomaly Detection As A Service in Azure
• Conclusions

4. SCENARIO

5. SCENARIO
 In an industrial fridge, you monitor temperatures to check not the
temperature «per se», but to check the healthy of the plant
 Opening a door
 Condenser
 Evaporator
 You can considering each of these events as anomalies that alter the
temperature you measure in different part of the fridge
From real industrial fridges 

6. SIMPLE CLOUD ARCHITECTURE (SMART CITY*)
Storage
Account
IoT Hub
Devices
Events
Ingest

7. CURRENT DATA ANALYSIS
Data collecting
Data Analysis
Data Report...?
«What happens in this scenario?»

8. WHAT HAPPENS IN PRODUCTION?
Efficiency Anomalies
Batch Streaming

9. WHERE WE CAN EVOLVE...
Storage
Account
Function App
IoT Hub
Devices
Events Notification
Logic App
Ingest Process

10. ANOMALY DETECTION IN TIME SERIES

11. ANOMALY DETECTION
 Anomaly detection is the process of identifying unexpected items or events in data sets, which differ from
the norm.
 And anomaly detection is often applied on unlabeled data which is known as unsupervised anomaly
detection.

12. TIME SERIES
 Definition
• Time series is a sequence of data points recorded in time order, often taken at successive equally paced
points in time.
 Examples
• Stock prices, Sales demand, website traffic, daily temperatures, quarterly sales
 Time series is different from regression analysis because of its time-dependent nature.
• Auto-correlation: Regression analysis requires that there is little or no autocorrelation in the data. It occurs
when the observations are not independent of each other. For example, in stock prices, the current price is
not independent of the previous price. [The observations have to be dependent on time]
• Seasonality, a characteristic which we will discuss below.

13. COMPONENTS OF A TIME SERIES
 Trend
• is a general direction in which something is developing or changing. A trend can be upward(uptrend) or
downward(downtrend). It is not always necessary that the increase or decrease is consistently in the same direction in a
given period.
 Seasonality
• Predictable pattern that recurs or repeats over regular intervals. Seasonality is often observed within a year or less.
 Irregular fluctuation
• These are variations that occur due to sudden causes and are unpredictable. For example the rise in prices of food due
to war, flood, earthquakes, farmers striking etc.

14. ANOMALY DETECTION IN TIME SERIES
 In time series data, an anomaly or outlier can be termed as a data point which is not following the common
collective trend or seasonal or cyclic pattern of the entire data and is significantly distinct from rest of the
data. By significant, most data scientists mean statistical significance, which in order words, signify that the
statistical properties of the data point is not in alignment with the rest of the series.
 Anomaly detection has two basic assumptions:
• Anomalies only occur very rarely in the data.
• Their features differ from the normal instances significantly.

15. HOW TO DO TIME SERIES ANOMALY DETECTION?
 Statistical Profiling Approach
 This can be done by calculating statistical values like mean or median moving average of the historical data and using a
standard deviation to come up with a band of statistical values which can define the uppermost bound and the lower most
bound and anything falling beyond these ranges can be an anomaly.
 By Predictive Confidence Level Approach
 One way of doing anomaly detection with time series data is by building a predictive model using the historical data to
estimate and get a sense of the overall common trend, seasonal or cyclic pattern of the time series data.
 Clustering Based Unsupervised Approach
 Unsupervised approaches are extremely useful for anomaly detection as it does not require any labelled data, mentioning
that a particular data point is an anomaly.

16. MULTIVARIATE ANOMALY DETECTION
 All described is “univariate” anomaly detection, on
a single time serie
 The multivariate anomaly detection allows
detecting anomalies from groups of metrics
 Dependencies and inter-correlations between
different signals
 News are already announced in this area, else not
yet available
16
#GLOBALAZURE2021

17. DATA SCIENCE FOR THE .NET DEVELOPER

18. DATA SCIENCE AND AI FOR THE .NET DEVELOPER
 ML.NET is first and foremost a framework that you can use to
create your own custom ML models. This custom approach
contrasts with “pre-built AI,” where you use pre-designed
general AI services from the cloud (like many of the offerings
from Azure Cognitive Services). This can work great for many
scenarios, but it might not always fit your specific business
needs due to the nature of the machine learning problem or to
the deployment context (cloud vs. on-premises).
 ML.NET enables developers to use their existing .NET skills to
easily integrate machine learning into almost any .NET
application. This means that if C# (or F# or VB) is your
programming language of choice, you no longer have to learn a
new programming language, like Python or R, in order to
develop your own ML models and infuse custom machine
learning into your .NET apps.

19. ML.NET COMPONENTS
Anomaly Detection

20. CLASSIC PROCESS

21. HELPING NO-DATA SCIENTITS DEVELOPERS (ALL! )
 Unsupervised Machine LearningNo labelling
 Auto(mated) MLfind the best tuning for you with parameters and algorithms
 Automated Training Set for Anomaly Detection Algorithms
 the algorithms automatically generates a simulated training set based non your input data
https://docs.microsoft.com/en-us/azure/machine-learning/algorithm-cheat-sheet

22. INDEPENDENT IDENTICALLY DISTRIBUTED (IID)
 Data points collected in the time series are independently sampled from the same distribution
(independent identically distributed). Thus, the value at the current timestamp can be viewed as the value at
the next timestamp in expectation.

23. SINGULAR SPECTRUM ANALYSIS (SSA)
 This class implements the general anomaly detection transform based on Singular Spectrum Analysis (SSA).
SSA is a powerful framework for decomposing the time-series into trend, seasonality and noise
components as well as forecasting the future values of the time-series.
 In principle, SSA performs spectral analysis on the input time-series where each component in the spectrum
corresponds to a trend, seasonal or noise component in the time-series

24. SPECTRUM RESIDUAL CNN (SRCNN)
 to monitor the time-series continuously and alert for potential incidents on time
 The algorithm first computes the Fourier Transform of the original data. Then it computes the spectral
residual of the log amplitude of the transformed signal before applying the Inverse Fourier Transform to
map the sequence back from the frequency to the time domain. This sequence is called the saliency map.
The anomaly score is then computed as the relative difference between the saliency map values and their
moving averages. If the score is above a threshold, the value at a specific timestep is flagged as an outlier.
 There are several parameters for SR algorithm. To obtain a model with good performance, we suggest to
tune windowSize and threshold at first, these are the most important parameters to SR. Then you could
search for an appropriate judgementWindowSize which is no larger than windowSize. And for the
remaining parameters, you could use the default value directly.
 Time-Series Anomaly Detection Service at Microsof [https://arxiv.org/pdf/1906.03821.pdf]

25. SOME TOOLS REQUIRED
 .NET 5 + WPF + ML.NET
 Mandatory , the platform where we try to make experiments
 Xplot.Ploty (soon you will understand I use this) https://fslab.org/XPlot/
 XPlot is a cross-platform data visualization package for the F# programming language powered by popular
JavaScript charting libraries Plotly and Google Charts. The library provides a complete mapping for the
configuration options of the underlying libraries and so you get a nice F# interface that gives you access to the
full power of Plotly and Google Charts. The XPlot library can be used interactively from F# Interactive, but charts
can equally easy be embedded in F# applications and in HTML reports.
 WebView2 https://docs.microsoft.com/en-us/microsoft-edge/webview2/gettingstarted/wpf
 The Microsoft Edge WebView2 control enables you to embed web technologies (HTML, CSS, and JavaScript) in
your native apps. The WebView2 control uses Microsoft Edge (Chromium) as the rendering engine to display the
web content in native apps. With WebView2, you may embed web code in different parts of your native app. Build
all of the native app within a single WebView instance.

26. EXPERIMENTING ML.NET WITH WPF

27. HOW DATA SCIENTISTS WORK

28. JUPYTER
 Evolution and generalization of the seminal role of Mathematica
 In web standards way
 Web (HTTP+Markdown)
 Python adoption (ipynb)
 Written in Java
 Python has an interop bridge...not native (if ever important)Python is a kernel for Jupyter

29. .NET INTERACTIVE AND JUPYTER
AND VISUAL STUDIO CODE
 .NET Interactive gives C# and F# kernels to Jupyter
 .NET Interactive gives all tools to create your hosting application independently from Jupyter
 In Visual Studio Code, you have two different notebooks (looking similar but developed in parallel by
different teams)
 .NET Interactive Notebook (by the .NET Interactive Team) that can run also Python
 Jupyter Notebook (by the Azure Data Studio Team – probably) that can run also C# and F#
 There is a little confusion on that 
 .NET Interactive has a strong C#/F# Kernel...
 ...a less mature infrastructure (compared to Jupiter)

30. EXPERIMENTING ML.NET
WITH .NET INTERACTIVE

31. BRING ML.NET TO AZURE

32. .NET (5) HOSTING IN AZURE
Existing apps
.NET web apps (on-premises)
Cloud-Optimized
PaaS
Cloud-Native
PaaS for microservices and serverless
Monolithic / N-Tier
architectures
Monolithic / N-Tier
architectures
Microservices and serverless architectures
Cloud
Infrastructure-Ready
Monolithic / N-Tier
architectures
Relational
Database
VMs
Managed services
On-premises Azure
PaaS for containerized microservices
+ Serverless computing
+ Managed services
And Windows Containers
IaaS
(Infrastructure as a Service)
Azure Azure

33. FUNCTIONS EVERYWHERE
Platform
App delivery
OS
On-premises
Code
App Service on Azure Stack
Windows
●●●
Non-Azure hosts
●●●
●●●
+
Azure Functions
host runtime
Azure Functions
Core Tools
Azure Functions
base Docker image
Azure Functions
.NET Docker image
Azure Functions
Node Docker image
●●●

34. LOGIC APPS
 Visually design workflows in the cloud
 Express logic through powerful control flow
 Connect disparate functions and APIs
 Utilize declarative definition to work with CI/CD

35. BRING ML.NET TO AZURE

36. ANOMALY DETECTION AS A SERVICE IN AZURE

37. AZURE COGNITIVE SERVICES
 Cognitive Services brings AI within reach of every developer—without requiring machine-learning
expertise. All it takes is an API call to embed the ability to see, hear, speak, search, understand, and
accelerate decision-making into your apps. Enable developers of all skill levels to easily add AI capabilities
to their apps.
 Five areas:
• Decision
• Language
• Speech
• Vision
• Web search
Anomaly Detector
Identify potential problems early on.
Content Moderator
Detect potentially offensive or unwanted
content.
Metrics Advisor PREVIEW
Monitor metrics and diagnose issues.
Personalizer
Create rich, personalized experiences for
every user.

38. ANOMALY DETECTOR
 Through an API, Anomaly Detector ingests time-series data of all types and selects the best-fitting
detection model for your data to ensure high accuracy. Customize the service to detect any level of
anomaly and deploy it where you need it most -- from the cloud to the intelligent edge with containers.
Azure is the only major cloud provider that offers anomaly detection as an AI service.

39. ANOMALY DETECTION AS A SERVICE

40. ANOMALY DETECTOR
 Through an API, Anomaly Detector ingests time-series data of all types and selects the best-fitting
detection model for your data to ensure high accuracy. Customize the service to detect any level of
anomaly and deploy it where you need it most -- from the cloud to the intelligent edge with containers.
Azure is the only major cloud provider that offers anomaly detection as an AI service.
It seems too much simple…

41. METRICS ADVISOR(PREVIEW)

42. CONCLUSIONS

43. CONCLUSIONS
 Start simple and bulk: you already have data
 If you have daily data, you need to aggregate (a month?) to have training
• take time for a correct Data Lake strategy
• there is time for realtime
 The right algorithm is the one that gives you what you want to see
• Also professionals make the same (besides REAL data scientists)
• But if you know statistics, if better for you 
 Azure Cognitive Services will become more important
• New Metrics Advisor Service!