On the way to Azure: monitoring and analytics based on Elastic stack

1. On the way to Azure:
monitoring and analytics
based on Elastic stack
Artsiom Baranovskiy, Solution Architect at EPAM Systems
Artsiom_Baranovskiy@epam.com
April 22, 2017

2. [TOPIC]
We will cover…
1. Baseline architecture
2. New requirements for the cloud use-case
3. Deep dive into the Solution
• App Services
• Event Hubs
• Service Fabric
• ElasticSearch in Azure
| On the way to Azure: monitoring and analytics based on Elastic stack

3. [TOPIC]
Baseline architecture: Problem statement
The products under the hood
1. Black boxes
2. High complexity
3. Few diagnostics capabilities
What clients see
1. “Irrelevant” new features
2. High regression rate
3. Issue denial by Support
Key facts about products to monitor:
1. 10000+ tables, 70000+ indexes and 10000+ unique queries in MS SQL Server
2. 100+ on-premise installations
3. Regression testing takes 2-3 months – never run for hot fixes in the old versions
4. Still brings a lot of money to our customer
| On the way to Azure: monitoring and analytics based on Elastic stack

4. [TOPIC]
Baseline architecture: Considered Options
Products (free/commercial)
1. Zabbix / Nagios / OpenNMS /
New Relic
2. Azure application insights
3. Elastic stack
Major driving requirements
1. OOTB features
2. Flexibility
3. Performance, Availability and
Scalability
| On the way to Azure: monitoring and analytics based on Elastic stack

5. [TOPIC]
Baseline architecture: Solution Context View
Monitoring System
Support EngineerSupport Engineer
QAQA
How latest fix
affected performance?
What is this process
at all doing?
What are the most important
user processes?
What is User A doing?
Performance metrics Application logs
Configuration changes Analytics
Monitored Product #1
Monitored Product #N
Metrics and data
Metrics and data
Legend
System Function
DeveloperDeveloper
Product ManagerProduct Manager
Why is my
process slow?
User AUser A
| On the way to Azure: monitoring and analytics based on Elastic stack

6. [TOPIC]
New requirements: limitations and drivers
| On the way to Azure: monitoring and analytics based on Elastic stack
1. Monitored products targeting Azure
2. No way to do cross-customer analysis
3. Support costs optimization
4. Higher availability
5. Higher interoperability

7. [TOPIC]
New requirements: cloud-native options
| On the way to Azure: monitoring and analytics based on Elastic stack
1. Application Insights
2. Operations Management
Suite
3. Azure Search
1. Provide similar
experience for cloud and
on-prem
2. Maintain single codebase
vs.

8. [TOPIC]
The solution: Overview
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS

9. Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
[TOPIC]
The solution: Data collection
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Agent Service
Web Jobs
Monitoring Service
Web Jobs
Application
Insights
Stream Analytics
HTTPS
HTTPS
1. Azure Functions
2. Azure Batch
3. Service Fabric
4. Cloud Services
5. IaaS VMs
6. Web Jobs

10. Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
[TOPIC]
The solution: Staging area
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Event Hubs
Staging area
Event Hub
1. Queue
2. Service Bus
3. Event Hubs

11. Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
[TOPIC]
The solution: Services
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Availability set (Windows VMs)
subnet /28
User
HTTPS
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Thresholds
Jobs
Pentaho
Jobs
Ingest
Jobs
1. Azure Functions
2. Azure Batch
3. Web Jobs
4. Cloud Services
5. IaaS VMs
6. Service Fabric
1. Kibana (per tenant)
2. ETLs
3. Reverse Proxy
4. Ingest Jobs (new)

12. Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
[TOPIC]
The solution: ElasticSearch
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
1. Elastic Cloud
2. AWS ElasticSearch
3. Private ElasticSearch hoster
4. IaaS VMs
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Backup Storage
Azure
Diagnostics
Elasticsearch
data
Elasticsearch
master
Load balancer
Load balancer

13. [TOPIC]
The solution: Summary
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS

14. Thank you!
Your Questions are Welcome! 
Artsiom_Baranovskiy@epam.com
| On the way to Azure: monitoring and analytics based on Elastic stack