Scipy stack vs. InfluxDB and Grafana

1. 1© Würth Phoenix … more than software
SciPy Stack vs.
InfluxDB and Grafana
Susanne Greiner, PhD
http://www.neteye-blog.com/
https://www.linkedin.com/company/wuerth-phoenix
https://www.linkedin.com/in/susanne-greiner-5746ba1b/

2.  IT and Consulting Company of the Würth-Group
 Headquartered in Italy, Europe-wide presence, more than 160 highly skilled employees
 International experience in Business Software and IT Management
 Core competencies in trading processes, wholesale distribution and logistics
 Microsoft Gold Certified Partner, ITIL certified, OTRS Preferred Partner
2
ABOUT WÜRTH PHOENIX
Facts & figures
 More than 1,000 customers
worldwide
 Over 70 successfully
implemented ERP and CRM
projects
 400 NetEye customers
 HQ in Italy
We improve business productivity by
delivering world class software
solutions with a team of highly
motivated and skilled IT experts
© Würth Phoenix … more than software

3. OUR STARTING POINT
© Würth Phoenix … more than software 3
Application Performance Monitoring of Business Critical Applications
Application Server
ERP SAN NAS
Virtualization
Datacenter
on premise
Messages
Collaboration
SFA
CRM
ERP
Company LAN
Home network
Internet
CRM
E-Shop
Custom
application
Mail
Database

4. OUR STARTING POINT
© Würth Phoenix … more than software 4
Human Expert | Expertise | Experience | Domain Knowledge
human expert
Tool 1
Tool 2
Tool 3
Tool 4
Tool #
METRICS

5. PROBLEMS
© Würth Phoenix … more than software 5
 More customers than human experts
 Experience is difficult to split
 Expertise is difficult to split
?

6. Problem Analysis
© Würth Phoenix … more than software 6
 How is the human expert spending his time?
 How can we optimize his time?
 Which parts of his work can be automated?
Type Duration Location Impact Solution Prevention

7. Solution Nr 1: Scipy Stack
© Würth Phoenix … more than software 7
 Numpy
 Matplotlib
 Seaborn
 Pandas
 Scikit Learn
Seaborn

8. Solution Nr 1: Tasks
© Würth Phoenix … more than software 8
Read data from different data sources
Add visualization
Prepare reports on a daily/weekly basis

9. Solution Nr 1: ADVANCED TASKS
© Würth Phoenix … more than software 9
Anomaly Detection
Forecasting
Metrics
MODEL
STANDARD
BEHAVIOR
TRAIN
DATA
• Historical data of same metric
• Historical data of similar metric
• Historical data of similar machine
WHAT DO WE EXPECT?
TEST
DATA
• The farther away from expectations
the higher the AScore
• AScore: different and rare
HOW FAR ARE WE FROM OUR EXPECTATIONS?
ASCORE

10. TURNING POINT
© Würth Phoenix … more than software 10
WHAT ABOUT IF THIS WAS DYNAMIC AND NAVIGABLE?
Can we do it?
YES!

11. Solution Nr 2: Tasks
© Würth Phoenix … more than software 11
Get data from different data sources into InfluxDB
Create dashboard(s)
Maintain additional value

12. Solution Nr 2: Request – Add multi-level dashboards
© Würth Phoenix … more than software 12

13. Solution Nr 2: Request – Add User Experience
© Würth Phoenix … more than software 13
http://www.alyvix.com

14. Solution Nr 2: Request – x
© Würth Phoenix … more than software 14
New request → How do I get the new data into InfluxDB
Telegraf for Windows - Linux
Collectd for HP-UX, AIX, Solaris
VMWare ESX
Analytics
Alyvix on a Windows VM
Visual Synthetic Monitoring
Alyvix VM
Windows 64bit
time series db
Representation on
dashboards in real
time
IT Operations
Customer
NetEye VM
Linux CentOS6
Nats.io
Streaming Server
Nats.io
Streaming Server
Würth Phoenix NetEye Cloud
Agents for Servers

15. Solution Nr 2: NetEye ITOA
© Würth Phoenix … more than software 15
Server
Performance
Metrics
Anomaly
Score
Batch
Analysis
Alyvix
User
Experience

16. Scipy Stack vs. InfluxDB&Grafana
© Würth Phoenix … more than software 16
Data accessible via InfluxQL
Zooming Navigation
No more printing ;D

17. Moving mean
© Würth Phoenix … more than software 17

18. Calculation between different time periods
© Würth Phoenix … more than software 18
Reassigning timestamps
Mean over WEEKS of the last month
GROUP BY - Continuous Queries

19. Use Case: Performance monitoring
© Würth Phoenix … more than software 19
Questions
• Does my business-critical
application run well?
• Was it right to invest in
software/hardware?
• Did our performance change
compared to the last month?
Infrastructure
Batch Data
Application-specific
metrics
?
THE
COMPLETE
PICTURE
Scope oriented
visualization
User Experience
Anomaly Detection
SQL Database
Growth
Machine Learning
& Deep Learning
SQL Query
Details/ Statistics
Forecasting

20. Use Case: Performance monitoring
© Würth Phoenix … more than software 20

21. Use Case: Beegy
© Würth Phoenix … more than software 21
decentralized energy management

22. Use Case: SmartLand
© Würth Phoenix … more than software 22
Smart Region South Tyrol
Sustainability
Water consumption: 250 Mio m³/ year
Irrigation: 110 Mio m³/ year
Potential saving: > 50% (intelligent irrigation)

23. Use Case: SmartLand
© Würth Phoenix … more than software 23

24. © Würth Phoenix
www.wuerth-phoenix.com
24… more than software