1. © 2020 Allscripts Healthcare, LLC and/or its affiliates.
Allscripts
IT Monitoring
Solution
How Allscripts Uses InfluxDB to
Monitor its Healthcare IT Platform

2. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 2
• Introduction
• Allscripts company overview
• The Problem
• Solution Requirements and Overview
• Outcomes and Future Projects
• Q&A
Agenda

3. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 3
Introduction

4. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 4
Allscripts Company
Overview

5. 5
Global Healthcare Crisis
How we
Interact
COVID-19
Protection
Working
Location
Social distancing
Personal Protection
Equipment (PPE)
Stay at (work from)
home
Challenges we see in
healthcare are
unprecedented

6. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 6
Together, we can connect medicine
We can change what’s possible
Strengthen
Your Foundation
Drive quality care with
essential solutions
Engage
Your Patients
Coordinate care and help
patients take charge of their
health
Connect
Your Community
Deliver data and insights to
where they are needed most
Build
Your Future
Ensure organizational agility
and capacity to thrive

7. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 7
Helping you deliver
smarter care
• Industry-leading Specialty Content
• Actionable Insights at Point of Care
• Intuitive Workflows for Ease of Use
• Care Coordination Across the Continuum

8. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 8
We’re recognized by the industry
• 2017: Mindshare Population
Health Management
• 2016-2017: Mindshare Leader
in Revenue Cycle Management
• 2016–2017: Mindshare Leader
in Patient Engagement
• 2018: #1 Population Health
Management Market Leader
• 2020: Best in KLAS Global
(non-US) Acute Care EMR –
Canada
• 2018 – 2019: Best in KLAS
Anesthesia—iProcedures
2017 Highest in usability in
small practice ambulatory
EHR|PM
• 2016: Highest in usability in
Population Health Management
Report—Population Health UX
• 2015–2016, 2013: Lab Best in
KLAS
• 2020: #1 Critical access, specialty
hospitals, and integrated
community health systems
• 2020: #1 EHR-Based Analytics
& Support
• 2020: #1 Ambulatory EHR for
Government Providers
• 2020 Black Book #1 Practice
Management solution (10-49
physicians)
• 2019: #1 Healthcare IT vendor in
UK, Canada, Australia, and New
Zealand
• 2019: #1 Ambulatory EHR for
Multispecialty clinics
• 2017-2020: #1 Inpatient Hospital
EHR for Community Hospitals 101-
250 beds
• 2014-2017, 2019: #1 Large
Inpatient Hospital 250+
• 2014-2020: #1 Ambulatory EHR
26-99
• 2018-2019: #1 Rehabilitation
Hospitals
• 2018: #1 Hospital Chains, IDNs and
Networks for Integrated EHR,
Interoperability, PHM & RCM
Solutions
• 2019, 2017: Microsoft Health
Partner of the Year
• 2015: Microsoft CXO Next
Innovation Award
• 2015: Microsoft Health User
Group Innovation Award
• 2020: US Healthcare
Interoperability EMR Market
Leader
• 2019: Best Practices Award—US
Population Health Management
Product Leadership Award
• 2019, 2017: Best Practices
Award—US Ambulatory Revenue
Cycle Management Product
Leadership Award
• 2018: Becker’s Hospital Review
Top Places to Work in
Healthcare

9. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 9

10. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 10
The Problem
Various Cloud Solutions
3rd Party Solutions
Self-Hosted
Solutions
Multiple Allscripts
Datacenters
Various Monitoring Solutions
3rd Party Datacenters
Different VisualizationsDifferent Alerting Methods
Support
Operations and Development
Management

11. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 11
Unified platform to monitor any system,
Allscripts solutions, and allow custom
development
Key Solution Requirements
Flexible
Economical
Adaptable
Scalable
Replace costly commercial tools and
leverage solution to monetize new
service offerings
Monitor any environment – Hosted, Self-
hosted, 3rd Party, Cloud, Hybrid and rapidly
changing technologies
Extreme scalability and implement
quickly through orchestration

12. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 12
No downtime required to implement
meeting the demand of highly available
healthcare systems
Additional Solution Requirements
Zero
Downtime
Security
Modular
Reporting
Meet Customer, Allscripts, HIT, and in
some cases government/regional
security requirements
Swap out modules without affecting the entire
monitoring solution
Centralized, accessible, robust
reporting and dashboards

13. © 2020 Allscripts Healthcare, LLC and/or its affiliates.
Solution Overview

14. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 14
Solution Overview
Three Primary Components
 Configuration Management
 Log Analytics
 Telemetry/Metrics Analytics
In Scope:
 Operating System: primarily Windows Server; some Linux
 Application: Middleware (IIS, SQL Server, etc.) and Business services
Out of Scope:
 Infrastructure (e.g. Storage, Network, & Virtualization layers)
 APM / Distributed Tracing (for now)
Composed of open-source (enterprise backed) pluggable, modular, event driven
frameworks
Stretch goal: provide a framework for automation/orchestration of product installation/upgrade/troubleshooting activity, in
addition to monitoring

15. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 15
Telemetry
Vs. Existing Proprietary Tools
 Operational and Cost complexity
Vs. TimeScaleDB
 Product Maturity
 Data Model Flexibility: TagSet vs Relational Schema
 TimeScaleDB lacks alerting
Vs. Prometheus
 Push vs Scraping
 Horizontal Scalability
Why InfluxData?

16. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 16
Configuration Management & Log Analytics
SaltStack: Configuration Management
Vs. Ansible, Chef, Puppet, and Azure Automation
 Push vs Pull
 Conflicting/Multi-Homing agents
 Salt's event driven architecture and modularity fit well with project requirements
 Python vs Ruby
ElasticSearch: Log Analytics
Vs. GreyLog, Splunk, and Azure Log Analytics
 Cost, Complexity; Keep it simple

17. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 17
Visualization & Alerting
Visualization:
Grafana - Single pane of glass
Alerting
Telemetry: Kapacitor
Logs: ElasticSearch Alerting vs Open Distro (work in progress)

18. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 18
Architecture: Data Collection
* Light outlined arrows are network connections
* Dark solid arrows are data flow

19. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 19
Architecture: Data Visualization
* Light outlined arrows are network connections
* Dark solid arrows are data flow

20. © 2020 Allscripts Healthcare, LLC and/or its affiliates.
Outcomes and
Future Projects

21. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 21
Outcomes:
• Fast delivery – thousands of agents deployed in
minutes, not manually over days or weeks
• Smaller footprint to manage, maintain and
secure; reducing operational costs and security
risks
• New service revenue opportunities – solution
allows us to deploy monitoring in areas not
feasible in the past due to costs
• Improved support and service delivery -
centralized dashboards accessible directly
from ITSM solution
Learning Experiences:
• Tuning InfluxDB Disk IO performance
• Saturating network PAT & bandwidth
• Saturating Telegraf Relay request volume
• Input-specific jitter for Telegraf plugins
(#3634)
• Tuning batch size, metric buffers, and flush
intervals
• TLS client certificate monitoring (#7754)
• Fixed in a day! Thanks @danielnelson!
Outcomes

22. © 2020 Allscripts Healthcare, LLC and/or its affiliates. 22
• Consolidate custom developed monitoring tools using Influxdata and other
solutions – Shim Plugin for custom SQL Queries
• Resolve challenges with Telegraf relays in multi-datacenter configuration
• Consolidate commercial monitoring tools and footprint – line of sight to
drastic cost reduction in 2021
• Leverage the solution to “monitor the monitoring systems”
• InfluxDB 2.0/Flux
Future Projects

23. 23
Questions?

24. facebook.com/allscripts
allscripts.com
@allscripts
linkedin.com/company/allscripts

25. We look forward to bringing together our
community of developers to learn, interact
and share tips and use cases.
www.influxdays.com/virtual-experience-2020/
November 10 – 11, 2020
Virtual Experience