How eStruxture Data Centers is Using ECE to Rapidly Scale Their Business

Eric Charpentier | Enterprise Architect
eStruxture Data Centers
September 18th 2019
How eStruxture Data Centers is
using ECE to Rapidly Scale Our
Business

WHO IS ESTRUXTURE?
AND WHY DOES IT MATTER?
!2

!3
eStruxture | TIMELINE OF FUNDING  
AND ACQUISITIONS
Acquisition of Montreal-
based Netelligent Hosting
Services and launch of
eStruxture with initial
funding led by Canderel
February
2017
Second round of funding by
CDPQ
eStruxture announces
expansion of MTL-1 facility
doubling cabinet space and
increasing power capacity  
to 5MW
June 2017
Acquisition of Montreal
Gazette printing facility for
2nd data center expansion
in Montreal
April 2017
November
2017
MTL-1
MTL-2
Third round of funding  
by Fengate
November
2017

!4
Acquisition of Kolotek from  
Gaz Metro
Expansion to Vancouver via
the acquisition of Backbone
Datavault facility
April 2018
Expansion of VAN-1 and
acquisition of second
Vancouver area data center
(Burnaby)
February
2018
January 2019
MTL-3
VAN-2
VAN-1
New credit facility with
Scotiabank, NBC and IQ
June 2019
Expansion in Calgary
through the acquisition of
the Shaw data center
August 2019
CAL-1

eStruxture | CURRENT DATA CENTERS
!5
High-Density Power in The
Heart 
of Downtown Montréal
▪ 5MW
▪ 25,000 square feet
▪ 20+ carriers present in the
facility
▪ High power density available
(30kW/cabinet available
standard)
Capacity and Flexibility
▪ 30MW
▪ Large, expandable facility
standard)
▪ Dedicated office and storage
space
Tier III Uptime Certified
Facility
▪ 1.2MW
standard)
▪ Easy to access, away from
downtown traffic
Innovative and Strategically
Located Facility
▪ 2.4MW
(up to 30kW/cabinet)
▪ All major carriers on site
▪ Mount Pleasant location – a
hotbed for media studios
MTL-1 MTL-2 MTL-3
Scalable, Tier III Uptime
Certified Facility
▪ 6MW
standard)
▪ Multiple customer amenities  
on site
VAN-1 VAN-2 CAL-1
State-of-the-art,  
Expandable Facility
▪ 10MW
(up to 30kW/cabinet)
▪ Located in Burnaby – proximal
to stable, plentiful power and
dense fiber optic
infrastructure

!7
BACKGROUND KEN NAME
CUSTOMER DEMOGRAPHICS
GOALS HOBBIES & INTERESTS
COMMON OBJECTIONS
BIGGEST FEARS
▪ Experienced manager with a
dedicated 7X24 L1/L2 support
team
▪ Ken Oliver
▪ Father, husband great colleague
▪ Colocation services
▪ Network
▪ Managed services
▪ Cloud services
▪ Too many systems
▪ Alarm pollution
▪ Too many manual interventions
▪ Support his team
▪ Provide the best possible
service to his customers
▪ Meet the highest levels of
industry compliance
▪ Overworking his employees
▪ Missing customer commitments
▪ Not working on the right things
▪ Play with his kids
▪ Star Wars
▪ Make sure that everyone has a
fun day

!8
WELCOME TO THE REALM OF DATA-DRIVEN DATA CENTERS
Imagine a world where you, as a support
manager, can rest easy at night, knowing
that your infrastructure is monitored,
analyzed, and troubleshooted
automatically so that your team can focus
on serving our customers.

9
MACHINES DO THE
BORING, REPETITIVE
TASKS AND HUMANS
SOLVE PROBLEMS

!10
Eliminate the duplication of costly tools that serve the same purpose
Reduce manual interventions during support/installation
Eliminate the eyes on glass operations model
Speed up time to integration upon acquisition/construction of a site
Avoid the old mindset of command & control approach to NOC management
Eliminate the exceptions to manage for each sites (80/20 rule)
WHAT PROBLEMS ARE WE TRYING TO SOLVE

!11
TRADITIONAL METHODOLOGY
▪ Technology based approach
▪ Cooling
▪ Electrical
▪ Mechanical
▪ Network
▪ Hardware
▪ Application
▪ Vendor based tools
▪ Vendor locked toolsets
▪ DCIM (partial monitoring)
▪ Multi-screen approach
▪ Workload intensive
Connect to tools and databases
to aggregate your data which
increases the risks of data loss,
connectivity problems and
latency.

!12
A NEW APPROACH 
Based on philosophy, scalability and innovation
Monitor Potential Bad Things1
Alert before they happen
Monitor Actual Bad Things2
Alert when they do happen, which is,
unfortunately, inevitable
Monitor Good Things3
Alert when they stop happening
4 Steps of MonitoringOur Core Tenets
Identify as many problems as possible.  
Good monitoring doesn't just tell you when
your site is completely down.
Identify problems as early as possible.  
The sooner you know about an issue, the
better chance you'll have to address it
before it affects users. Longer lead times
are your friend.
Generate as few false alarms as
possible.  
False alarms (aka false positives) can lead
to "alert fatigue“
Tune and Continuously Improve4
Iterate!

!13
A NEW APPROACH 
Based on philosophy, scalability and innovation
Why Elastic?
1.Multiple sources of data under one roof
2.Ingest data from app logs, device logs,
infra logs
3.It indexes it with its own tool,
correlates, infers data and can
aggregate it
4.Tool to understand the data and answer
questions we have
5.Can build visuals of the data
6.Xpacs for extra features
NextGen monitoring vs today
1. Automated monitoring vs Manual
monitoring
2. What are the goals for monitoring vs
alarm storms
3.What resources will we monitor vs A
sea of resources
4.How often will we monitor the
resources vs Constant monitoring
5.Who should be notified when
something goes wrong vs Notify
everyone all the time

14
I HAVE HEARD THIS
PROMISE BEFORE…
Image courtesy of StateTechMagazine.com

Why did we choose
ECE
One word: Versatility
High availability
Ability to build our architecture based on availability
zones to eliminate the risk of downtime
Hardware customization
Ability to choose the hardware appropriate to the
business condition
Separation of roles for scalability
Ability to re-engineer the architecture to scale. If we
require more resources, we can get additional nodes or
modify the architecture to separate allocators, directors
and proxies (depending on need)
Speed
Ability to rapidly create new, customized deployments on
demand without having to provision new nodes of VMs
Elastic Cloud Enterprise

!16
ITERATE – PHASE 1
Watcher
Dashboards
Logstash pipeline
?

MODBUS PROTOCOLS
Sometimes, hybrid functions work best
input{
exec {
command => "/opt/rh/rh-python36/root/usr/bin/python3.6 /etc/logstash/scripts/script_name.py ip_address 168 2 f7
--device=1"
interval => 10
add_field => { host => “host_name"}
add_field => { IP_Address => “ip_address"}
add_field => { CLLI_Code => “device_name"}
add_field => { Country => “Country"}
add_field => { Location => “City"}
add_field => { Floor => “Floor"}
add_field => { Room => “Room"}
add_field => { DeviceType => “Equipment_type"}
add_field => { DeviceNumber => “Equipment_number"}
type => "OutputFrequency"
}
output {
kafka {
codec => json
topic_id => “topic_name"
bootstrap_servers => “kafka_ip:customized_kafka_port"
client_id => “client_name"
}
}

SNMP WALKS VIA CONF FILE
Do what work!
input{
snmp {
interval => 10
get => [
"1.3.6.1.2.1.1.5.0"
]
walk => [
"1.3.6.1.4.1.476.1.42.3.9.20.1.20.1.2.1" ,
"1.3.6.1.4.1.476.1.42.3.9.20.1.10.1.2.1"
]
hosts => [
{host => "udp:ip_address/port" community => “community_name" version => “snmp_version" } ,
{host => "udp:ip_address/port" community => "community_name " version => "snmp_version " } ,
{host => "udp:ip_address/port" community => "community_name " version => "snmp_version " }
]
}
}
filter {
de_dot {}
}
output {
kafka {
codec => json
topic_id => “topic_name"
bootstrap_servers => “kafka_ip:customized_kafka_port"
client_id => “client_name"
}
}

!19
ITERATE, ITERATE – PHASE 2
Watcher
Dashboards
Logstash pipeline
Multiple
Logstash
Modbus
SNMP
SNMP
poller
IT
Life cycle
management
Users & Roles

Confidential and Proprietary !20
THE CASE FOR ENRICHING DATA
9/16/2019
Consumption Self-serve
CONSUMPTION
▪ Enable consumption-
based billing to all
sites
▪ Enable consumption-
based billing to all
services
▪ Eliminates days of
manual labor in favor
of processing in
seconds
CORRELATION
▪ All systems and/or
equipment logs into
one data repository
▪ Ability to relate
simultaneous alarms
to one root cause
▪ Ability to relate all
impacted customers
to events in seconds
SELF-SERVE
▪ Customer portals
▪ Maintenance
notifications
▪ Ad hoc requests
▪ Ability to drill-down
the data chain in
seconds

MAXIMIZE THE USE OF YOUR INPUT PLUGINS
JDBC Input Plugin
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
jdbc_user => "mysql"
parameters => { "favorite_artist" => "Beethoven" }
schedule => "* * * * *"
statement => "SELECT * from songs where artist = :favorite_artist"
}
}

Watcher
Dashboards
!22
ITERATE, ITERATE, ITERATE
Logstash pipeline
Multiple Logstash
Modbus SNMP
SNMP
poller
IT
winlogbeat
metricbeat
Life cycle
management
APM
Machine
Learning
Canvas
Users & Roles
Logstash pipeline
Client portal

24
WOW, THAT’S REALLY
ELEGANT…

Over power alarms
Per month before we
started using Watcher
RESULTS AT A GLANCE
Additional panels
1344 breakers more
monitored than before
Per week
Ability to set a range and
interval criteria compared
to the devices which are
point-in-time
7000+ 16 600

THE NEED TO MODIFY HISTORICAL DATA
How we learned to go with the data flow
GET /test_topic/_search
{
"query": {
"match": {
“device_label": “CAMTL200EL08UPS1001"
}
}
}
}
POST test_topic/_update_by_query
{
"query": {
"match": {
“device_label": "CAMTL200EL08UPS1001"
}
},
"script": {
"source": "ctx._source.device_label='CAMTL200EL08UPS2001'"
}
}

!27
• Make your work visible from current status to the roadmap
• Use a JSON friendly editor (really!)
• Version control everything
• Create a production environment and a development environment completely
isolated from one another
• Define your storage policy before you start
• Bring all your logstash and kafka logs into separate indices to analyze on a
regular basis while you develop
• Don’t assume that your environment will be static (dynamic mapping?!?) but
also basic components like device names
LESSONS LEARNED

!28
And while we’re not quite at the stage
where Ken can rest easy at night, we are
now allowing him to take the occasional
nap… but not too many…

Thank you!
Presented by Eric Charpentier
Enterprise Architect
eStruxture Data Centers
Eric.Charpentier@estruxture.com
https://www.linkedin.com/in/eric-charpentier/

How eStruxture Data Centers is Using ECE to Rapidly Scale Their Business

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How eStruxture Data Centers is Using ECE to Rapidly Scale Their Business

Similar to How eStruxture Data Centers is Using ECE to Rapidly Scale Their Business (20)

More from Elasticsearch

More from Elasticsearch (20)

Recently uploaded

Recently uploaded (20)

How eStruxture Data Centers is Using ECE to Rapidly Scale Their Business