Despite all the recent advancements in the operations management field, data center management today still largely remains as a black art. Administrators have limited visibility into their data center operations today and yet they have to make important operations management decisions every day. A typical data center generates about a Billion data points every day. A lot of insight could be gathered from this data but due to the large volume and scale, on-premise software solutions only collect limited subset of this data. This limits them to a very narrow view of the data center. We at CloudPhysics have taken a different approach to this problem. We created an analytics platform in the cloud, that provides the ability to query, slice and dice and mashup the data with multiple data-sources. This approach not only yields incredible insights but also solves many of the teething operational management issues that have not been solved before. In this talk we give an overview of the data center metadata and provide details on how CloudPhysics handles this data at scale using its platform.

1. Big
Data
for
Better
Datacenters
!
Balaji
Parimi
@vimAPIGuru
Krishna
Raj
Raja
@esxtopGuru
!
Strata
2014,
Santa
Clara

2. World
Today

3. The
Irony

4. Analytics

5. 1990s
Application
Application
OS
OS
Compute
Compute
Storage
Storage
Network
Networking
becomes
mainstream
!
No
effective
means
to
pool
storage
and
compute
capacity
!
100s
of
silos
to
manage

6. 2000s
VM
VM
VM
VM
VM
VM
VM
VM
Hypervisor
Hypervisor
Compute
Compute
Virtualization
becomes
mainstream
!
Improved
resource
pooling
!
Storage
Network
Increased
utilization

7. 2010s
VM
VM
VM
VM
VM
VM
VM
VM
Software
Defined
Datacenter
Compute
Cloud
has
become
mainstream
!
Centralized
management
Compute
Storage
Network
70%
of
all
the
x86
OS
instances
run
in
virtual
machines,
80%
by
2016
!
-­‐
Gartner
(2012)

8. Data
Centers
Have
Evolved
But
IT
Operations
Has
Not

9. IT
Operations
Today
Retrofitted

10. IT
Operations
Today
Tedious

11. IT
Operations
Today
Paralyzing

12. IT
Operations
Today
Lot
of
Guesswork!

13. IT
Operations
Should
Be
Data-­‐Driven,
“Intuition
and
Guesswork”
Should
be
Replaced
By
“Analytics
and
Predictive
Intelligence”

14. Datacenter
Efficiency
Statistically Significant Machine
Metadata Available for Data
Scientists

15. Can
Private
Clouds
Match
The
Efficiency?
Data
Scientists
+
Machine
Metadata
Half
Million
Private
Clouds

16. Operational
Metadata
Per
Day
1
Billion
Data
Points
500
Virtual
Machines
50
Servers
Private
Cloud
25GB

17. Massive
Scale
650
Million
Users
60 million tweets
1
Billion
Users
2.5 billion pieces of content
500,000
Datacenters
50
Million
VMs
Quadrillion datapoints (1015)

18. Challenges
Management
Server
Control
Plane
VM VM VM
VM VM VM
(Predict/Alert)
VM VM VM
VM VM
Server
Server
Server
Server
Data
Center
VM
(Collect)
Management
Driven
by
Collective
Intelligence

19. Data
Model
ORG
CLUSTER
MGMT SERVER
SERVER
VSWITCH
STORAGE!
CLUSTER
VNIC
VDISK
VM
VPORTGROUP
DISK STORE
One to Many

20. Types
of
Data
Streams
PERFORMANCE
CONFIGURATION
TASK & EVENTS
EXTERNAL DATA
SOURCES

21.
Collector
Administrator
On-­‐Demand
Analytics
Engine
Data
Stream
Complex
Stream
Processing
Partner
Prediction
and
Simulation
Engine
Distributed
Datastore
Data
Analyst
Pre-­‐Alerts
and
Notification

22. Enabling
Private
Clouds
Extract
/
Transform
Job
Scheduling
Persist
Stream
Mashup
Model
Resource
Management
Alerting
Prediction
Collective
Intelligence

23. Use
Cases

24. Predicting
Out
of
Disk
Space
Event
Datastore
1
Datastore
2
Datastore
3
Datastore
Capacity
100%
75%
50%
25%
0%
April
May
June
July
Disk
Space
Usage
is
Hard
to
Predict
August

25. Predicting
Out
of
Disk
Space
100%
Full
90%
Full
80%
Full
Probability
100%
75%
50%
25%
0%
<
1
Month
6
Months
1
Year
Probability
Distribution
Using
Monte
Carlo
Simulation

26. Discovering
Causality
1. Virtual
Machine
18
2. Virtual
Machine
23
3. Virtual
Machine
108

27. Predicting
Potential
Outage

28. Predicting
SSD
Performance
Impact
Latency
Reducjon
100%
75%
50%
VM
1
VM
2
cy
ten
a
et
L
g
Tar
25%
VM
3
0%
16
GB
B
4
G
2
32
GB
B
2
G
6
64
GB
Solid
State
Drive
Cache
Per
VM
128
GB

29. Predicting
Configuration
Outliers

30. Predicting
Performance
Outliers
Organizaion
A
Organizajon
B
Organizajon
C
CPU
Co-­‐efficient
90
67.5
45
22.5
0
0
3
6
Virtual
Machine
#
9
12

31. Some
Closing
Thoughts

32. Towards
a
New
World

33. Thank
You
!
Balaji
Parimi
@vimAPIGuru
Krishna
Raj
Raja
@esxtopGuru