A slides which is presented when speaking in the session of OpenInfra Summit Vancouver 2023, about information management.

1. © DMM.com
HIROYASU OHYAMA - DMM.com LLC
Why SSoT is important for infrastructure operation
and how to make it
2023-06-15 OpenInfra Summit Vancouver ‘23
1

2. © DMM.com
Introduction
2
Software Engineer
- Former OpenStack Technical Contributor
(oslo.messaging)
- StackStorm (IFTTT workflow engine) Contributor
- AirOne (yet another SSoT solution) Developer
About my-self

3. © DMM.com
Introduction
3
(c.f. https://dmm-corp.com/business/)
About our company (DMM.com)

4. © DMM.com
Introduction
4
Centering on video-streaming service and other businesses in any areas.
- Areas: 17
- Businesses: 60+
- Number of members: 39.14 million
- Sales: 305.5 billion YEN
(c.f. https://dmm-corp.com/business/)
About our company (DMM.com)
[*1]
[*2]
[*1] Number of DMM.com service subscribers (as of the end of February 2021)
[*2] DMM.com, DMM.com Securities, DMM.com BASE and other consolidated subsidiaries (as of the end of February 2021)

5. © DMM.com
Introduction
5
Hosting any services of any businesses
- Compute nodes for production: 7k+
- Maximum Traffic: 505.9+ Gbps
About our infrastructure

6. © DMM.com
Case: DMM.com
- Straggling with distributed and duplicated information
6

7. © DMM.com
Summary of this story
7
- How we’ve been troubled by scattered data sources
1. Story of the past before SSoT (dark age)
- The reason why we were reached to the idea of SSoT
- Introduction of our SSoT system
- Reference information about SSoT
2. An ideal way to manage information
- Problems which is caused by SSoT
3. Hardships of SSoT

8. © DMM.com
8
Information to operate entire infrastructure
Hardships in that (dark) age
- Data center structure (e.g. site, rack capacity, power supply, etc...)
- Physical appliances (e.g. Servers, NW switches, LBs, Storage, FW, etc…)
- Peripheral equipments related with appliance (e.g. SFP module, Video cards, etc...)
- Logical instances related with appliance (e.g. VM, IP address, etc...)
- Support expiring date of each appliances
- Who use our appliance and instance and purpose (e.g. production / development)
- Configuration to deploy our infrastructure service (e.g. environmental variables)
- Fee cost for each lines of telecommunication

9. © DMM.com
9
- RackTables [*]
Information management method (at that time)
Hardships in that (dark) age
(*) https://www.racktables.org/
A datacenter asset management system for datacenter, appliance
and part of logical information.

10. © DMM.com
10
- RackTables [*]
Information management method (at that time)
Hardships in that (dark) age
(*) https://www.racktables.org/
A datacenter asset management system for datacenter, appliance
and part of logical information.
- (tons of) SpreadSheets
All information except for the one that is managed by Racktables.

11. © DMM.com
1. Spawning duplicated data-sheets (1)
Hardships in that (dark) age
11
To separate information that has confidential data
TeamA
Authentication information is necessary,
but I don't think it's good to save those
confidential ones are also saved in the
same place.
Racktables [public]
Infrastructure
Information

12. © DMM.com
1. Spawning duplicated data-sheets (1)
Hardships in that (dark) age
12
To separate information that has confidential data
TeamA
Authentication information is necessary,
but I don't think it's good to save those
confidential ones are also saved in the
same place.
Authentication data
Spread Sheet [private]
Racktables [public]
Infrastructure
Information

13. © DMM.com
2. Spawning duplicated data-sheets (2)
Hardships in that (dark) age
13
Others don't know existence of its sheets
TeamA TeamB
Authentication data
Spread Sheet [private]
Racktables [public]
Infrastructure
Information
Authentication information is necessary,
but I don't think it's good to save those
confidential ones are also saved in the
same place.
...

14. © DMM.com
Hardships in that (dark) age
14
2. Spawning duplicated data-sheets (2)
Others don't know existence of its sheets
TeamA TeamB
Authentication information is necessary,
but I don't think it's good to save those
confidential ones are also saved in the
same place.
...
Authentication data
Spread Sheet [private]
Racktables [public] Github Repository [private]
Confidential Data
Infrastructure
Information

15. © DMM.com
Hardships in that (dark) age
15
3. Tragedy of scattered data sources (inconsistency)
Both of them have same information but these columns might not be same
Hostname Username Password
… … …
stg.web.serviceA root A2zXiJ
… … …
...
- hostname: stg.web.serviceA
username: root
password: Lk9mMWKh
...
Infrastructure
Information Authentication data
Spread Sheet [private]
Racktables [public] Github Repository [private]
Confidential Data

16. © DMM.com
Hardships in that (dark) age
16
3. Tragedy of scattered data sources (inconsistency)
Both of them have same information but these columns might not be same
Hostname Username Password
… … …
stg.web.serviceA root A2zXiJ
… … …
...
- hostname: stg.web.serviceA
username: root
password: Lk9mMWKh
...
Increase management cost!
Infrastructure

18. - Thinking about ideal information management,
then carry it out.
18

19. © DMM.com
Ideal situation to solve these problems
19
Transitional period to present day
Servers / VMs IP Addresses
Network
Whole information is managed in a system and each of them are associated
TeamB
TeamA TeamC

20. © DMM.com
Ideal situation to solve these problems
20
Transitional period to present day
Servers / VMs
Authentication data
IP Addresses
Network
Support
Accounting
Whole information is managed in a system and each of them are associated
TeamB
TeamA TeamC

21. © DMM.com
Ideal situation to solve these problems
21
Transitional period to present day
Servers / VMs
Authentication data
IP Addresses
Network
Accounting
Whole information is managed in a system and each of them are associated
Support
TeamB
TeamA TeamC

22. © DMM.com
Our major requirements for SSoT
Transitional period to present day
22
1. Enable to save information whatever we want and expand it on-demand.
2. Enable to set permission according to user and group for each attributes.

23. © DMM.com
- Decouple mechanism from data context
(enable to define any kind of schemaful data structure)
The main concepts of AirOne
Information, which is managed by SSoT
23
- Enable to connect with each data instances
(registered data seems like to be a node in graph networks)
- RBAC for each schema, instance and attribute layers
(enable to set flexible permission setting for user)
(*)
(*) https://github.com/dmm-com/airone

24. © DMM.com
How to manage information in AirOne
Information, which is managed by SSoT
24
These are main data-structure to store all information we have
1. Model: blueprint or template for creating Entry
- this defines the attributes that objects of that Model will possess.
(similar to Table of RDBMS)
2. Entry: instance of Model
- this has its own set of data (AttributeValue)
(similar to Record of RDBMS)
(*)
(*) https://github.com/dmm-com/airone

25. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
25
Each Models have ”Attribute(s)” that describes what kind (type) data it has.
Network switch
・ServerRack [reference]
Site
・Address [string]
Floor
・Floor [reference]
ServerRack
・Floor [reference]
・Power supply [string]
・Maximum load weight [string]
・Is being contracted [boolean]
... ...
...
Compute node
・ServerRack [reference]
...
Load balancer
・ServerRack [reference]
...

26. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
26
Network switch
Site
Floor
Compute node
Load balancer
Sw1
Sw2
...
Server1
Server2
...
LB1
LB2
...
Rack101
Rack102
Rack103
...
Floor103 Floor205
Tokyo Inzai
...
...
ServerRack
Each Entries have value and reference to other Entry

27. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
27
Network switch
Site
Floor
Compute node
Load balancer
Sw1
Sw2
...
Server1
Server2
...
LB1
LB2
...
Rack101
Rack102
Rack103
Floor103 Floor205
Tokyo Inzai
...
...
ServerRack
...
Each Entries have referral to other Entry

28. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
28
ServerRack
Rack102
Floor
Power supply AC200V / 30A
Is being contracted
☑︎
ServerRack
Floor103
Each Entries have referral to other Entry

29. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
29
Enable to add new Attribute to Model and it affects to each Entries
ServerRack
Rack102
Floor
Power supply AC200V / 30A
Maximum load weight
Is being contracted
☑︎
ServerRack
Floor103
NEW Attribute!!

30. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
30
ServerRack
Rack102
Floor
Power supply AC200V / 30A
Maximum load weight 500 kg
Is being contracted
☑︎
ServerRack
Floor103
Enable to add new Attribute to Model and it affects to each Entries
Setting value

31. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
31
ServerRack
Rack102
Floor
Power supply AC200V / 30A
Maximum load weight 500 kg
Is being contracted
☑︎
TeamA
TeamB
TeamC
can access any Attribute values
Data access can be configured by RBAC for each Models, Entries and Attributes

32. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
32
ServerRack
Rack102
Floor
Power supply AC200V / 30A
Maximum load weight - Prohibited -
Is being contracted - Prohibited -
TeamA
TeamB
TeamC
can’t access some of Attributes
Data access can be configured by RBAC for each Models, Entries and Attributes

33. © DMM.com
An example of stored information in AirOne
Information, which is managed by SSoT
33
ServerRack
Rack102
Floor
Power supply AC200V / 30A
Maximum load weight 500 kg
Is being contracted

35. - In our on-premise infrastructure operation
35

36. © DMM.com
Application deployment
Use case of AirOne (1)
36
1. Get source-code of deploying service.
2. Read configuration and secrets (and so on) from AirOne at deploying.
3. Deploy service to Nodes according to specified configuration and variables.
Node sets
AirOne

37. © DMM.com
Application deployment
Use case of AirOne (1)
37
1. Get source-code of deploying service.
2. Read configuration and secrets (and so on) from AirOne at deploying.
3. Deploy service to Nodes according to specified configuration and variables.
Node sets
AirOne
1

38. © DMM.com
Application deployment
Use case of AirOne (1)
38
1. Get source-code of deploying service.
2. Read configuration and secrets (and so on) from AirOne at deploying.
3. Deploy service to Nodes according to specified configuration and variables.
AirOne
2
1
Node sets

39. © DMM.com
Application deployment
Use case of AirOne (1)
39
1. Get source-code of deploying service.
2. Read configuration and secrets (and so on) from AirOne at deploying.
3. Deploy service to Nodes according to specified configuration and variables.
AirOne
2
1
Node sets

40. © DMM.com
Application deployment
Use case of AirOne (1)
40
AirOne has information which Node is belonged to which NodeSet.
NodeSet-A
NodeSet-B
AirOne
NodeSet-C
NodeSet-D
(Configuration)

41. © DMM.com
Application deployment
Use case of AirOne (1)
41
AirOne has information which Node is belonged to which NodeSet.
NodeSet-A
NodeSet-B
AirOne
NodeSet-C
NodeSet-D
(Configuration)

42. © DMM.com
Application deployment
Use case of AirOne (1)
42
AirOne has information which Node is belonged to which NodeSet.
NodeSet-A
NodeSet-B
AirOne
NodeSet-C
NodeSet-D
(Configuration)

43. © DMM.com
Application deployment
Use case of AirOne (1)
43
AirOne has information environmental variables to be used for deploying
NodeSet-A
NodeSet-B
AirOne
NodeSet-C
NodeSet-D
(Secret)

44. © DMM.com
Application deployment
Use case of AirOne (1)
44
1. Get source-code of deploying service.
2. Read configuration and secrets (and so on) from AirOne at deploying.
3. Deploy service to Nodes according to specified configuration and variables.
AirOne
2
3
1
Node sets

45. © DMM.com
Application deployment
Use case of AirOne (1)
45
1. Get source-code of deploying service.
2. Read configuration and secrets (and so on) from AirOne at deploying.
3. Deploy service to Nodes according to specified configuration and variables.
AirOne
2
3
1
Node sets
?

46. © DMM.com
Data-center information management
Use case of AirOne (2)
46
Provide a feature of DCIM that provided former system

47. © DMM.com
Data-center information management
Use case of AirOne (2)
47
Provide a feature of DCIM that provided former system

48. © DMM.com
Data-center information management
Use case of AirOne (2)
48
Basic mechanism to access Entry’s data.
Data Source
ServerRack
Rack101 Rack102 ... Floor101 Floor102 ...
View
Floor
URL
URL URL URL

49. © DMM.com
Data-center information management
Use case of AirOne (2)
49
Data Source
ServerRack
Rack101 Rack102 ... Floor101 Floor102 ...
View
Floor
URL
URL URL URL
Basic mechanism to access Entry’s data.

50. © DMM.com
Data-center information management
Use case of AirOne (2)
50
Basic mechanism to access Entry’s data.
Data Source
ServerRack
Rack101 Rack102 ... Floor101 Floor102 ...
View
Floor
URL
URL URL URL

51. © DMM.com
Data-center information management
Use case of AirOne (2)
51
Advanced (customizable) mechanism to access data from across multiple Entries.
Data Source
ServerRack
Rack101 Rack102 ... Floor101 Floor102 ...
View
Floor
URL
URL URL URL

52. © DMM.com
Data-center information management
Use case of AirOne (2)
52
Advanced (customizable) mechanism to access data from across multiple Entries.
Data Source
ServerRack
Rack101 Rack102 ... Floor101 Floor102 ...
View
Floor
URL
URL URL URL

53. © DMM.com
Data-center information management
Use case of AirOne (2)
53
Advanced (customizable) mechanism to access data from across multiple Entries.
Data Source
ServerRack
Rack101 Rack102 ... Floor101 Floor102 ...
View
Floor
URL
URL URL URL

54. © DMM.com
Data-center information management
Use case of AirOne (2)
54
Advanced (customizable) mechanism to access data from across multiple Entries.
Data Source
ServerRack
Rack101 Rack102 ... Floor101 Floor102 ...
View
Floor
URL
URL URL URL

55. © DMM.com
Reference Information about SSoT
55

56. © DMM.com
Robotron [by Meta (Facebook)]
Reference Information related with SSoT
56
Network automation configuration and monitoring system
[*]
[*] “Robotron: Top-down network management at scale” , AUGUST 24, 2016, Engineering at Meta.
- It manages whole information about network design, configuration,
deployment and monitoring at FBNet
- FBNet serves as the single source of truth for network component state

57. © DMM.com
MALT [by Google]
Reference Information related with SSoT
57
[*]
[*] Jeffrey C. Mogul, Drago Goricanec, Martin Pool, Anees Shaikh, Douglas Turk, and Bikash Koley, Google; Xiaoxue
Zhao, Alibaba Group Inc., “Experiences with Modeling Network Topologies at Multiple Levels of Abstraction”, NSDI’20
- It manages whole network topology, capacity panning, access control,
routing policy and so on.
- It generate SDN controllers and device configurations through automated
high-level designer based on MALT information.
Network automation management system

58. © DMM.com
Hardships of SSoT
- “There is no Silver Bullet”
— The Mythical Man-Month, Frederick Phillips Brooks, Jr (1975)
58

59. © DMM.com
Mediation between users
59
Changing model requires consensus between other users who use same model
Operation Team
Hey, the system is wrong!
I can’t update our VM Entries!
Hardships of SSoT (1)
Virtual Machine
・IP address [reference]
・Hostname [string]
・Owner [reference]
... Status
・Status [reference]
IP address
・Network [reference]
...
Owner

60. © DMM.com
Mediation between users
60
Changing model requires consensus between other users who use same model
Operation Team
Hey, the system is wrong!
I can’t update our VM Entries!
Hardships of SSoT (1)
Virtual Machine
・IP address [reference]
・Hostname [string]
・Owner [reference]
... Status
・Status [reference]
IP address
・Network [reference]
...
Owner
It’s hard to know which VM is
actually used, or not.
So let’s change it!
Accounting Team
(REQ)

61. © DMM.com
Relentless restructure
61
Information keeps growing and models are always becoming more complex
Network Appliance
Hardships of SSoT (2)
ServerRack
...
Physical Port
...
ServerRack [reference]
Ports [reference]
...
Vendor [string]
( Entires ) ( Model structures )
Network Appliance
sw1023
sw1025
sw1030
...

62. © DMM.com
Relentless restructure
62
Information keeps growing and models are always becoming more complex
Network Appliance
Hardships of SSoT (2)
sw1023
sw1025
sw1030
...
LB0245
LB0246
FW0024
...
...

63. © DMM.com
Relentless restructure
63
Information keeps growing and models are always becoming more complex
Network Appliance
Hardships of SSoT (2)
sw1023
sw1025
sw1030
...
LB0245
LB0246
FW0024
...
...
It’s needed to save ACL policy in here
It’s needed to save VirtualServer information in here

64. © DMM.com
Relentless restructure
64
Information keeps growing and models are always becoming more complex
Network Appliance
Hardships of SSoT (2)
sw1023
sw1025
sw1030
...
LB0245
LB0246
FW0024
...
...
It’s needed to save ACL policy in here
It’s needed to save VirtualServer information in here
Network Appliance
ServerRack [reference]
Ports [reference]
Vendor [string]
( Model structures )
VirtualServer [reference of array]
ACL policy [reference of array]
...

65. © DMM.com
Relentless restructure
65
Information keeps growing and models are always becoming more complex
L2 / L3 switch
Hardships of SSoT (2)
ServerRack [reference]
Ports [reference]
...
Vendor [string]
Load balancer
ServerRack [reference]
Ports [reference]
...
Vendor [string]
VirtualServer [reference of array]
FireWall
ServerRack [reference]
Ports [reference]
...
Vendor [string]
ACL policy [reference of array]
( Model structures )

66. © DMM.com
Conclusion
66

67. © DMM.com
- I don’t know
(I guess there is no answer of this question)
What is the best way for information management?
Conclusion
67
- But, I believe SSoT is the best (for infrastructure information management)
to reduce operation and management cost of infrastructure(application)
information management.

68. © DMM.com
Thank you
68

69. © DMM.com
Appendix
69

70. © DMM.com
Q. How distinguish information to save at the SSoT, or not.
Appendix.1
70
A. It’s good to manage Information that is associated with
Object at the SSoT.
- Good example: Server, Virtual Machine, Secrets (deployment parameters)
- BAD example: SourceCode, Schedule, Documents (knowledge)

71. © DMM.com
Appendix.2
71
This Information has following features
- Physical (e.g. Rack position, Wiring, Power supply capacity)
- Administrative (e.g. Depreciation, Lease expiration)
- Logical (e.g. IP address, VLAN-ID)
- Operational (e.g. Hosting users, Status)
What Information for Infrastructure Operation means?

72. © DMM.com
Appendix.3
72
It means to keep following features
- Being accessible to stored information
- Being consistent with any other information
- Proper permissions for each Users / Groups
What state of well managed is?

73. © DMM.com
IP address management & Vulnerability assessment
Use case of AirOne (3)
73
AirOne provides IPAM feature and has LB, Network (etc) configurations.
IPv4 Address
...
IPv6 Address
...
LB Virtual Server
...
LB Service Group
...
Physical Server
...
Virtual Server
...
Physical Port
...
Network Appliance
...
Server Rack
...
Reference relationship

74. © DMM.com
IP address management & Vulnerability assessment
Use case of AirOne (3)
74
AirOne provides IPAM feature and has LB, Network (etc) configurations.
IPv4 Address
...
IPv6 Address
...
LB Virtual Server
...
LB Service Group
...
Physical Server
...
Virtual Server
...
Physical Port
...
Network Appliance
...
Server Rack
...
Reference relationship

75. © DMM.com
IP address management & Vulnerability assessment
Use case of AirOne (3)
75
AirOne provides IPAM feature and has LB, Network (etc) configurations.
IPv4 Address
...
IPv6 Address
...
LB Virtual Server
...
LB Service Group
...
Physical Server
...
Virtual Server
...
Physical Port
...
Network Appliance
...
Server Rack
...
Reference relationship