Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
從零開始的 Configuration Management
Levi Chen @ DevOpsDays Taipei 2019
About Me
陳俊毅 (Levi Chen)
Engineer in Common Service & Infra Team @ 91APP
Contact me
● FB: https://fb.me/chenjiunyi
● Blog:...
Responsibilities @ 91APP
● Develop 3 common services
○ Product, Notification and Rate Limiter
● Help RDs to use cloud tech...
4
5
https://paulhammant.com/2014/08/27/provisioning-deployment-and-app-config-cycles/
Agenda
● Why configuration management?
● How to manage configuration?
● How to go live?
6
Agenda
● Why configuration management?
● How to manage configuration?
● How to go live?
7
台灣最大 & 成長最快
品牌新零售解決方案公司
- 2013 年成立
- 總部在台北,馬來西亞/香港分公司
- 員工人數 400 人
8
9
品牌客戶超過 10,000 家
10
Challenge #1
Deploying to Mutiple Environments
11
TW
HK
MY
Expand to New Markets
2013 TW
2018 MY
2019 HK
12
Load Test
https://www.monitis.com/blog/load-test-and-be-sure-your-website-is-ready/13
RD + PD Team: 150 people, 9 function teams
Multiple Test Environments
14
Team 1
15
Team 2
Team 3
Team 4
Team 5
SIT Staging Production
Team 1
16
Team 2
Team 3
Team 4
Team 5
SIT Staging Production
Env 1
Env 2
Env 3
Env 4
Env 5
The Complexity of Deployment
=
Number of Environments
x
Number of Services
x
Number of Instances
17
Environment 1
Service CD Server
1. Build
2. Deploy
18
Environment 1
Service CD Server
1. Build
2. Deploy
Environment 2
Service CD Server
1. Build
2. Deploy
19
Challenge #2
Multiple Tenancies
20
Different configurations between tenancies
https://sodelicious.recipes/in-the-kitchen/christmas-food-around-the-world/
21
The Complexity of Deployment
=
Number of Environments
x
Number of Services
x
Number of Instances
x
Number of Configuration...
If we have 10 environments, 40 services, 2 instances, 20
configuration…
10 environments x 40 services x 2 instances x 20 c...
Challenge #3
Changing Configurations in Realtime
24
Change configurations != Change systems
http://ellenhorn.com/blog/the-ten-reasons-not-to-change/25
Missing configuration after deployment
https://www.express.co.uk/life-style/health/431422/Busy-Then-forget-it-Hectic-lifes...
Challenge #4
Too Many Domain Names
27
28
Instance
#1
Instance
#2
Instance
#3
LBClient
Server-Side Service Discovery
https://www.nginx.com/blog/service-discovery...
Environment 1
A
B
C
D
E
a.environment1.com
b.environment1.com
c.environment1.com
d.environment1.com
e.environment1.com
Env...
40 services = 40 domains
30
If we have 10 environments, 40 services…
10 environments x 40 services = 400 domain names
31
● Deploying to multiple environments
● Multiple tenancies
● Changing configuration in realtime
● Too many domain names
Rec...
Agenda
● Why configuration management?
● How to manage configuration?
● How to go live?
33
Step #1
Make Service Artifacts Stateless
34
Environment 1
Service CD Server
1. Build
2. Deploy
Environment 2
Service CD Server
1. Build
2. Deploy
35
Responsibility of CD Server
● Build Artifacts for a specific environment
● Deployment
○ Permission
○ Network Topology
○ De...
number of artifacts
=
number of environments
=
number of CD server
37
What is your performance?
https://www.humanresourcesonline.net/how-to-effectively-set-and-communicate-performance-expectat...
How does open source deal with it?
39
We should ….
Single Codebase Multiple Deployment
Deployment = Artifact + Configurations
40
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
CMService
41
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
CM
1. Build Config 1. Build Code
Service
42
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
2. Get Artifact
2. Get Artifact
CM
1. Build Config 1. Build C...
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
3. Get Config
2. Get Artifact
2. Get Artifact
CM
1. Build Con...
Everyone could use artifacts to build services everywhere,
by different configurations
https://www.sc.qa/en/news/2022-fifa...
Step 2. Configuration Hierarchy
46
Global Region Market Shop
Default Settings
Schema
Comments
Regional Settings
ap-northeast-1
ap-southeast-1
tw
hk
my
Shop I...
48
49
50
51
global
region
market
shop
52
Global Region Market Shop
Default Settings
Schema
Comments
Regional Settings
ap-northeast-1
ap-southeast-1
tw
hk
my
Shop I...
54
$ docker run 
-v ${PWD}:/build 
$DOCKER_IMG 
cfgctl build_local_file 
--config ./config 
--artifacts ./artifacts 
--module...
shop 8888 market
56
57
$ docker run 
-v ${PWD}:/build 
$DOCKER_IMG 
cfgctl build_local_file 
--config ./config 
--artifacts ./artifacts 
--mod...
58
Benefits of Configuration Hierarchy
● Tenancies level configuration
● No copy paste (remove redundant configuration)
59
Step #3
Configuration Build Tools: Cfgctl
60
● Command line interface for building configuration hierarchy
● Test for Configuration
● JSON -> Key-Value pairs
● Build a...
Play with cfgctl: For developer in local environment
Local
Files
Configuration
Local
Consul
Cfgctl build for
local files
C...
Play with cfgctl: For OOO Environment
S3
Configuration
Consul
Cfgctl build for S3
Cfgctl build for
consul
ServiceASDK
Serv...
Step #4
Configuration SDK
64
Configuration SDK
● Languages Supported: C#, Node.js
● Features
○ Interface
○ Supported multiple configuration storages
○ ...
Service SDK
Consul
S3
Local Files
66
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
3. Get Config
2. Get Artifact
2. Get Artifact
CM
1. Build Con...
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
3. Get Config
2. Get Artifact
2. Get Artifact
CM
1. Build Con...
Step #5
Service Discovery
69
70
Instance
#1
Instance
#2
Instance
#3
LBClient
Server-Side Service Discovery
https://www.nginx.com/blog/service-discovery...
71
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
72
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Se...
73
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Se...
74
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Se...
75
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Se...
76
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Se...
77
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Se...
78
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Se...
79
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to c...
80
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to c...
81
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to c...
82
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to c...
83
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to c...
84
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to c...
85
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to c...
Environment 1
A
B
C D
E
cm.environment1.com
cm.environment2.com
CM
Environment 2
A
B
C D
ECM
86
Multiple tenancies infrastructure in client-side service discovery
https://www.nginx.com/blog/service-discovery-in-a-micro...
VIP
Normal
Cleint
VIP.client.service.com
Normal.client.service.com
Multiple tenancies infrastructure in server-side servic...
Step #6
Mininum Configuration
(How to Discover Service Discovery?)
89
90
https://www.slideshare.net/AmazonWebServices/leveraging-elastic-webscale-computing-with-aws
91
Put bootstring configuration on AWS EC2 tags
Convert to Environment Variables
It Could Cover Container, VM base Application
92
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
CM
0. Build Config
By Cfgctl
0. Build Code
Service
S
D
K
S
D
...
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
CM
0. Build Config
By Cfgctl
0. Build Code
Service
S
D
K
S
D
...
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
2. Get Artifact
2. Get Artifact
CM
0. Build Config
By Cfgctl
...
Environment 1
AM
CI Server
CMService
Environment 2
CI Server
3. Get Config
2. Get Artifact
2. Get Artifact
CM
0. Build Con...
Service Provisioning
● Build code and configuration
● Get Metadata
● Get Artifacts
● Get Confugration
● Run Service
97
Step #7
Types of Configuration
98
3 Levels of Configuration
● Realtime
○ Configuration SDK + Configuration Storages (Consul
or S3)
● Service Discovery
○ Con...
Agenda
● Why configuration management?
● How to manage configuration?
● How to go live?
100
Rule #1.
Consider what you can contribute to teams and minimize the
impact on them
101
102
https://www.fabiaoqing.com/search/search/keyword/我委屈但是我不哭
● Arch Team
○ Design
○ Configuration SDK
■ Easy to change configuration storages
■ Cache
■ Hotreload
■ Service Registratio...
Consider what you can contribute to teams and
minimize the impact on them
104
● Infra Team & Common Service Team
○ PoC / P...
Rule #2.
Prove it works
105
106
Rule #3.
Use it in small project or your services
107
Rule #4.
Prove it is stable
108
Service SDK
Consul
S3
Local Files
109
110
Performance across the S3 Storage Classes
https://aws.amazon.com/s3/storage-classes/?nc1=h_ls
Server
(Follower)
Server
(Leader)
Server
(Follower)
GOSSIP
Replication
Leader
Forwarding
Replication
Client Client Client
...
Server
(Follower)
Server
(Leader)
Server
(Follower)
GOSSIP
Replication
Leader
Forwarding
Replication
Client Client Client
...
Server
(Follower)
Server
(Follower)
GOSSIP
Leader
Forwarding
Replication
Client Client Client
RPC
GOSSIP
RPCGOSSIP
Consul ...
Consul Cluster Availability: Multiple AZ deployment
https://aws.amazon.com/quickstart/architecture/consul/114
Performance Tuning
https://www.consul.io/docs/install/performance.html115
116
117
Design for Operation
http://www.fgsbmc.org.tw/news_latestnews_c.aspx?News_Id=201801084
Questions
119
Thanks
Upcoming SlideShare
Loading in …5
×

of

20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 1 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 2 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 3 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 4 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 5 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 6 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 7 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 8 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 9 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 10 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 11 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 12 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 13 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 14 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 15 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 16 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 17 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 18 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 19 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 20 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 21 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 22 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 23 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 24 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 25 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 26 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 27 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 28 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 29 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 30 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 31 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 32 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 33 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 34 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 35 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 36 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 37 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 38 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 39 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 40 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 41 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 42 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 43 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 44 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 45 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 46 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 47 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 48 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 49 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 50 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 51 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 52 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 53 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 54 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 55 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 56 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 57 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 58 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 59 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 60 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 61 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 62 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 63 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 64 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 65 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 66 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 67 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 68 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 69 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 70 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 71 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 72 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 73 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 74 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 75 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 76 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 77 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 78 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 79 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 80 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 81 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 82 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 83 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 84 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 85 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 86 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 87 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 88 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 89 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 90 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 91 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 92 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 93 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 94 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 95 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 96 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 97 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 98 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 99 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 100 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 101 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 102 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 103 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 104 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 105 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 106 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 107 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 108 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 109 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 110 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 111 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 112 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 113 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 114 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 115 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 116 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 117 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 118 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 119 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management Slide 120
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

2 Likes

Share

Download to read offline

20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management

Download to read offline

隨著公司的業務發展,我們開始注意到要快速的部署一個新的環境、市場越來越沒有效率,而 CM(Configuration Management)在我們解決這些問題中扮演著重要角色。本演講將專注於解決下列問題:

1. 為什麼我們需要 CM?
2. 如何管理不同種類的 Configuration: Realtime, Service Discovery and Provision?
3. 如何降低團隊的導入 CM 複雜度?

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management

  1. 1. 從零開始的 Configuration Management Levi Chen @ DevOpsDays Taipei 2019
  2. 2. About Me 陳俊毅 (Levi Chen) Engineer in Common Service & Infra Team @ 91APP Contact me ● FB: https://fb.me/chenjiunyi ● Blog: https://blog.levichen.tw 2
  3. 3. Responsibilities @ 91APP ● Develop 3 common services ○ Product, Notification and Rate Limiter ● Help RDs to use cloud technologies ○ Cloud Native, Serverless ● Help architecture team to implement new policies About Me 3
  4. 4. 4
  5. 5. 5 https://paulhammant.com/2014/08/27/provisioning-deployment-and-app-config-cycles/
  6. 6. Agenda ● Why configuration management? ● How to manage configuration? ● How to go live? 6
  7. 7. Agenda ● Why configuration management? ● How to manage configuration? ● How to go live? 7
  8. 8. 台灣最大 & 成長最快 品牌新零售解決方案公司 - 2013 年成立 - 總部在台北,馬來西亞/香港分公司 - 員工人數 400 人 8
  9. 9. 9
  10. 10. 品牌客戶超過 10,000 家 10
  11. 11. Challenge #1 Deploying to Mutiple Environments 11
  12. 12. TW HK MY Expand to New Markets 2013 TW 2018 MY 2019 HK 12
  13. 13. Load Test https://www.monitis.com/blog/load-test-and-be-sure-your-website-is-ready/13
  14. 14. RD + PD Team: 150 people, 9 function teams Multiple Test Environments 14
  15. 15. Team 1 15 Team 2 Team 3 Team 4 Team 5 SIT Staging Production
  16. 16. Team 1 16 Team 2 Team 3 Team 4 Team 5 SIT Staging Production Env 1 Env 2 Env 3 Env 4 Env 5
  17. 17. The Complexity of Deployment = Number of Environments x Number of Services x Number of Instances 17
  18. 18. Environment 1 Service CD Server 1. Build 2. Deploy 18
  19. 19. Environment 1 Service CD Server 1. Build 2. Deploy Environment 2 Service CD Server 1. Build 2. Deploy 19
  20. 20. Challenge #2 Multiple Tenancies 20
  21. 21. Different configurations between tenancies https://sodelicious.recipes/in-the-kitchen/christmas-food-around-the-world/ 21
  22. 22. The Complexity of Deployment = Number of Environments x Number of Services x Number of Instances x Number of Configurations22
  23. 23. If we have 10 environments, 40 services, 2 instances, 20 configuration… 10 environments x 40 services x 2 instances x 20 configuration = 16,000 configurations 23
  24. 24. Challenge #3 Changing Configurations in Realtime 24
  25. 25. Change configurations != Change systems http://ellenhorn.com/blog/the-ten-reasons-not-to-change/25
  26. 26. Missing configuration after deployment https://www.express.co.uk/life-style/health/431422/Busy-Then-forget-it-Hectic-lifestyle-affects-our-memory26
  27. 27. Challenge #4 Too Many Domain Names 27
  28. 28. 28 Instance #1 Instance #2 Instance #3 LBClient Server-Side Service Discovery https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/
  29. 29. Environment 1 A B C D E a.environment1.com b.environment1.com c.environment1.com d.environment1.com e.environment1.com Environment 2 A B C D E a.environment2.com b.environment2.com c.environment2.com d.environment2.com e.environment2.com29
  30. 30. 40 services = 40 domains 30
  31. 31. If we have 10 environments, 40 services… 10 environments x 40 services = 400 domain names 31
  32. 32. ● Deploying to multiple environments ● Multiple tenancies ● Changing configuration in realtime ● Too many domain names Recap: Why configuration management? 32
  33. 33. Agenda ● Why configuration management? ● How to manage configuration? ● How to go live? 33
  34. 34. Step #1 Make Service Artifacts Stateless 34
  35. 35. Environment 1 Service CD Server 1. Build 2. Deploy Environment 2 Service CD Server 1. Build 2. Deploy 35
  36. 36. Responsibility of CD Server ● Build Artifacts for a specific environment ● Deployment ○ Permission ○ Network Topology ○ Deployment Strategy 36
  37. 37. number of artifacts = number of environments = number of CD server 37
  38. 38. What is your performance? https://www.humanresourcesonline.net/how-to-effectively-set-and-communicate-performance-expectations/ 38
  39. 39. How does open source deal with it? 39
  40. 40. We should …. Single Codebase Multiple Deployment Deployment = Artifact + Configurations 40
  41. 41. Environment 1 AM CI Server CMService Environment 2 CI Server CMService 41
  42. 42. Environment 1 AM CI Server CMService Environment 2 CI Server CM 1. Build Config 1. Build Code Service 42
  43. 43. Environment 1 AM CI Server CMService Environment 2 CI Server 2. Get Artifact 2. Get Artifact CM 1. Build Config 1. Build Code Service 43
  44. 44. Environment 1 AM CI Server CMService Environment 2 CI Server 3. Get Config 2. Get Artifact 2. Get Artifact CM 1. Build Config 1. Build Code 3. Get ConfigService 44
  45. 45. Everyone could use artifacts to build services everywhere, by different configurations https://www.sc.qa/en/news/2022-fifa-world-cup-stadium-workers-given-5m-repayment45
  46. 46. Step 2. Configuration Hierarchy 46
  47. 47. Global Region Market Shop Default Settings Schema Comments Regional Settings ap-northeast-1 ap-southeast-1 tw hk my Shop Id: 9527 Shop Id: 9528 /{module}/global.json /{module}/{region}/global.json /{module}/{region}/{market}.json /{module}/{region}/{market}-{shop}.json Configuration Hierarchy (Source) 47
  48. 48. 48
  49. 49. 49
  50. 50. 50
  51. 51. 51
  52. 52. global region market shop 52
  53. 53. Global Region Market Shop Default Settings Schema Comments Regional Settings ap-northeast-1 ap-southeast-1 tw hk my Shop Id: 9527 Shop Id: 9528 Configuration Hierarchy (Build) /{module}/index.json /{module}/{market}.json /{module}/{market}-{shop}.json 53
  54. 54. 54
  55. 55. $ docker run -v ${PWD}:/build $DOCKER_IMG cfgctl build_local_file --config ./config --artifacts ./artifacts --module MsaDemo --datacenter dc-01 --tag 1.1.0 55
  56. 56. shop 8888 market 56
  57. 57. 57 $ docker run -v ${PWD}:/build $DOCKER_IMG cfgctl build_local_file --config ./config --artifacts ./artifacts --module MsaDemo --datacenter dc-01 --tag 1.1.0 --map true
  58. 58. 58
  59. 59. Benefits of Configuration Hierarchy ● Tenancies level configuration ● No copy paste (remove redundant configuration) 59
  60. 60. Step #3 Configuration Build Tools: Cfgctl 60
  61. 61. ● Command line interface for building configuration hierarchy ● Test for Configuration ● JSON -> Key-Value pairs ● Build and deploy configuration to 3 types of storages ○ Local files ○ Consul ○ S3 Cfgctl 61
  62. 62. Play with cfgctl: For developer in local environment Local Files Configuration Local Consul Cfgctl build for local files Cfgctl build for consul CodeSDK 62
  63. 63. Play with cfgctl: For OOO Environment S3 Configuration Consul Cfgctl build for S3 Cfgctl build for consul ServiceASDK ServiceBSDK 63
  64. 64. Step #4 Configuration SDK 64
  65. 65. Configuration SDK ● Languages Supported: C#, Node.js ● Features ○ Interface ○ Supported multiple configuration storages ○ Caching ○ Hot reload 65
  66. 66. Service SDK Consul S3 Local Files 66
  67. 67. Environment 1 AM CI Server CMService Environment 2 CI Server 3. Get Config 2. Get Artifact 2. Get Artifact CM 1. Build Config 1. Build Code 3. Get ConfigService 67
  68. 68. Environment 1 AM CI Server CMService Environment 2 CI Server 3. Get Config 2. Get Artifact 2. Get Artifact CM 1. Build Config By Cfgctl 1. Build Code 3. Get ConfigService S D K S D K 68
  69. 69. Step #5 Service Discovery 69
  70. 70. 70 Instance #1 Instance #2 Instance #3 LBClient Server-Side Service Discovery https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/
  71. 71. 71 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2
  72. 72. 72 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register)
  73. 73. 73 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat)
  74. 74. 74 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B
  75. 75. 75 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B Service B has a Instance #1
  76. 76. 76 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B Service B has a Instance #1 Call Service B Instance #1
  77. 77. 77 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B Service B has a Instance #1 Call Service B Instance #1 Hi, I am Service B (register)
  78. 78. 78 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B Service B has a Instance #1 Call Service B Instance #1 Hi, I am Service B (register) Hi, I still alive (send heartbeat)
  79. 79. 79 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B
  80. 80. 80 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2
  81. 81. 81 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%)
  82. 82. 82 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%) I wanna to leave (deregister)
  83. 83. 83 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%) I wanna to leave (deregister) I want to call Service B
  84. 84. 84 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%) I wanna to leave (deregister) I want to call Service B Service B has a Instance #2
  85. 85. 85 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%) I wanna to leave (deregister) I want to call Service B Service B has a Instance #2 Call Service B Instance #2
  86. 86. Environment 1 A B C D E cm.environment1.com cm.environment2.com CM Environment 2 A B C D ECM 86
  87. 87. Multiple tenancies infrastructure in client-side service discovery https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/ VIP VIP Normal 87
  88. 88. VIP Normal Cleint VIP.client.service.com Normal.client.service.com Multiple tenancies infrastructure in server-side service discovery 88
  89. 89. Step #6 Mininum Configuration (How to Discover Service Discovery?) 89
  90. 90. 90 https://www.slideshare.net/AmazonWebServices/leveraging-elastic-webscale-computing-with-aws
  91. 91. 91 Put bootstring configuration on AWS EC2 tags
  92. 92. Convert to Environment Variables It Could Cover Container, VM base Application 92
  93. 93. Environment 1 AM CI Server CMService Environment 2 CI Server CM 0. Build Config By Cfgctl 0. Build Code Service S D K S D K 93
  94. 94. Environment 1 AM CI Server CMService Environment 2 CI Server CM 0. Build Config By Cfgctl 0. Build Code Service S D K S D K 94 1. Get Metadata 1. Get Metadata Service Name, Version, AM, CM, etc.. Service Name, Version, AM, CM, etc..
  95. 95. Environment 1 AM CI Server CMService Environment 2 CI Server 2. Get Artifact 2. Get Artifact CM 0. Build Config By Cfgctl 0. Build Code Service S D K S D K 95 1. Get Metadata 1. Get Metadata Service Name, Version, AM, CM, etc.. Service Name, Version, AM, CM, etc..
  96. 96. Environment 1 AM CI Server CMService Environment 2 CI Server 3. Get Config 2. Get Artifact 2. Get Artifact CM 0. Build Config By Cfgctl 0. Build Code 3. Get ConfigService S D K S D K 96 1. Get Metadata 1. Get Metadata Service Name, Version, AM, CM, etc.. Service Name, Version, AM, CM, etc..
  97. 97. Service Provisioning ● Build code and configuration ● Get Metadata ● Get Artifacts ● Get Confugration ● Run Service 97
  98. 98. Step #7 Types of Configuration 98
  99. 99. 3 Levels of Configuration ● Realtime ○ Configuration SDK + Configuration Storages (Consul or S3) ● Service Discovery ○ Configuration SDK + Advance Configuration Services (Consul) ● Provision ○ Configuration Storages (Consul or S3) 99
  100. 100. Agenda ● Why configuration management? ● How to manage configuration? ● How to go live? 100
  101. 101. Rule #1. Consider what you can contribute to teams and minimize the impact on them 101
  102. 102. 102 https://www.fabiaoqing.com/search/search/keyword/我委屈但是我不哭
  103. 103. ● Arch Team ○ Design ○ Configuration SDK ■ Easy to change configuration storages ■ Cache ■ Hotreload ■ Service Registration Consider what you can contribute to teams and minimize the impact on them 103
  104. 104. Consider what you can contribute to teams and minimize the impact on them 104 ● Infra Team & Common Service Team ○ PoC / Provision / Maintain Configuration Storage ○ Consider network topology / security ○ Build tools: Cfgctl ■ Resolve level configuration issue ○ Use it in our own services first
  105. 105. Rule #2. Prove it works 105
  106. 106. 106
  107. 107. Rule #3. Use it in small project or your services 107
  108. 108. Rule #4. Prove it is stable 108
  109. 109. Service SDK Consul S3 Local Files 109
  110. 110. 110 Performance across the S3 Storage Classes https://aws.amazon.com/s3/storage-classes/?nc1=h_ls
  111. 111. Server (Follower) Server (Leader) Server (Follower) GOSSIP Replication Leader Forwarding Replication Client Client Client RPC Leader Forwarding GOSSIP RPCGOSSIP Consul Cluster https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide111
  112. 112. Server (Follower) Server (Leader) Server (Follower) GOSSIP Replication Leader Forwarding Replication Client Client Client RPC Leader Forwarding GOSSIP RPCGOSSIP Consul Cluster Availability 112
  113. 113. Server (Follower) Server (Follower) GOSSIP Leader Forwarding Replication Client Client Client RPC GOSSIP RPCGOSSIP Consul Cluster Availability, leader election 113
  114. 114. Consul Cluster Availability: Multiple AZ deployment https://aws.amazon.com/quickstart/architecture/consul/114
  115. 115. Performance Tuning https://www.consul.io/docs/install/performance.html115
  116. 116. 116
  117. 117. 117 Design for Operation http://www.fgsbmc.org.tw/news_latestnews_c.aspx?News_Id=201801084
  118. 118. Questions
  119. 119. 119
  120. 120. Thanks
  • GregChou

    Dec. 1, 2021
  • hakeclover

    Oct. 19, 2019

隨著公司的業務發展,我們開始注意到要快速的部署一個新的環境、市場越來越沒有效率,而 CM(Configuration Management)在我們解決這些問題中扮演著重要角色。本演講將專注於解決下列問題: 1. 為什麼我們需要 CM? 2. 如何管理不同種類的 Configuration: Realtime, Service Discovery and Provision? 3. 如何降低團隊的導入 CM 複雜度?

Views

Total views

627

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

14

Shares

0

Comments

0

Likes

2

×