2021/11/21 JAWS PANKRATION 2021で発表した時の資料です。 日本語版の資料は以下です。 https://speakerdeck.com/coopsapporo/what-makes-me-to-migrate-entire-vpc

1. What makes me to migrate entire VPC
JAWS-UG Architecture / JAWS-UG Systems Administrators
Naomi Yamasaki

2. AWS SAMURAI 2015
JAWS-UG Architecture
JAWS-UG System Administrators
Naomi Yamasaki
Infrastructure Engineer
Digital Headquarters
Consumers CO-OP Sapporo
@nao_spon
I ♡
Route53 IAM Organizations

3. Why do I migrate entire VPC?

4. We are going to AWS All in!!
● 190 Systems, 650 Servers at Datacenter
● Migrate with CloudEndure Migration
● Think CloudFirst for new systems

5. I love AWS Organizations
● Control accounts, service and user role
○ Control Tower
○ SCP
○ OU
○ AWS SSO
● Standardize environments and Policy
○ CloudFormation Stack Sets
○ AWS Security Hub
○ AWS Config

6. Account designing
● Basically, 3 accounts for 1 system
○ Separate accounts for development, staging, production
○ Some of them have fewer accounts for some reason
● Network
○ Each VPCs are connected with Transit Gateway
○ AWS and Datacenter is connected with Direct Connect

7. Increasing AWS accounts
Migrated systems accounts: 59
New systems accounts: 67
Other accounts: 46

8. Why do I migrate entire VPC?

9. There are 3 accounts created before 2020
● They created before we started using AWS in earnest
● There was intended to be a small start at the time
● We want to put these accounts in Organizations too.

10. Why we include those accounts in AWS Organizations
Not under the control of Organizations means configure these settings individual
○ Control accounts, service and user role
■ Control Tower
■ SCP
■ OU
■ AWS SSO
○ Standardize environments and Policy
■ CloudFormation Stack Sets
■ AWS Security Hub
■ AWS Config

11. Most effect and wanted things

12. Most effect and wanted things

13. Problems in attaching to Transit Gateway

14. Problems in attaching to Transit Gateway
● Duplicated network segment
● Too much big CIDR blocks for VPC

15. We decided to migrate VPC to another account
● Prefer
○ Separate accounts for each environments
○ Separate frontend and backend system’s resources being
together
○ Less confusing than changing the current environment.
● Not prefer
○ Minimize the VPCs CIDR blocks
○ Change current go live production environment

16. Before

17. After

18. Migration schedule
1st: Staging environment
2nd: Production environment
3rd: Develop environment

19. The most difficult part of the migration process was...
Aurora migration

20. Which method to choose for DB migration?
● Prefer simple and easy way
● Make it easy to create a replication environment
● Binary log replication seemed to be a pain to configure
● DMS is looks like easy way

21. Which method to choose for DB migration?
● Prefer simple and easy way
● Make it easy to create a replication environment
● Binary log replication seemed to be a pain to configure
● DMS is looks like easy way♡

22. Not easy way...

23. Traps in DMS
● Defaults, unique keys, comments were disappear
and the number of int digits had changed
● Replication was failed on Out Of Memory
in nighttime batch.

24. Nighttime batch with DMS to be a nightmare batch
● After the nightmare batch, failed replication is replicated little
by little
● I tried scaled up maximum the DMS replication instance, but
nightmare batch made me nightmare…
● Out Of Memory with dms.r5.24xlarge: 96vCPUs, 768GiB
memory

25. We gave up on using DMS
● 1 week left to the staging environment switchover
● 3 weeks left to production environment switchover
● We don't want to take a different approach to switchover each
staging and production environment
● We had no time to try Binary log replication.

26. I couldn't say “Let’s do it!” anymore...

27. How can we resolve this issue?
● The service is stopped at 2:00 to 4:00 AM due to nighttime
batch,
so we can take down time at that time.
● We have a time to shift for the nighttime batches.
● We decided to sharing DB snapshots and restore from
● It was the best way for us at that time

28. Another issue with AWS KMS key
DB snapshot cannot be shared to another account
when using default aws/rds KMS key to encrypt the database

29. How to resolve it?
● Work at the current account
○ Create Customer managed key on KMS
○ Put permissions to IAM User or Role on Key Policy
○ Create DB snapshot
○ Copy DB snapshot and choose the KMS Key
○ Share the copied DB snapshot to new account
● Work at the new account
○ Copy the shared DB snapshot and choose KMS Key as ‘aws/rds’
○ Create Aurora cluster from copied DB snapshot

30. CloudFormation issues
● There were manually created resources
● Some resources were created with CloudFormation, but some
of them manually modified after all

31. CloudFormation issues
● I want to use the same templates for each staging, production
and development environments
● I had to rewrite the all of the templates for import resources
into CloudFormation stack,
so I re-creating everything to new.

32. CloudFormation vs CDK
● Need low context description
when I try to accurately resource settings in detail.
● I'm more familiar with YAML.

33. ● Using describe commands by AWS CLI
● Mappings like a variable
How did I create CloudFormation templates

34. ● 1 templates for 1 resource
● I created:
○ Staging environment: 44 templates
○ Production environment: 60 templates
● Just copy templates and change the Mappings value
How many CloudFormation templates I created

35. We’ve done it!!
● There was no big trouble.
● Also no big problems in service delivery after switched
● Development environment migration will be postponed until the
major upcoming release have done

36. What I learned...
● Ensure good network design.
● Don't create a VPC with too large CIDR block
● DMS will match if there is not a large amount of processing at
once
● Understand what your database doing
● Choose the best migration method as my DB
● Infrastructure as Code is soooooooooooo important!

37. The best way is sometimes
different from the best practice

38. 2 another accounts are left!!
Again?!