TBD

2. Currently on the Cloud

3. 1 Our situation in 2014
2 How we improved
3 Sweet things
4 More sweet things
5 Future

4. 1 Our situation in 2014
2 How we improved
3 Sweet things
4 More sweet things
5 Future

5. Our situation in 2014
1st gen
“HSP” – (2014) from cybercafe/漫画木さ/PC방 platform
2nd gen
ECDH-based key exchange
Platform
Billing/AAA/Monitoring, etc. & Game servers
“LGC” – (cloud release)
3rd gen: “Trident” – (current)

6. Globalization
Issues abroad
Loading…
Loading…
Loading…
Loading…
Fail!

7. Process
Dev/QA/Sandbox/REAL…
Get VM
Get L4 binding
ACLs/storage, etc.
From hours to days…

8. Thus,
the Game Cloud project began…

9. 1 Our situation in 2014
2 How we improved
3 Sweet things
4 More sweet things
5 Future

10. Global
In our case
GSLB (Global Server Load Balancing)
HAProxy instead of hardware L4
Multi-team effort
client, server, cloud
We get
more flexibility
less latency

11. LINE Global POP
New York
Tokyo
Seoul
HK
Singapole
Beijing
Frankfurt
a
a

12. Network layer control
High latency
Fit for cloud
Global
Testing in Thailand
TH  KR
GSLB
TH  SG  KR
HAProxy
500
1000
1500
2 12 206 16

13. Process
Dev
Ops
Government
The structure of your organization affects the structure of your software.
And vice-versa!
DevOps
Small StartupCHOOSE

14. Process
Progressive / Easy, simple
Requirements for our new platform :
We have many third parties and technology stacks involved...
Etc.

15. Process
For ourselves
simple reliable future proof

16. Process
Why not…
Docker Swarm
or CoreOS
Mesos Kubernetes

18. Process
KEEP THINGS SIMPLE AND RELIABLE!
For distributed systems, minimize coordination
A good paper: https://blog.acolyer.org/2016/01/19/dcft/
• Polling
• 1way dataflow
• Idempotency
• Commutativity
• limited trust

19. LGC Story
Games planned for release were suddenly canceled
but we needed to show results!
Strong “sales” efforts to release other games
on the LGC platform

21. Putting Out Fires
The release was a success,followed by a
quick scaling-up, and then our first fires…
TECHNICAL
Riak fire : the system works with Riak down
OE fire: the system works with OE down
Hardware and conf fires (TDI! Soon to come!)
Full container reboot
improved our design through limited trust

23. 1 Our situation in 2014
2 How we improved
3 Sweet things
4 More sweet things
5 Future

25. Domain HAP
Launch
service
Configure/
load balance
Expose
ports
Bind
URL
In one click!

26. Monitoring

27. 1 Our situation in 2014
2 How we improved
3 Sweet things
4 More sweet things
5 Future

29. Gearbox
Auto Scaling System

30. High
availability
Low cost
Why Do We Need It?

32. How Does It Work?

33. How Did We Build it?
Data Collector
Monitoring API
Predicator
Metrics
Raw Metrics
Scaler
States
Game Cloud API
Excute Scaling
Gearbox

34. Challenges
Complex query
Plenty of metrics
records
millions of records
per day
Scalability of the
auto-scaling
system itself

35. Solutions
Data Collector
Monitoring API
Predicator
Metrics
Metrics
Scaler
States
Game Cloud API
Excute
Scaling
Gearbox
Module
Storage
ElasticSearchAdmin Site
Strategy
Scaler Log
Metrics
1.Strategy
2.Metrics
{ }
Predicator
Log
1.Strategy
2.Scaler Log

36. Knife
Admin Site

37. Deploying a
New Service

38. Upgrade

39. Configuring the
Auto-Scaling
Policy

40. Back to Jojo
What’s coming next!

41. 1 Our situation in 2014
2 How we improved
3 Sweet things
4 More sweet things
5 Future

42. Future
QUIC
SDN
ACL, IP by container, VLAN etc.
Cloud storage
TDI
Distributed GC – link paper
DCTCP
Image GC

43. Future
UX
Helpers/presets
Speed
Doc/tests/guides…
Reliability

44. QUIC
Quick
UDP
Internet
Connections

45. Cloud Storage

46. SDN- Software-Defined Networking
Container
specific IP
ACLs VLANs

47. TDI- Test-Driven Infrastructure
hardware OS
configuration
images
backup/
restore
Automated testing for
Etc.
firmware,
version, etc.

48. Distributed GC
Max
Avg.GC pause
http://arxiv.org/pdf/1504.02578.pdf
Median
Std.Dev.
Mean
7.847
0.0
2.296
0.579
2.312
GC off
7.743
12.243
2.294
0.582
2.311
Blade
164.206
12.339
2.297
3.395
2.403
GC on

49. Added in Linux 3.18
https://kernelnewbies.org/
Linux_3.18
http://simula.stanford.edu/
~alizade/Site/DCTCP.html
DCTCP- Data Center TCP
- high burst tolerance
- low latency
- high throughput
http://simula.stanford.edu/~alizade/Site/DCTCP.html

50. Distributed GC
Because we generate
tons of
Docker images

51. And more and more…

54. • AP
• Optional CP
• Index/search
• CRDT
• Multiple backends
• User ACL support
RIAK/Choose a Safe and Simple Friend
Make a deliberate choice of consistency model
SQL
NoSQL
But actually…
With the authorization of Kingsbury Kyle (Aphyr)

55. DataScript /
Maintain
Queries