This document discusses lessons learned from the Diablo 3 launch, including mistakes made around quantity estimation, scalability, and customer service. It emphasizes the importance of planning for high traffic, having horizontally scalable architectures, and developing fast and responsive processes for incident management and communication. The document also provides examples of how Trend Micro's web reputation and parental control services can help protect users from online threats.

1. SERVICE PRODUCTION
Walter Liu
2012/06/11
Confidential | Copyright 2012 Trend
1 12/24/2012
Micro Inc.

2. About me
• Architect in Core Tech WRS
• Trend Micro 2007~ Now
• Chief Technical Director in Netgame Dep.
• Softstar Inc. Taiwan 1998~2007
• Expertise:
• Backend service development and operation

3. It’s coming ……

4. The beginning
Confidential | Copyright 2012 Trend 4
12/24/2012 Micro Inc.

5. Assault - Error 12 !!!
Confidential | Copyright 2012 Trend 5
12/24/2012 Micro Inc.

6. The nightmare Error 37

7. Out of stock in Taiwan D3 packages
• D3 packages are out of stock in largest 3 convenient
stores in Taiwan and all game shops.
• Some people shared they finally got packages in Jibei
island.
• Many people shared they pursued after trucks of
convenient stores to get the packages.
Confidential | Copyright 2012 Trend 7
12/24/2012 Micro Inc.

8. Out of Stock of Taiwan Game Card
Confidential | Copyright 2012 Trend 8
12/24/2012 Micro Inc.

9. Blizzard Korean lower priority of
Taiwan IP
Korean IP
Taiwan IP

10. Questions?

11. What’s the most important
things to users?
Confidential | Copyright 2012 Trend 11
12/24/2012 Micro Inc.

12. What mistakes Blizzard
make in this D3 service
production？
Confidential | Copyright 2012 Trend 1
12/24/2012 Micro Inc. 2

13. Why Quantity Estimation?
Cost Effective
What if it is wrong?

14. Scalability & Elasticity

15. Fail in Scalability
Performance
1200
1000
800
600
400
200
0
0 1 2 3 4 5 6 7 8 9 10 11

16. EPIC Fail in Scalability
Performance
1200
1000
800
600
400
200
0
0 1 2 3 4 5 6 7 8 9 10

17. Ideal Horizontal Scalability
Performance
2500
2000
1500
1000
500
0
0 1 2 3 4 5 6 7 8 9 10 11

18. Elasticity - Unpredictable Traffic

19. Elasticity - Cloud Solution

20. Elasticity – Others
• Elastic Application Architecture.
• Several flexible hardware providers.
• Flexible ISPs and pricing.
• ……

21. Customer Service and Social
Communication
• Bz is doing pretty bad.
• FB Event: Closed-beta account in Taiwan.
• Build the image to help their users. Like,
• No explanation about incidents.
Confidential | Copyright 2012 Trend
VS.
2
12/24/2012 Micro Inc. 1

22. Incident happens
- Especially your service goes production.

23. Avengers Assemble !!!

24. Fast and Responsive Organization
Teams
Effective
Awareness
Communication

25. Fast and Responsive Process
• Incident management
• Problem management

26. Some other practices
• Interlock with related teams at beginning/middle.
• Customer service prepares resources for burst incoming
calls/tickets.
• Customer service prepares training for the new service/product/
• Data Center team gives out advices and plan for your project.
• Recruit a Service Manager
• Fail case: something that not belong to any team.
• Have someone responsible for whole service.
• Update/patch/change SOP
• Fail case: Service changed, but your CS don’t know about it. Your
customers are confused when they call your CS.
• Fail case: Service changed and caused some trouble, but your
service manager said he didn’t decide/say it.

27. Not related to these Diablo 3 failures, but
important to any system
• Availability
• Security
• Easy to administrate
• System Health/Statistics Monitoring
• Easy Deployment
• Easy Configure
• Risk Management
Confidential | Copyright 2012 Trend 2
12/24/2012 Micro Inc. 7

28. 工商服務時間
• Web Reputation Service
• Parental Control and Productivity Control
• 像是Hinet色情守門員，防毒軟體的孩童防護鎖
• 或者公司不希望員工上班看色情、賭博等網站
• Web Threat Protection
• 防止惡意網頁如病毒/木馬下載、釣魚網站等等。
• Advanced Persistent Threat
• 鎖定特定目標
• 假冒信件或者其他
• 低調且緩慢
• 客製化惡意元件
• 安裝遠端控制工具
• 傳送情資
Confidential | Copyright 2012 Trend 2
12/24/2012 Micro Inc. 8

29. WRS – Parental Control

30. WRS – Web Threat Protection

31. Funny Diablo 3 sales on Taobao, China
Confidential | Copyright 2012 Trend 3
12/24/2012 Micro Inc. 1

32. Thank You!

33. Risk Management
• Identify Critical Failure
• Develop a feasible plan to stabilize customer’s
satisfaction.
• Workaround.
• Rollback.
Confidential | Copyright 2012 Trend 3
12/24/2012 Micro Inc. 3

34. Quantity Estimation
• Goal: Cost Effective Quantity Estimation
• For estimating
• For wrong estimation
• What if the estimation is not correct?
• Too few
• Too many
Confidential | Copyright 2012 Trend 3
12/24/2012 Micro Inc. 4

35. Scalability & Elasticity
• Scalability
• Is your application horizontal scalable?
• Elasticity
• Speed of commissioning / decommissioning
• Max amount of resource can be brought in
• Granularity of usage accounting
• Develop the plan for high traffic.
Confidential | Copyright 2012 Trend 3
12/24/2012 Micro Inc. 5

36. Dare to fail
• Fail is inevitable for fast changing application.
(Web, service)
• Fast changing
• Create dare-to-fail process and environment
• Facebook
• Backup plan
• Rollback plan