Wang Yuxi presented how Umeng migrated their PBs data from Beijing to Shanghai over the course of 6 months. They setup a dedicated 1Gbps fiber connection between the cities and built tools to transfer data from HBase, MongoDB, MySQL, and Redis. They also provisioned applications between the environments and performed benchmarking and stress tests. The migration was completed successfully over a Friday night with no outages.

1. How We Migrate PBs Data from Beijing to Shanghai
Wang Yuxi, Umeng
w@umeng.com

2. Agenda
● Why migrating
● Current Infrastructure
● Environment Setup
● Data Transfer(HBase)
● Data Transfer (MongoDB)
● Data Transfer (Mysql/Redis)
● Application Provision
● Monitoring
● Benchmark and Stress Testing
● Go!
● Results
● Recap

3. About Me
● Before 2014, the only ops at Umeng
● Now, core member of ops team
● Technical generalist, responsible for the overall reliability and performance
of Umeng
● ArchLinux user
@Jasey_Wang | http://JaseyWang.Me

4. About Umeng
● Founded on April 2010
● Incubated by Innovation
Works
● $10 Million raised from
Matrix China
● Acquired by Alibaba
● Largest Mobile app
analytical platform in
China
● 400K+ Apps
● ~1B mobile device

5. Why Migrating
● Capex/Opex
● Unlimited resources, no worries for IDC, Racks, Bandwidth, etc.
● Integration with group's internal systems, massive advanced tools
● Make our PBs data safer

6. Current Infrastructure
● Data center: 4
● Server: 1000
● Networking device: 100
● Bandwidth: 4Gbps+
● Realtime analytics: 150K qps
● Batch processing: 4P/5P storage usage
● Know more? see Umeng Operations Infrastructure & Practice

7. Current Infrastructure(Cont.)

8. Environment Setup
● 1G dedicated fiber between Beijing and Shanghai ready
● Due to security reason, can only send SYN from Shanghai to Beijing
● Setup DNAT for Beijing cluster
● Raw data transfer test, saturate the bandwidth(iperf -P/netperf)

9. Data transfer(HBase)
● HBase, 0.94 @ Beijing, 0.98 @ Shanghai
● Build-in import/export tools don’t work
● Write own import/export tool, integrity check
● Task scheduler, historical & daily incremental data
● 2 months data transfer

10. Data transfer(HBase)(Cont.)

11. Data transfer(MongoDB)
● Master/Slave
○ 2.4.11, obsolete
○ build-in replica mechanism
● Primary/Secondary/Arbiter
○ replica arch
○ since Beijing can’t connect to Shanghai, write tools to read Oplog and
replay into Shanghai DB cluster
● Oplog, lag, TCP keepalive, slow query, dead lock

12. Data Transfer(MySQL/Redis)
● MySQL
○ percona-xtrabackup, quite handy
● Redis
○ single instance, slaveof command
○ twemproxy, export and import later by tools

13. Application Provision
● Stateless or stateful?
● Kafka & Mirror
○ consumer/producer queue
○ qps, topics, io
● Storm
○ throughput, lag
● Zookeeper
○ 5 nodes, 4 letter words, log cleaning

14. Monitoring
● Internal monitoring system backed by HBase
● Zabbix, Ganglia
● Graphite for metrics
● Monit for process monitoring

15. Benchmark and Stress Testing
● Single component
○ system level metrics
○ application level metrics
● Multiple components
● Part of online traffic

16. Go!
● What about PLAN B or C?
○ plan B usually does not work
● Friday night from 00:00 to 08:00
● Route tens of products traffic to Shanghai smoothly
● The site is fully available without outage

17. Results
● Sophisticated projects, all members in
● 6 months work pays off
● Hugely successful, no roll-back
● Part of products now running on private cloud

18. Recap
● Tools are #1 productive forces
● Test, test and test
● Monitoring and metrics

19. End
Q & A