1.
Inside Zalo: Developing a mobile messenger for theaudience of millionsBY THANH DAO, VNG CORP.

2.
About meCareer:• VNG: Senior Manager - Head of Zalo & ZingMobile Product Group• Past: Microsoft, KofaxContact:• thanhdn@vng.com.vn• http://me.zing.vn/thanhngocdao• http://facebook.com/thanhdao

3.
My Goals• Talk about how we started a mobile firstproduct, growing the user base and thelearned lessons that lead us here.• Give you a blueprint for how to scale yourmobile server system to millions.

4.
Agenda• Background of Zalo• The user growth story• Continuous development

5.
Landscape before Zalo5Sticky, but...• Don’t support mobile firstuse-case and phonebookgraph.• Not focus on enhance chatand group chat• My mom, sister don’tcontact me via SNSSocial Networks Traditional Messengers OTT IMEarly movers & viral, but…• No profile, social features.• Lack of fun features appealto the young users• Can’t customize a featurefor the Vietnamesecommunity• Low barrier to entryStrong network effect on PC,but...• No social features• Not fun enough• Not mobile friendly

6.
Model of ZaloMobile social messaging:• Based on Mobile First & Phonebook Graph• Combine Context Messaging & Connecting &Circle Socializing• Viral (i.e SMS/Zing/Facebook…)• Sticky• Mass market appeal

7.
Target18-25 26-30Singles Core segment Core segmentUnmarriedcouplesPotential to grow share ?Potential to penetrateMarriedcouplesXNot target yetXNot target yet

8.
ProductPPPPlatformroduct1. Communication: Voice messaging & Call…2. Social: Timeline, People Near By, Room…3. Platform: Page, 3rd Apps, Games.“Fun & Connection & Private”Anchored by basic: Fast, Quality, Cheaproposition(Value)lace (Channel)1. App Stores: Itunes, Google Play, Nokia Store2. Factory Preload on Nokia Asha, WP…3. Side-load at retail shop

9.
Marketing• Position as an Expression/Lifestyle.• Focus on messaging the brand benefit– “Talk more to your loved ones”• 360 Campaign launch at Online, Social, PR, TVC, OOH (Cinema,University,…)

10.
Agenda• Background of Zalo• The user growth story• Continuous development

11.
Agenda• Background of Zalo• The user growth story• Continuous development

12.
0510152025MillionsRegistered User Daily msgUser Discovery Q3/2012: BornProduct• Zing & Phonebook mobile messenger• voice, sticker, draw and people near byUsers feedback and data told us• Good for dating purpose• The draw & games, sticker are fun• The concept of mixing up Zing &phonebook graph is confusing, they wanta separated buddy list• Sending speed is slow with 3G, drain thebattery lifecycle• Nokia version is not available yetSo we• Removed “Chat with Zing friends”feature and focused on the Mobile Firstuse-cases• Rewrote our own socket chat serversinstead of relaying via Zing Me Web Chatservers

13.
User Discovery Q4/2012: TotterProduct• New version Nokia/Droid/IOS• Group, Room• Social TimelineUsers feedback and data told us• Voice/text is fast• UX is much better than the prev one• Spam and privacy concerns• Not enough friends so both retention andengagement are problemsSo we• Improve notification services• Improve picture quality• Scale up advertising budget, Run 360degree marketing campaign 0510152025MillionsRegistered User Daily msg

14.
User Discovery Q1/2013: WalkProduct• Friend Circle• Zalo Page (VIP)• Friend invitation• Support 18 countriesUsers feedback and data told us• Voice message is addictive• Multi-picture upload is convenientSo we• Run Friend Invite Campaign• Developing Call.• Work with 3RD party apps0510152025MillionsRegistered User Daily msg

15.
Agenda• Background of Zalo• The user growth story• Continuous development

16.
Stats• 2.3M registered users (40% Droid; 30% IOS;30% Nokia).• >500K new users join a month.• 800K DAU• 380K peak CCU• 20M messages sent a day.• 39 servers, 150M requests a day.

17.
Server Platform• CentOS• Binary protocol, SSL, TCP/IP, UDP.• C++ for core, java for other business services.• Za-Router, long connection service map and load balancer.• Za-DB: a key-value db storage similar to Amazon Dynamo/Google Big T.• Za-Cache: memory caching system.• Za-Session manager• Za-CDN• Za-Messaging/Signal Socket Server• Za-Stream Download/Upload Server.• Za-Voice/Video Media Proxy Server• Za-Configuration center: synchronize config among services.• Za-Monitor center + zabbix + nagios + cacti + dogslow: trend monitoring.• Postgre SQL(PostGis): store lbs data, can handle >10k requests/sec with memory caching.• Dynamic DNS• HA Proxy: web load balancer(https login services).• Squid: static content caching, small photo…• Hadoop: data analyzing and mining• Elastic search• Scribe for log, backup…

18.
ZAComm RouterZA Socket ProtocolZAComm #1WorkerWorker PoolConn ManagerZAComm #2Worker PoolConn ManagerZAComm #NWorker PoolConn ManagerVIPLBSRoomWorkerMsgGroupShareWorkerAvatarPhotoCoverWorkerContactPrivacyFriendMemCacheLBSPersistentCachingMessage ItemsPersistentCachingPhotoPersistentCachingBackground JobMonitoring

19.
Server Infrastructure39 Servers:• 6 relay storage servers• 9 database servers• 6 business application servers• 3 proxy caching servers• 4 other caching servers• 4 backup servers• 3 log and monitor/configuration serversCapacity:• 1M CCU• I/O speed: 200K message/sec

20.
Scalable Mobile Servers• Scaling connection servers– Each connection server handle a shard of users.– Configuration center.– Service map.• Scaling database– Distributed hash table design.– Backup & recovery.– No alter on large table, create new one.– Soft deletion, Lazy write.• N+1 design– Build horizontally, never less than two of anything.– No single point of failure.• Think about monitor during design, not after.• Identify where bottlenecks are in each tier– Control Port(CPU).– Data Port(Network/Memory/Cache/Storage IO…).

21.
High Performance Mobile Client• Speed:– Different network type, different connecting strategy• Use wi-fi as much as possible.• Maximize bandwidth in fewer calls and group multiplerequests concurrently into a single multi-get request.• Use bigger buffers.– Decouple user transitions from data interactions:• I.e Apply tricky like, silently uploading the image inbackground during the user inputting description.– Avoid large thumb:• Send high-resolution thumb to high resolution devices andlow-resolution thumb to low resolution devices– Cache more often, flush cache regularly• Battery:– HTTP versus TCP/IP reuse.– Avoid constant polling(but be tricky when usingpoll/push)…– Download in bursts/chunks rather than continuouslystreaming.– Minimize aggressive behaviors– Read/Write to cache then flushing a big mount of datainto device storage once.– Use profiler tool to benchmark and measure your appbattery consume speed.

22.
Operation• Tracking userfeedbacks(CS, Store, FB, ZM, Noti5, Buzz…).• Monitor everything:– System performance(Server load/traffics, Memory, Network…).– Business metrics.– QoS statistics: Load-times; errors.– Trending monitors.

23.
Team & Process• 27 developers: Server & Backend, Mobile Client, R&D framework.• 2 system engineers• 5 QA engineers• 3 graphic designers: 1 artist & 2 graphic design• 5 product managers, each person:– Responsible for different group of features: acquisition, connection, communication, social…etc– Has key business metric: conversion, retention, engagement, chat…and combined metrics.• Scrum & continuous deployment process:– Build isolation, enable testing single build without deploy.– Auto Unit Test -> Integrating-> Staging -> Experiment -> Launch.

24.
Questions?

25.
Ready to embrace newchallenges?We are hiring…lepm@vng.com.vnThank you!