The document discusses how to build a system that can handle high access requests. It covers optimizing performance at the node level and scaling to multiple nodes. It then discusses various problems that can occur at different levels, from the client to the server to cross-server, and provides solutions for issues like caching, load balancing, and communication between servers. The overall goal is to understand where bottlenecks can occur and how to optimize each component to build a scalable system that can handle high traffic loads.

1. CACHE
Cách build hệ thống
đáp ứng truy cập cao
Châu Nguyễn Nhật Thanh
Head of Game Backend and Cloud
VNG Corp.

2. Content
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

3. How to scale
● Optimize at node
– Maximize the performances at node
● Scale to multiples nodes
– How to scale ?
● Scale UP
● Scale OUT

4. 1.Client Problems
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

5. 1.Client Problems
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

6. 1. Client Problems
● Web browser (HTTP)
– Static contents: js, css , image, video
– JS, CSS, image:
● minify, merge
● Compress header
– Caching:
● Expires header
● Local storage
● JS caching

7. 1. Client Problems
● Mobile client
– Local storage
– In app resources bundle

8. 1. Client Problems
● Tools
– Profiling:
● Chrome dev tools
– Minifying:
● Webpack, gulp, elixir (Laravel)
– File serving:
● Nginx

9. 1. Client Problems
● Profiling

10. 1. Client Problems
● Minifying
– gulp –production
● File serving with expire header
location ~* .(?:ico|css|js|gif|jpe?g|png)$ {
expires 30d;
add_header Pragma public;
add_header Cache-Control "public";
}

11. 2. Client to ISP
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

12. 2. Client to ISP
● CDN (Content Delivery Network)
– Redirect (302)
– DNS server
● Content in ISP DC
● Return IP of ISP near your devices

13. 2. Client to ISP

14. 2. Client to ISP Problems

15. 3.ISP Problem
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

16. 3.ISP Problem
● Routing
– Internal ISP
– Cross ISP
● Capacity of lease line between 2 ISPs
● Traffic monitoring
– 3G/4G users
● Telco routing

17. 4.ISP to DC Problem
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

18. 4.ISP to DC Problem
● Routing
– DC to ISP
– Internal DC
● FW
● Network hardware capability
● Cross product routing
– Bandwidth
– FW
– SDN ????

19. 5.Server Problems
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

20. 5.Server Problems
● Network card
– Gigabits
– Fiber
– Virtualization support ???
● CPU
● RAM
● GPU

21. 5.Server Problems
● Scale UP vs scale OUT

22. 5.Server Problems
● How to load balancing between nodes
– HA Proxy
– Nginx
– F5

23. 5.Server Problems
● HA Proxy config sample
backend web-backend
balance roundrobin
server web1 web1.yourdomain.com:80 check
server web2 web2.yourdomain.com:80 check
backend blog-backend
balance roundrobin
mode http
server blog1 blog1.yourdomain.com:80 check

24. 5.Server Problems
● HA Proxy config sample
frontend http
bind *:80
mode http
acl url_blog path_beg /blog
use_backend blog-backend if url_blog
default_backend web-backend

25. 6. Kernel Problems
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

26. 6. Kernel Problems
● Drivers for HW optimization
● Max fd
● IP forwarder
● Local FW
– Iptables
● Network IO multiplexing support ???
– EPOLL
● https://code.facebook.com/posts/938078729581886/impro
ving-the-linux-kernel-with-upstream-contributions/

27. 7. Application Problems
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

29. 7. Application Problems

30. 7. Application Problems
● Data structures
– Array
– Hash map
– Linked list
==> how to use ???

31. 7. Application Problems
● Caching
– Local Cache
● Use Data structure
– External Cache
● Memcache
● Redis
– Caching algorithm
● LRU
● FIFO
● ….

32. 7. Application Problems
● Data serialize
– Text Protocol
● Json, XML
– Binary
● Thrift, Protobuff …
– Compression
● Snapy, ZIP, GZIP

33. 7. Application Problems
● Local cache
– Create a hash map
● localCache = new HashMap();
– Serialize object need to be cached
● serializedObj = JSON.encode(obj);
– Compress data
● compressedObj = GZIP.compress(serializedObj);
– Put to hash cache
● localCache.put(key, compressedObj);

34. 7. Application Problems
– Get from cache
● compressedObj = localCache.get(key);
– Decompress data
● serializedObj = GZIP.decompress(compressedObj);
– Deserialize
● obj = JSON.decode(serializedObj);

35. 7. Application Problems

36. 7. Application Problems

37. 7. Application Problems
● Caching algorithm
– Memory is limit
– What to do when memory is full
– FiFO ( First In First Out)
● First item inserted will be removed first

38. 7. Application Problems
– LRU (Least recent used)
http://www.topjavatutorial.com/wp-content/uploads/2016/04/LRU-Cache.png

39. 7. Application Problems
● How to refresh cache
– Expires
– ????

40. 8. Application to kernel problems
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

41. 8. Application to kernel problems
● Memory copy overhead
– Share mem
– Zero copy buffer
● IO multiplexing
– Blocking
● Process base
● Thread base
– Nonblocking
● EPOLL
● KQUEUE
● Use libevent

42. 9.Cross Server Problems
Client Internet
1
2
3
4
Server
Kernel
App
Server
6
7
8
5
9

43. 9.Cross Server Problems
● Cross platform RPC
– Thrift
– GRPC
– Memcache protocol

44. 9.Cross Server Problems
● External Cache server
– Memcached
● Memory only
● Slab memory management
● Use libevent for scalable sockets
● Key/Value only
– Twemcache (twitter memcached)
● Heavy
● https://github.com/twitter/twemcache/wiki/Random-
Eviciton-vs-Slab-Automove

45. 9.Cross Server Problems
Memcached Slabs
(http://charlee.li/memcached-002.html)

46. 9.Cross Server Problems
● Redis
– Memory and disk
– Simple data structure support
● String, hash, sets, list, sorted sets
– Pub/Sub
– Lua scripts
– Support memcached protocol
– Cluster support

47. 9.Cross Server Problems
● Redis data structures
https://image.slidesharecdn.com/kickingasswithredis-
120618095415-phpapp02/95/kicking-ass-with-redis-3-728.jpg

48. Thank you
● Q&A
● We are hiring!!!!
● thanhcnn@vng.com.vn