Bedrock migrated its video delivery to the cloud. Very quickly, we needed to add a cache layer to absorb the load. AWS load balancers do not allow for advanced load balancing algorithms, such as consistent hashing, so we decided to use HAProxy. Thanks to the HAProxy Runtime API, we developed a tool in Python that allows us to synchronize the AWS AutoScalingGroup with HAProxy. In the end, we scale our load balancers as well as our applications, while optimizing the cache. We also improved our resilience, using among other things HAProxy's retry and redispatch to bypass the network limitations of AWS EC2 instances. During this presentation, I will describe the different versions deployed since the beginning of this cloud migration and their evolution until today. I will detail the significance of each element of our configuration, which allows Bedrock to provide videos to more than 45 million people, by explaining the gains in performance, or resilience.

1. Scaling Bedrock video
delivery to 50 million
users with HAProxy
HAProxyConf 2022 Paris, France
Vincent GALLISSOT
Bedrock Streaming
Lead Cloud Architect
@vgallissot

2. HAProxyConf 2019
RTL's Journey to Kubernetes with HAProxy

3. Bedrock Streaming

4. 50 million active users on our customers' platforms
HYBRID
Netherlands
SVOD
France
France
AVOD
Belgium
HYBRID
Hungary
HYBRID

5. Launched in 2020 Migrated to
in 2022

6. 32 years of episodes 870 episodes aired

7. Multiple CDNs for each customer
Origin of CDNs
● Performant
● Resilient
● Cost effective

8. VOD Origin: between storage and CDNs

9. Packaging: gives control to the player

10. Packaging: cutting one video into small chunks

11. Just in Time Packaging with USP Origin

12. Old days: On Premise VOD Origin
Video storage
Cache to absorb
the load
Just In Time
Packaging
No auto-
scaling
No customer
isolation

13. Issues during lockdown
July 2020
Let’s migrate to AWS

14. V1 on AWS: a NLB & an ASG
Video storage
Single TLS
endpoint
Auto scaling
(CPU &
Bandwidth)

15. Load tests
Photo by Andrew Wulf on Unsplash

16. Performances and scaling not prod ready
5500 GET/S
Scales well
Scales well
Load test:
2000 GET/S

17. Why do we reach this
limit?
How can we push this
limit?

18. Under the hood: metadata
65KB 15B

19. Metadata can be cached locally
1000 hours of
video = 70GB
of cache

20. NLB: no advanced load balancing algorithms
Local cache
RR & LOR

21. The cache is ineffective with Round Robin or Least
Outstanding Requests

22. The more we scale up, the worse it gets

23. Consistent Hashing to the rescue

24. Consistent Hashing + scaling = ♥

25. V2: Add HAProxy with Consistent Hashing
Local cache
Consistent
Hashing

26. Two levels of load balancing
Same list of
backends

27. Haproxy Service Discovery Orchestrator

28. HSDO: Server list stored in DynamoDB

29. HSDO client: Using the runtime API
def sendHaproxyCommand(self, command):
command = command + "n"
haproxySock = socket.socket(socket.AF_UNIX,
socket.SOCK_STREAM)
try:
haproxySock.connect(self.socketPath)
haproxySock.send(command.encode("utf-8"))
https://github.com/BedrockStreaming/hsdo

30. HSDO client: Using the runtime API
https://github.com/BedrockStreaming/hsdo
commands.append(
"set server %s/%s addr %s port %s"
% (
backendName,
serverName,
server.IPAddress,
self.backendServerPort
)
)

31. HSDO client: Using the runtime API
https://github.com/BedrockStreaming/hsdo
commands.append(
"set server %s/%s weight %s"
% (
backendName,
serverName,
server.weight
)
)

32. HSDO client: Makes sure the running configuration
respects the desired state
https://github.com/BedrockStreaming/hsdo
def checkBackendConf(self):
stat = self.sendHaproxyCommand("show stat")
check = self.backendConfReady(stat)

33. Simple backend definition
backend usp-origin
balance hdr(X-VIDEO-ID)
hash-type consistent sdbm avalanche
option httpchk HEAD /status
server-template usp-origin
${AUTO_SCALING_GROUP_MAX_SIZE}
127.0.0.1:80
check
disabled

34. V2: >60% fewer calls to S3 storage
Sudden
cache drop

35. Gradual ramping up controlled by HSDO Server

36. Further optimizations

37. Consistent Hashing unwanted behavior

38. Scaling policies are averages

39. HAProxyConf 2019 - Andrew Rodland

40. Consistent Hashing with Bounded Loads

41. Consistent overflow

42. hash-balance-factor = 140

43. Improve resilience

44. Too much traffic for a single server

45. Packets Per Second limit

46. Micro Bursts

47. Failover mechanisms: server throttles
timeout
connect 20ms

48. Failover mechanisms: bad HTTP codes
retry-on
502-504
502-504

49. Highlights of our HAProxy configuration
defaults
timeout connect 20ms
timeout server 2s
retries 2
retry-on 502 503 504
conn-failure empty-response response-timeout
option redispatch
default-server inter 1s fall 1 rise 10 observe layer7

50. Redispatch happens all the time

51. We migrated all
onprem video traffic
to this V2 in
September 2020

52. V2 is too expensive

53. Inter-AZ traffic: the financial abyss

54. V3: remove inter-AZ traffic

55. V3: remove inter-AZ traffic
Single AZ
HSDO client
sticks to its
own AZ
servers

56. V3: more resilient to an AZ failure

57. V3: cheaper

58. V3: fallback backend

59. V3: fallback backend
frontend fe_main
acl not-enough-ready-servers-in-AZ nbsrv(usp-origin) lt
${HAPROXY_BACKEND_SERVERS_MIN_SIZE}
use_backend fallback-usp-origin if not-enough-ready-servers-in-AZ
use_backend usp-origin

60. 100% EC2 Spot instances
15%

61. Running a Load Balancer on EC2 Spot instances
● Use at least 12 different instance types
● Mix instance families (C, M, R, etc.)
● Prefer many small servers rather than a few large ones
● Take care to the “UP to” vs “baseline” bandwidths

62. EC2 “Up to” bandwidth
Source: https://aws.amazon.com/ec2/instance-types/#Compute_Optimized

63. EC2 “Up to” bandwidth
Source: https://aws.amazon.com/ec2/instance-types/#Compute_Optimized

64. EC2 baseline bandwidth
Source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/compute-
optimized-instances.html#compute-network-performance

65. Time for V4 ?

66. V4: remove Network Load Balancer

67. V3 has a 100% uptime
over the past
22 months

68. Using HAProxy on AWS is not complex

69. Thank you
Vincent GALLISSOT
Bedrock Streaming
Lead Cloud Architect
@vgallissot