HAProxy as Egress Controller

1. Julien Pivotto Inuits.eu Open Source Consultant @roidelapluie roidelapluie@inuits.eu HAProxy as Egress Controller HAProxyConf - November 12

2. Introduction Setting up the scene

3. HAProxy, the other way around We use HAProxy in a quite unusual way... ● send requests to the external world ● initialize TLS with the external world ● throttle requests to the external world

4. Context ● Healthcare services in Belgium ● Transmitting millions of messages everyday between diﬀerent parties ○ Thousands of users ○ Dozens of partners ● Dozens of services: Monolith & Microservices ● Long lived services & technologies (> 10 years) ● SOAP-XML & REST-JSON

5. Challenges ● Ensure that transactions are successful ● Monitor and react upon failure at partners ● Provide a uniﬁed view over calls to the outside world ● Use modern technology (latest TLS versions, SNI), even with old apps ● Authenticate requests ● Make it easy for application owners to interact with the outside world

6. Architecture First things ﬁrst

7. ● HAProxy is isolated from Apps ● Only HAProxy has Internet Access

8. How HTTPS forward proxies work ● HTTPS forward proxies just open TCP sockets and pass them to clients ● Clients are in charge of all the TLS connection ● Proxies does not see the content of requests

9. Initiating TLS requests from HAProxy ● Client connects to HAProxy in TLS ● HAProxy connects to external partner in TLS

10. Identifying requests Instead of calling: https://www.example.com/helloworld Application “myback” will call: https://proxy.inuits.eu/myback/prod/example/prod/high/helloworld

11. Wait … What? Instead of calling: https://www.example.com/helloworld Application “myback” will call: https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld

12. Proxy URL composition https://proxy.inuits.eu/ <APP>/<ENV>/ <PARTNER>/<ENV>/<APP>/ <SLA>/ <PATH> Identiﬁes the caller app - the partner app - the expected response time.

13. What the URL tells us ... https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld The application myback in production is calling the URI /helloworld of the service www of the partner example in production and expects an quick answer (high sla) If you can read one, you can read all of them.

14. First remarks ● Use HTTPS internally: ○ Before the HAProxy, direct HTTPS connections were made from the apps. ○ Everything that was encrypted stays encrypted in the new model. ● Applications need to change the URL they use to contact partners. ● This method “cuts” tls; there are two https connections (one to the HAProxy and one from the HAProxy).

15. Access Control

16. Easy Access control: IP-Based We use HAProxy’s ACL’s to deﬁne who are our clients. frontend proxy acl client:myback:prod src 172.21.132.0/25 acl client:myback:dev src 172.21.131.0/25 acl client:myback:acc src 172.21.130.0/25 acl client:legback:dev src 172.21.132.2 172.21.132.4 acl client:3rdapp:prod src 172.21.132.0/25 ACL Name = client:<application-name>:<application-env>

17. Who access what? Remember: https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld frontend proxy acl client:myback:prod src 172.21.132.0/25 acl partner:myback:prod:example:prod:www:high path_beg /myback/prod/example/prod/www/high use_backend example:prod:www:high if partner:myback:prod:example:prod:www:high client:myback:prod Use speciﬁc backend if: URL matches a known backend and comes from the app’s IP address

18. Where are we? ● The client identifies itself in the URL ● HAProxy checks that app is correct with the source IP address ○ Monitoring purpose ○ IP-Based ACL is not security ● The client identifies the partner, env, app it wants to reach ● A “SLA” is defined that redirects to a correctly configured backend

19. HAProxy features used so far... ● ACL with source IP address ● ACL with path_beg to match the start of the URI ● use_backend to specify the backend to use depending on conditions Note: in our case, “backend” is an external partner.

20. SLA’s

21. SLA’s are simply: setting timeouts Timeouts are set per backend in HAProxy. Some transactions are expected to last several minutes, other a few milliseconds. Deﬁning those timeouts in each application is not practical, but you want safe values to avoid blocking your app because partners respond slowly.

22. Our “SLA” levels towards partners 1. Asynchronous calls: low - posting big files a. 301 s (client, server) b. 5 s (connect) 2. Normal calls: medium a. 31s (server) b. 5s (client) c. 1s (connect) 3. Synchronous calls: high - an end-user is waiting behind their screen a. 11s (server) b. 5s (client) c. 1s (connect) 4. Specific SLA for specific apps (3s up to 3000s)

23. 1 backend / partner / sla backend example:prod:www:high timeout connect 1000 timeout client 5000 timeout server 11000 timeout http-request 5000 timeout queue 0s Each “SLA” requires a backend. We disable queuing.

24. Masquerading requests HAProxy isn’t a forward proxy!

25. How to make the request we want. Instead of calling: https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld We want to call: https://www.example.com/helloworld What needs to change? ● Hostname ● SNI ● Path

26. Altering the query backend example:prod:www:high balance first http-request set-header Host www.example.com reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2 fullconn 20 server www www.example.com:443 maxconn 20 sni str(www.example.com) ssl ca-file /etc/ssl/certs/ca-bundle.crt resolvers mydns resolve-prefer ipv4

27. Step by step: changing URI From /myback/prod/example/prod/www/high/helloworld to /helloworld backend example:prod:www:high reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2 reqrep will replace the http request line. 1 will be the METHOD and 2 the actual URI. From … POST /myback/prod/example/prod/www/high/helloworld To … POST /helloworld

28. Step by step: changing the hostname 2 diﬀerent things: the HTTP host header + the SNI TCP header. SNI - TLS extension to specify hostname upon TLS negotiation. backend example:prod:www:high http-request set-header Host www.example.com server www www.example.com:443 sni str(www.example.com) ssl ca-file /etc/ssl/certs/ca-bundle.crt We validate partners certiﬁcate with OS CA bundle.

29. HAProxy features used... ● reqrep to alter request line and change URI ● http-request set-header to change/add a header ● The str() function to work with strings ● The sni instruction to tell HAProxy to do SNI with the backends Note: in our case, “backend” is not a “backend”, it is an external partner.

30. A word about DNS ...

31. Remember our backend? backend example:prod:www:high resolvers mydns resolve-prefer ipv4 resolvers mydns nameserver dns1 172.21.16.6:53 nameserver dns2 172.21.16.34:53 timeout resolve 1s timeout retry 1s resolve_retries 5 hold other 10s hold refused 10s hold nx 10s hold timeout 10s hold valid 300s hold obsolete 10s

32. Lessons learned about DNS ● When DNS resolution fails, error message in the logs in unclear ● HAProxy uses OS DNS resolution at startup, not resolvers ○ If a hostname is in /etc/hosts, HAProxy will accept the conﬁg, but the backend won’t work ○ Same can happen if local resolver (/etc/resolv.conf) != HAProxy resolvers section

33. Who needs DNS anyway? Real world scenario: ● Partner does not publish DNS entries ● Partner does not publish DNS entries … yet ● Partner uses the same hostname but with diﬀerent IP addresses for diﬀerent environments (don’t ask why...)

34. NO-DNS Scenario backend example:prod:www:high http-request set-header Host www.example.com server www 93.184.216.34:443 sni str(www.example.com) ssl ca-file /etc/ssl/certs/ca-bundle.crt With this conﬁguration, no DNS entry is required. HAProxy will still alter the query to set hostname and do correct SNI.

35. Advanced topics

36. Canary releases Objective: redirect X % of requests to a new service at partner (requests stay the same) frontend proxy http-request set-path /myback/prod/example/prod/www2/high if { path /myback/prod/example/prod/www/high } { rand(100) lt 10 } If that is set before the ACL with use_backend, then this is the URI that those ACL will use, redirecting 10% of the traﬃc from www to www2.

37. Point in time roll out Objective: Partner informs us that on Sunday 10AM they will change URL/URI. Before: putting someone oncall to change all the apps at 10AM. Now: frontend proxy http-request set-path /myback/prod/example/prod/www2/high if { path /myback/prod/example/prod/www/high } { date() ge 1571558400 }

38. Advanced SSL Interesting SSL keywords: ● 2-way SSL with client certiﬁcate: crt <path to the crt ﬁle> ● Force a TLS version: force-tlsv12 ensures that we talk to backend only on TLS 1.2

39. Setup & maintenance

40. Configuration Management ● This setup produces a big file (4895 lines) ● But the input is minimal: ○ Who are the clients ○ Who are the partners ○ What are the SLA ● Then, we use ansible to mix them all ● Achievements: ○ Decouple the data from the config ○ Abstract HAProxy knowledge from developers, who just need to alter high level YAML files

41. Monitoring ● Make HAProxy log to a ﬁle ● Read the ﬁle, you will see: ○ client/env ○ partner/env ○ backend actually used (useful for canaries etc...) ○ status ○ duration ● We use: prometheus, grafana, HAProxy_exporter, mtail

42. mtail metrics Parsing HAProxy log file to get Prometheus metrics that match our URL model. sum(rate(http_requests_duration_ms_count{ partner="exemple",partner_env="prod",partner_service="www", client="myback",client_env="prod" }[5m])) by(code) github.com/roidelapluie/haproxy-egress

43. Conclusion

44. How we dit if ● Abstract configuration in simple concepts (client, partner, sla) for cfgmgmt ○ App maintainers provide simple input ○ Config management tools turn the input in a haproxy config file ● Putting correct monitoring in place (analyzing log files) ● Using advanced HAProxy features

45. The benefits ● Full understanding of egresses of our applications ● Detailed metrics about connectivity and response time of partners ● Quick alerts when partners are not responding ○ Identification of the apps ○ Quick evaluation of business impact ● Egress with a modern TLS stack (TLS 1.2) ● Unified timeouts / tcp retries rules ● Delegated 2-way-ssl ● DNS bypass, canary releases, date-triggered URL changes… ● Flexibility over requests without restarting the client apps!

46. Questions & Answers

47. Julien Pivotto Inuits.eu Open Source Consultant @roidelapluie roidelapluie@inuits.eu Thank you