Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HAProxy as Egress Controller


Published on

HAProxy is often used to route ingress traffic, but we use it the other way around. We use it for egress. Our applications talk to the outside world through HAProxy. We get a lot of benefits from this unique approach: throttling, guaranteed response times, unified monitoring, and path rewriting. I will highlight how we use HAProxy at Inuits and how we achieve observability via Prometheus and Grafana.

Published in: Technology
  • Login to see the comments

HAProxy as Egress Controller

  1. 1. Julien Pivotto Open Source Consultant @roidelapluie HAProxy as Egress Controller HAProxyConf - November 12
  2. 2. Introduction Setting up the scene
  3. 3. HAProxy, the other way around We use HAProxy in a quite unusual way... ● send requests to the external world ● initialize TLS with the external world ● throttle requests to the external world
  4. 4. Context ● Healthcare services in Belgium ● Transmitting millions of messages everyday between different parties ○ Thousands of users ○ Dozens of partners ● Dozens of services: Monolith & Microservices ● Long lived services & technologies (> 10 years) ● SOAP-XML & REST-JSON
  5. 5. Challenges ● Ensure that transactions are successful ● Monitor and react upon failure at partners ● Provide a unified view over calls to the outside world ● Use modern technology (latest TLS versions, SNI), even with old apps ● Authenticate requests ● Make it easy for application owners to interact with the outside world
  6. 6. Architecture First things first
  7. 7. ● HAProxy is isolated from Apps ● Only HAProxy has Internet Access
  8. 8. How HTTPS forward proxies work ● HTTPS forward proxies just open TCP sockets and pass them to clients ● Clients are in charge of all the TLS connection ● Proxies does not see the content of requests
  9. 9. Initiating TLS requests from HAProxy ● Client connects to HAProxy in TLS ● HAProxy connects to external partner in TLS
  10. 10. Identifying requests Instead of calling: Application “myback” will call:
  11. 11. Wait … What? Instead of calling: Application “myback” will call:
  12. 12. Proxy URL composition <APP>/<ENV>/ <PARTNER>/<ENV>/<APP>/ <SLA>/ <PATH> Identifies the caller app - the partner app - the expected response time.
  13. 13. What the URL tells us ... The application myback in production is calling the URI /helloworld of the service www of the partner example in production and expects an quick answer (high sla) If you can read one, you can read all of them.
  14. 14. First remarks ● Use HTTPS internally: ○ Before the HAProxy, direct HTTPS connections were made from the apps. ○ Everything that was encrypted stays encrypted in the new model. ● Applications need to change the URL they use to contact partners. ● This method “cuts” tls; there are two https connections (one to the HAProxy and one from the HAProxy).
  15. 15. Access Control
  16. 16. Easy Access control: IP-Based We use HAProxy’s ACL’s to define who are our clients. frontend proxy acl client:myback:prod src acl client:myback:dev src acl client:myback:acc src acl client:legback:dev src acl client:3rdapp:prod src ACL Name = client:<application-name>:<application-env>
  17. 17. Who access what? Remember: frontend proxy acl client:myback:prod src acl partner:myback:prod:example:prod:www:high path_beg /myback/prod/example/prod/www/high use_backend example:prod:www:high if partner:myback:prod:example:prod:www:high client:myback:prod Use specific backend if: URL matches a known backend and comes from the app’s IP address
  18. 18. Where are we? ● The client identifies itself in the URL ● HAProxy checks that app is correct with the source IP address ○ Monitoring purpose ○ IP-Based ACL is not security ● The client identifies the partner, env, app it wants to reach ● A “SLA” is defined that redirects to a correctly configured backend
  19. 19. HAProxy features used so far... ● ACL with source IP address ● ACL with path_beg to match the start of the URI ● use_backend to specify the backend to use depending on conditions Note: in our case, “backend” is an external partner.
  20. 20. SLA’s
  21. 21. SLA’s are simply: setting timeouts Timeouts are set per backend in HAProxy. Some transactions are expected to last several minutes, other a few milliseconds. Defining those timeouts in each application is not practical, but you want safe values to avoid blocking your app because partners respond slowly.
  22. 22. Our “SLA” levels towards partners 1. Asynchronous calls: low - posting big files a. 301 s (client, server) b. 5 s (connect) 2. Normal calls: medium a. 31s (server) b. 5s (client) c. 1s (connect) 3. Synchronous calls: high - an end-user is waiting behind their screen a. 11s (server) b. 5s (client) c. 1s (connect) 4. Specific SLA for specific apps (3s up to 3000s)
  23. 23. 1 backend / partner / sla backend example:prod:www:high timeout connect 1000 timeout client 5000 timeout server 11000 timeout http-request 5000 timeout queue 0s Each “SLA” requires a backend. We disable queuing.
  24. 24. Masquerading requests HAProxy isn’t a forward proxy!
  25. 25. How to make the request we want. Instead of calling: We want to call: What needs to change? ● Hostname ● SNI ● Path
  26. 26. Altering the query backend example:prod:www:high balance first http-request set-header Host reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2 fullconn 20 server www maxconn 20 sni str( ssl ca-file /etc/ssl/certs/ca-bundle.crt resolvers mydns resolve-prefer ipv4
  27. 27. Step by step: changing URI From /myback/prod/example/prod/www/high/helloworld to /helloworld backend example:prod:www:high reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2 reqrep will replace the http request line. 1 will be the METHOD and 2 the actual URI. From … POST /myback/prod/example/prod/www/high/helloworld To … POST /helloworld
  28. 28. Step by step: changing the hostname 2 different things: the HTTP host header + the SNI TCP header. SNI - TLS extension to specify hostname upon TLS negotiation. backend example:prod:www:high http-request set-header Host server www sni str( ssl ca-file /etc/ssl/certs/ca-bundle.crt We validate partners certificate with OS CA bundle.
  29. 29. HAProxy features used... ● reqrep to alter request line and change URI ● http-request set-header to change/add a header ● The str() function to work with strings ● The sni instruction to tell HAProxy to do SNI with the backends Note: in our case, “backend” is not a “backend”, it is an external partner.
  30. 30. A word about DNS ...
  31. 31. Remember our backend? backend example:prod:www:high resolvers mydns resolve-prefer ipv4 resolvers mydns nameserver dns1 nameserver dns2 timeout resolve 1s timeout retry 1s resolve_retries 5 hold other 10s hold refused 10s hold nx 10s hold timeout 10s hold valid 300s hold obsolete 10s
  32. 32. Lessons learned about DNS ● When DNS resolution fails, error message in the logs in unclear ● HAProxy uses OS DNS resolution at startup, not resolvers ○ If a hostname is in /etc/hosts, HAProxy will accept the config, but the backend won’t work ○ Same can happen if local resolver (/etc/resolv.conf) != HAProxy resolvers section
  33. 33. Who needs DNS anyway? Real world scenario: ● Partner does not publish DNS entries ● Partner does not publish DNS entries … yet ● Partner uses the same hostname but with different IP addresses for different environments (don’t ask why...)
  34. 34. NO-DNS Scenario backend example:prod:www:high http-request set-header Host server www sni str( ssl ca-file /etc/ssl/certs/ca-bundle.crt With this configuration, no DNS entry is required. HAProxy will still alter the query to set hostname and do correct SNI.
  35. 35. Advanced topics
  36. 36. Canary releases Objective: redirect X % of requests to a new service at partner (requests stay the same) frontend proxy http-request set-path /myback/prod/example/prod/www2/high if { path /myback/prod/example/prod/www/high } { rand(100) lt 10 } If that is set before the ACL with use_backend, then this is the URI that those ACL will use, redirecting 10% of the traffic from www to www2.
  37. 37. Point in time roll out Objective: Partner informs us that on Sunday 10AM they will change URL/URI. Before: putting someone oncall to change all the apps at 10AM. Now: frontend proxy http-request set-path /myback/prod/example/prod/www2/high if { path /myback/prod/example/prod/www/high } { date() ge 1571558400 }
  38. 38. Advanced SSL Interesting SSL keywords: ● 2-way SSL with client certificate: crt <path to the crt file> ● Force a TLS version: force-tlsv12 ensures that we talk to backend only on TLS 1.2
  39. 39. Setup & maintenance
  40. 40. Configuration Management ● This setup produces a big file (4895 lines) ● But the input is minimal: ○ Who are the clients ○ Who are the partners ○ What are the SLA ● Then, we use ansible to mix them all ● Achievements: ○ Decouple the data from the config ○ Abstract HAProxy knowledge from developers, who just need to alter high level YAML files
  41. 41. Monitoring ● Make HAProxy log to a file ● Read the file, you will see: ○ client/env ○ partner/env ○ backend actually used (useful for canaries etc...) ○ status ○ duration ● We use: prometheus, grafana, HAProxy_exporter, mtail
  42. 42. mtail metrics Parsing HAProxy log file to get Prometheus metrics that match our URL model. sum(rate(http_requests_duration_ms_count{ partner="exemple",partner_env="prod",partner_service="www", client="myback",client_env="prod" }[5m])) by(code)
  43. 43. Conclusion
  44. 44. How we dit if ● Abstract configuration in simple concepts (client, partner, sla) for cfgmgmt ○ App maintainers provide simple input ○ Config management tools turn the input in a haproxy config file ● Putting correct monitoring in place (analyzing log files) ● Using advanced HAProxy features
  45. 45. The benefits ● Full understanding of egresses of our applications ● Detailed metrics about connectivity and response time of partners ● Quick alerts when partners are not responding ○ Identification of the apps ○ Quick evaluation of business impact ● Egress with a modern TLS stack (TLS 1.2) ● Unified timeouts / tcp retries rules ● Delegated 2-way-ssl ● DNS bypass, canary releases, date-triggered URL changes… ● Flexibility over requests without restarting the client apps!
  46. 46. Questions & Answers
  47. 47. Julien Pivotto Open Source Consultant @roidelapluie Thank you