1.
Julien Pivotto
Inuits.eu
Open Source Consultant
@roidelapluie
roidelapluie@inuits.eu
HAProxy as
Egress Controller
HAProxyConf - November 12

2.
Introduction
Setting up the scene

3.
HAProxy, the other way around
We use HAProxy in a quite unusual way...
● send requests to the external world
● initialize TLS with the external world
● throttle requests to the external world

4.
Context
● Healthcare services in Belgium
● Transmitting millions of messages everyday between different parties
○ Thousands of users
○ Dozens of partners
● Dozens of services: Monolith & Microservices
● Long lived services & technologies (> 10 years)
● SOAP-XML & REST-JSON

5.
Challenges
● Ensure that transactions are successful
● Monitor and react upon failure at partners
● Provide a unified view over calls to the outside world
● Use modern technology (latest TLS versions, SNI), even with old apps
● Authenticate requests
● Make it easy for application owners to interact with the outside world

6.
Architecture
First things first

7.
● HAProxy is isolated from Apps
● Only HAProxy has Internet Access

8.
How HTTPS forward proxies work
● HTTPS forward proxies just open TCP sockets and pass them to clients
● Clients are in charge of all the TLS connection
● Proxies does not see the content of requests

9.
Initiating TLS requests from HAProxy
● Client connects to HAProxy in TLS
● HAProxy connects to external partner in TLS

10.
Identifying requests
Instead of calling:
https://www.example.com/helloworld
Application “myback” will call:
https://proxy.inuits.eu/myback/prod/example/prod/high/helloworld

11.
Wait … What?
Instead of calling:
https://www.example.com/helloworld
Application “myback” will call:
https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld

12.
Proxy URL composition
https://proxy.inuits.eu/
<APP>/<ENV>/
<PARTNER>/<ENV>/<APP>/
<SLA>/
<PATH>
Identifies the caller app - the partner app - the expected response time.

13.
What the URL tells us ...
https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld
The application myback in production
is calling the URI /helloworld
of the service www of the partner example in production
and expects an quick answer (high sla)
If you can read one, you can read all of them.

14.
First remarks
● Use HTTPS internally:
○ Before the HAProxy, direct HTTPS connections were made from the apps.
○ Everything that was encrypted stays encrypted in the new model.
● Applications need to change the URL they use to contact partners.
● This method “cuts” tls; there are two https connections (one to the
HAProxy and one from the HAProxy).

15.
Access Control

16.
Easy Access control: IP-Based
We use HAProxy’s ACL’s to define who are our clients.
frontend proxy
acl client:myback:prod src 172.21.132.0/25
acl client:myback:dev src 172.21.131.0/25
acl client:myback:acc src 172.21.130.0/25
acl client:legback:dev src 172.21.132.2 172.21.132.4
acl client:3rdapp:prod src 172.21.132.0/25
ACL Name = client:<application-name>:<application-env>

17.
Who access what?
Remember:
https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld
frontend proxy
acl client:myback:prod src 172.21.132.0/25
acl partner:myback:prod:example:prod:www:high path_beg /myback/prod/example/prod/www/high
use_backend example:prod:www:high if
partner:myback:prod:example:prod:www:high client:myback:prod
Use specific backend if: URL matches a known backend and comes from the
app’s IP address

18.
Where are we?
● The client identifies itself in the URL
● HAProxy checks that app is correct with the source IP address
○ Monitoring purpose
○ IP-Based ACL is not security
● The client identifies the partner, env, app it wants to reach
● A “SLA” is defined that redirects to a correctly configured backend

19.
HAProxy features used so far...
● ACL with source IP address
● ACL with path_beg to match the start of the URI
● use_backend to specify the backend to use depending on conditions
Note: in our case, “backend” is an external partner.

20.
SLA’s

21.
SLA’s are simply: setting timeouts
Timeouts are set per backend in HAProxy.
Some transactions are expected to last several minutes, other a few
milliseconds. Defining those timeouts in each application is not practical, but
you want safe values to avoid blocking your app because partners respond
slowly.

22.
Our “SLA” levels towards partners
1. Asynchronous calls: low - posting big files
a. 301 s (client, server)
b. 5 s (connect)
2. Normal calls: medium
a. 31s (server)
b. 5s (client)
c. 1s (connect)
3. Synchronous calls: high - an end-user is waiting behind their screen
a. 11s (server)
b. 5s (client)
c. 1s (connect)
4. Specific SLA for specific apps (3s up to 3000s)

23.
1 backend / partner / sla
backend example:prod:www:high
timeout connect 1000
timeout client 5000
timeout server 11000
timeout http-request 5000
timeout queue 0s
Each “SLA” requires a backend.
We disable queuing.

24.
Masquerading
requests
HAProxy isn’t a forward proxy!

25.
How to make the request we want.
Instead of calling:
https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld
We want to call:
https://www.example.com/helloworld
What needs to change?
● Hostname
● SNI
● Path

26.
Altering the query
backend example:prod:www:high
balance first
http-request set-header Host www.example.com
reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2
fullconn 20
server www
www.example.com:443 maxconn 20
sni str(www.example.com)
ssl ca-file /etc/ssl/certs/ca-bundle.crt
resolvers mydns resolve-prefer ipv4

27.
Step by step: changing URI
From /myback/prod/example/prod/www/high/helloworld to /helloworld
backend example:prod:www:high
reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2
reqrep will replace the http request line. 1 will be the METHOD and 2 the
actual URI.
From … POST /myback/prod/example/prod/www/high/helloworld
To … POST /helloworld

28.
Step by step: changing the hostname
2 different things: the HTTP host header + the SNI TCP header.
SNI - TLS extension to specify hostname upon TLS negotiation.
backend example:prod:www:high
http-request set-header Host www.example.com
server www
www.example.com:443
sni str(www.example.com)
ssl ca-file /etc/ssl/certs/ca-bundle.crt
We validate partners certificate with OS CA bundle.

29.
HAProxy features used...
● reqrep to alter request line and change URI
● http-request set-header to change/add a header
● The str() function to work with strings
● The sni instruction to tell HAProxy to do SNI with the backends
Note: in our case, “backend” is not a “backend”, it is an external partner.

30.
A word about DNS ...

31.
Remember our backend?
backend example:prod:www:high
resolvers mydns resolve-prefer ipv4
resolvers mydns
nameserver dns1 172.21.16.6:53
nameserver dns2 172.21.16.34:53
timeout resolve 1s
timeout retry 1s
resolve_retries 5
hold other 10s
hold refused 10s
hold nx 10s
hold timeout 10s
hold valid 300s
hold obsolete 10s

32.
Lessons learned about DNS
● When DNS resolution fails, error message in the logs in unclear
● HAProxy uses OS DNS resolution at startup, not resolvers
○ If a hostname is in /etc/hosts, HAProxy will accept the config, but the backend won’t work
○ Same can happen if local resolver (/etc/resolv.conf) != HAProxy resolvers section

33.
Who needs DNS anyway?
Real world scenario:
● Partner does not publish DNS entries
● Partner does not publish DNS entries … yet
● Partner uses the same hostname but with different IP addresses for
different environments (don’t ask why...)

34.
NO-DNS Scenario
backend example:prod:www:high
http-request set-header Host www.example.com
server www
93.184.216.34:443
sni str(www.example.com)
ssl ca-file /etc/ssl/certs/ca-bundle.crt
With this configuration, no DNS entry is required. HAProxy will still alter the
query to set hostname and do correct SNI.

35.
Advanced topics

36.
Canary releases
Objective: redirect X % of requests to a new service at partner (requests stay
the same)
frontend proxy
http-request set-path /myback/prod/example/prod/www2/high if
{ path /myback/prod/example/prod/www/high }
{ rand(100) lt 10 }
If that is set before the ACL with use_backend, then this is the URI that those
ACL will use, redirecting 10% of the traffic from www to www2.

37.
Point in time roll out
Objective: Partner informs us that on Sunday 10AM they will change URL/URI.
Before: putting someone oncall to change all the apps at 10AM.
Now:
frontend proxy
http-request set-path /myback/prod/example/prod/www2/high if
{ path /myback/prod/example/prod/www/high }
{ date() ge 1571558400 }

38.
Advanced SSL
Interesting SSL keywords:
● 2-way SSL with client certificate: crt <path to the crt file>
● Force a TLS version: force-tlsv12 ensures that we talk to backend only on
TLS 1.2

39.
Setup & maintenance

40.
Configuration Management
● This setup produces a big file (4895 lines)
● But the input is minimal:
○ Who are the clients
○ Who are the partners
○ What are the SLA
● Then, we use ansible to mix them all
● Achievements:
○ Decouple the data from the config
○ Abstract HAProxy knowledge from developers, who just need to alter high level YAML files

41.
Monitoring
● Make HAProxy log to a file
● Read the file, you will see:
○ client/env
○ partner/env
○ backend actually used (useful for canaries etc...)
○ status
○ duration
● We use: prometheus, grafana, HAProxy_exporter, mtail

42.
mtail metrics
Parsing HAProxy log file to get Prometheus metrics that match our URL model.
sum(rate(http_requests_duration_ms_count{
partner="exemple",partner_env="prod",partner_service="www",
client="myback",client_env="prod"
}[5m])) by(code)
github.com/roidelapluie/haproxy-egress

43.
Conclusion

44.
How we dit if
● Abstract configuration in simple concepts (client, partner, sla) for cfgmgmt
○ App maintainers provide simple input
○ Config management tools turn the input in a haproxy config file
● Putting correct monitoring in place (analyzing log files)
● Using advanced HAProxy features

45.
The benefits
● Full understanding of egresses of our applications
● Detailed metrics about connectivity and response time of partners
● Quick alerts when partners are not responding
○ Identification of the apps
○ Quick evaluation of business impact
● Egress with a modern TLS stack (TLS 1.2)
● Unified timeouts / tcp retries rules
● Delegated 2-way-ssl
● DNS bypass, canary releases, date-triggered URL changes…
● Flexibility over requests without restarting the client apps!

46.
Questions & Answers

47.
Julien Pivotto
Inuits.eu
Open Source Consultant
@roidelapluie
roidelapluie@inuits.eu
Thank you