SlideShare a Scribd company logo
Julien Pivotto
Inuits.eu
Open Source Consultant
@roidelapluie
roidelapluie@inuits.eu
HAProxy as
Egress Controller
HAProxyConf - November 12
Introduction
Setting up the scene
HAProxy, the other way around
We use HAProxy in a quite unusual way...
● send requests to the external world
● initialize TLS with the external world
● throttle requests to the external world
Context
● Healthcare services in Belgium
● Transmitting millions of messages everyday between different parties
○ Thousands of users
○ Dozens of partners
● Dozens of services: Monolith & Microservices
● Long lived services & technologies (> 10 years)
● SOAP-XML & REST-JSON
Challenges
● Ensure that transactions are successful
● Monitor and react upon failure at partners
● Provide a unified view over calls to the outside world
● Use modern technology (latest TLS versions, SNI), even with old apps
● Authenticate requests
● Make it easy for application owners to interact with the outside world
Architecture
First things first
● HAProxy is isolated from Apps
● Only HAProxy has Internet Access
How HTTPS forward proxies work
● HTTPS forward proxies just open TCP sockets and pass them to clients
● Clients are in charge of all the TLS connection
● Proxies does not see the content of requests
Initiating TLS requests from HAProxy
● Client connects to HAProxy in TLS
● HAProxy connects to external partner in TLS
Identifying requests
Instead of calling:
https://www.example.com/helloworld
Application “myback” will call:
https://proxy.inuits.eu/myback/prod/example/prod/high/helloworld
Wait … What?
Instead of calling:
https://www.example.com/helloworld
Application “myback” will call:
https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld
Proxy URL composition
https://proxy.inuits.eu/
<APP>/<ENV>/
<PARTNER>/<ENV>/<APP>/
<SLA>/
<PATH>
Identifies the caller app - the partner app - the expected response time.
What the URL tells us ...
https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld
The application myback in production
is calling the URI /helloworld
of the service www of the partner example in production
and expects an quick answer (high sla)
If you can read one, you can read all of them.
First remarks
● Use HTTPS internally:
○ Before the HAProxy, direct HTTPS connections were made from the apps.
○ Everything that was encrypted stays encrypted in the new model.
● Applications need to change the URL they use to contact partners.
● This method “cuts” tls; there are two https connections (one to the
HAProxy and one from the HAProxy).
Access Control
Easy Access control: IP-Based
We use HAProxy’s ACL’s to define who are our clients.
frontend proxy
acl client:myback:prod src 172.21.132.0/25
acl client:myback:dev src 172.21.131.0/25
acl client:myback:acc src 172.21.130.0/25
acl client:legback:dev src 172.21.132.2 172.21.132.4
acl client:3rdapp:prod src 172.21.132.0/25
ACL Name = client:<application-name>:<application-env>
Who access what?
Remember:
https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld
frontend proxy
acl client:myback:prod src 172.21.132.0/25
acl partner:myback:prod:example:prod:www:high path_beg /myback/prod/example/prod/www/high
use_backend example:prod:www:high if 
partner:myback:prod:example:prod:www:high client:myback:prod
Use specific backend if: URL matches a known backend and comes from the
app’s IP address
Where are we?
● The client identifies itself in the URL
● HAProxy checks that app is correct with the source IP address
○ Monitoring purpose
○ IP-Based ACL is not security
● The client identifies the partner, env, app it wants to reach
● A “SLA” is defined that redirects to a correctly configured backend
HAProxy features used so far...
● ACL with source IP address
● ACL with path_beg to match the start of the URI
● use_backend to specify the backend to use depending on conditions
Note: in our case, “backend” is an external partner.
SLA’s
SLA’s are simply: setting timeouts
Timeouts are set per backend in HAProxy.
Some transactions are expected to last several minutes, other a few
milliseconds. Defining those timeouts in each application is not practical, but
you want safe values to avoid blocking your app because partners respond
slowly.
Our “SLA” levels towards partners
1. Asynchronous calls: low - posting big files
a. 301 s (client, server)
b. 5 s (connect)
2. Normal calls: medium
a. 31s (server)
b. 5s (client)
c. 1s (connect)
3. Synchronous calls: high - an end-user is waiting behind their screen
a. 11s (server)
b. 5s (client)
c. 1s (connect)
4. Specific SLA for specific apps (3s up to 3000s)
1 backend / partner / sla
backend example:prod:www:high
timeout connect 1000
timeout client 5000
timeout server 11000
timeout http-request 5000
timeout queue 0s
Each “SLA” requires a backend.
We disable queuing.
Masquerading
requests
HAProxy isn’t a forward proxy!
How to make the request we want.
Instead of calling:
https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld
We want to call:
https://www.example.com/helloworld
What needs to change?
● Hostname
● SNI
● Path
Altering the query
backend example:prod:www:high
balance first
http-request set-header Host www.example.com
reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2
fullconn 20
server www 
www.example.com:443 maxconn 20 
sni str(www.example.com) 
ssl ca-file /etc/ssl/certs/ca-bundle.crt 
resolvers mydns resolve-prefer ipv4
Step by step: changing URI
From /myback/prod/example/prod/www/high/helloworld to /helloworld
backend example:prod:www:high
reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2
reqrep will replace the http request line. 1 will be the METHOD and 2 the
actual URI.
From … POST /myback/prod/example/prod/www/high/helloworld
To … POST /helloworld
Step by step: changing the hostname
2 different things: the HTTP host header + the SNI TCP header.
SNI - TLS extension to specify hostname upon TLS negotiation.
backend example:prod:www:high
http-request set-header Host www.example.com
server www 
www.example.com:443 
sni str(www.example.com) 
ssl ca-file /etc/ssl/certs/ca-bundle.crt
We validate partners certificate with OS CA bundle.
HAProxy features used...
● reqrep to alter request line and change URI
● http-request set-header to change/add a header
● The str() function to work with strings
● The sni instruction to tell HAProxy to do SNI with the backends
Note: in our case, “backend” is not a “backend”, it is an external partner.
A word about DNS ...
Remember our backend?
backend example:prod:www:high
resolvers mydns resolve-prefer ipv4
resolvers mydns
nameserver dns1 172.21.16.6:53
nameserver dns2 172.21.16.34:53
timeout resolve 1s
timeout retry 1s
resolve_retries 5
hold other 10s
hold refused 10s
hold nx 10s
hold timeout 10s
hold valid 300s
hold obsolete 10s
Lessons learned about DNS
● When DNS resolution fails, error message in the logs in unclear
● HAProxy uses OS DNS resolution at startup, not resolvers
○ If a hostname is in /etc/hosts, HAProxy will accept the config, but the backend won’t work
○ Same can happen if local resolver (/etc/resolv.conf) != HAProxy resolvers section
Who needs DNS anyway?
Real world scenario:
● Partner does not publish DNS entries
● Partner does not publish DNS entries … yet
● Partner uses the same hostname but with different IP addresses for
different environments (don’t ask why...)
NO-DNS Scenario
backend example:prod:www:high
http-request set-header Host www.example.com
server www 
93.184.216.34:443 
sni str(www.example.com) 
ssl ca-file /etc/ssl/certs/ca-bundle.crt
With this configuration, no DNS entry is required. HAProxy will still alter the
query to set hostname and do correct SNI.
Advanced topics
Canary releases
Objective: redirect X % of requests to a new service at partner (requests stay
the same)
frontend proxy
http-request set-path /myback/prod/example/prod/www2/high if 
{ path /myback/prod/example/prod/www/high } 
{ rand(100) lt 10 }
If that is set before the ACL with use_backend, then this is the URI that those
ACL will use, redirecting 10% of the traffic from www to www2.
Point in time roll out
Objective: Partner informs us that on Sunday 10AM they will change URL/URI.
Before: putting someone oncall to change all the apps at 10AM.
Now:
frontend proxy
http-request set-path /myback/prod/example/prod/www2/high if 
{ path /myback/prod/example/prod/www/high } 
{ date() ge 1571558400 }
Advanced SSL
Interesting SSL keywords:
● 2-way SSL with client certificate: crt <path to the crt file>
● Force a TLS version: force-tlsv12 ensures that we talk to backend only on
TLS 1.2
Setup & maintenance
Configuration Management
● This setup produces a big file (4895 lines)
● But the input is minimal:
○ Who are the clients
○ Who are the partners
○ What are the SLA
● Then, we use ansible to mix them all
● Achievements:
○ Decouple the data from the config
○ Abstract HAProxy knowledge from developers, who just need to alter high level YAML files
Monitoring
● Make HAProxy log to a file
● Read the file, you will see:
○ client/env
○ partner/env
○ backend actually used (useful for canaries etc...)
○ status
○ duration
● We use: prometheus, grafana, HAProxy_exporter, mtail
mtail metrics
Parsing HAProxy log file to get Prometheus metrics that match our URL model.
sum(rate(http_requests_duration_ms_count{
partner="exemple",partner_env="prod",partner_service="www",
client="myback",client_env="prod"
}[5m])) by(code)
github.com/roidelapluie/haproxy-egress
Conclusion
How we dit if
● Abstract configuration in simple concepts (client, partner, sla) for cfgmgmt
○ App maintainers provide simple input
○ Config management tools turn the input in a haproxy config file
● Putting correct monitoring in place (analyzing log files)
● Using advanced HAProxy features
The benefits
● Full understanding of egresses of our applications
● Detailed metrics about connectivity and response time of partners
● Quick alerts when partners are not responding
○ Identification of the apps
○ Quick evaluation of business impact
● Egress with a modern TLS stack (TLS 1.2)
● Unified timeouts / tcp retries rules
● Delegated 2-way-ssl
● DNS bypass, canary releases, date-triggered URL changes…
● Flexibility over requests without restarting the client apps!
Questions & Answers
Julien Pivotto
Inuits.eu
Open Source Consultant
@roidelapluie
roidelapluie@inuits.eu
Thank you

More Related Content

What's hot

Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK Stack
Rohit Sharma
 
Istio a service mesh
Istio   a service meshIstio   a service mesh
Istio a service mesh
Chandresh Pancholi
 
Gitlab, GitOps & ArgoCD
Gitlab, GitOps & ArgoCDGitlab, GitOps & ArgoCD
Gitlab, GitOps & ArgoCD
Haggai Philip Zagury
 
Istio service mesh introduction
Istio service mesh introductionIstio service mesh introduction
Istio service mesh introduction
Kyohei Mizumoto
 
Secret Management with Hashicorp’s Vault
Secret Management with Hashicorp’s VaultSecret Management with Hashicorp’s Vault
Secret Management with Hashicorp’s Vault
AWS Germany
 
OpenTelemetry For Developers
OpenTelemetry For DevelopersOpenTelemetry For Developers
OpenTelemetry For Developers
Kevin Brockhoff
 
Designing a complete ci cd pipeline using argo events, workflow and cd products
Designing a complete ci cd pipeline using argo events, workflow and cd productsDesigning a complete ci cd pipeline using argo events, workflow and cd products
Designing a complete ci cd pipeline using argo events, workflow and cd products
Julian Mazzitelli
 
Service mesh
Service meshService mesh
Service mesh
Arnab Mitra
 
cilium-public.pdf
cilium-public.pdfcilium-public.pdf
cilium-public.pdf
Sanjeev Rampal
 
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
Ji-Woong Choi
 
NGINX, Istio, and the Move to Microservices and Service Mesh
NGINX, Istio, and the Move to Microservices and Service MeshNGINX, Istio, and the Move to Microservices and Service Mesh
NGINX, Istio, and the Move to Microservices and Service Mesh
NGINX, Inc.
 
Drone CI/CD 自動化測試及部署
Drone CI/CD 自動化測試及部署Drone CI/CD 自動化測試及部署
Drone CI/CD 自動化測試及部署
Bo-Yi Wu
 
Deploying your first application with Kubernetes
Deploying your first application with KubernetesDeploying your first application with Kubernetes
Deploying your first application with Kubernetes
OVHcloud
 
Bare Metal Cluster with Kubernetes, Istio and Metallb | Nguyen Phuong An, Ngu...
Bare Metal Cluster with Kubernetes, Istio and Metallb | Nguyen Phuong An, Ngu...Bare Metal Cluster with Kubernetes, Istio and Metallb | Nguyen Phuong An, Ngu...
Bare Metal Cluster with Kubernetes, Istio and Metallb | Nguyen Phuong An, Ngu...
Vietnam Open Infrastructure User Group
 
[2018] 오픈스택 5년 운영의 경험
[2018] 오픈스택 5년 운영의 경험[2018] 오픈스택 5년 운영의 경험
[2018] 오픈스택 5년 운영의 경험
NHN FORWARD
 
Cloud Native Bern 05.2023 — Zero Trust Visibility
Cloud Native Bern 05.2023 — Zero Trust VisibilityCloud Native Bern 05.2023 — Zero Trust Visibility
Cloud Native Bern 05.2023 — Zero Trust Visibility
Raphaël PINSON
 
OpenStack Architecture
OpenStack ArchitectureOpenStack Architecture
OpenStack Architecture
Mirantis
 
Admission controllers - PSP, OPA, Kyverno and more!
Admission controllers - PSP, OPA, Kyverno and more!Admission controllers - PSP, OPA, Kyverno and more!
Admission controllers - PSP, OPA, Kyverno and more!
SebastienSEYMARC
 
08 04 mise en place d'un serveur mandataire (proxy)
08 04 mise en place d'un serveur mandataire (proxy)08 04 mise en place d'un serveur mandataire (proxy)
08 04 mise en place d'un serveur mandataire (proxy)
Noël
 
Kubernetes Problem-Solving
Kubernetes Problem-SolvingKubernetes Problem-Solving
Kubernetes Problem-Solving
All Things Open
 

What's hot (20)

Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK Stack
 
Istio a service mesh
Istio   a service meshIstio   a service mesh
Istio a service mesh
 
Gitlab, GitOps & ArgoCD
Gitlab, GitOps & ArgoCDGitlab, GitOps & ArgoCD
Gitlab, GitOps & ArgoCD
 
Istio service mesh introduction
Istio service mesh introductionIstio service mesh introduction
Istio service mesh introduction
 
Secret Management with Hashicorp’s Vault
Secret Management with Hashicorp’s VaultSecret Management with Hashicorp’s Vault
Secret Management with Hashicorp’s Vault
 
OpenTelemetry For Developers
OpenTelemetry For DevelopersOpenTelemetry For Developers
OpenTelemetry For Developers
 
Designing a complete ci cd pipeline using argo events, workflow and cd products
Designing a complete ci cd pipeline using argo events, workflow and cd productsDesigning a complete ci cd pipeline using argo events, workflow and cd products
Designing a complete ci cd pipeline using argo events, workflow and cd products
 
Service mesh
Service meshService mesh
Service mesh
 
cilium-public.pdf
cilium-public.pdfcilium-public.pdf
cilium-public.pdf
 
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
 
NGINX, Istio, and the Move to Microservices and Service Mesh
NGINX, Istio, and the Move to Microservices and Service MeshNGINX, Istio, and the Move to Microservices and Service Mesh
NGINX, Istio, and the Move to Microservices and Service Mesh
 
Drone CI/CD 自動化測試及部署
Drone CI/CD 自動化測試及部署Drone CI/CD 自動化測試及部署
Drone CI/CD 自動化測試及部署
 
Deploying your first application with Kubernetes
Deploying your first application with KubernetesDeploying your first application with Kubernetes
Deploying your first application with Kubernetes
 
Bare Metal Cluster with Kubernetes, Istio and Metallb | Nguyen Phuong An, Ngu...
Bare Metal Cluster with Kubernetes, Istio and Metallb | Nguyen Phuong An, Ngu...Bare Metal Cluster with Kubernetes, Istio and Metallb | Nguyen Phuong An, Ngu...
Bare Metal Cluster with Kubernetes, Istio and Metallb | Nguyen Phuong An, Ngu...
 
[2018] 오픈스택 5년 운영의 경험
[2018] 오픈스택 5년 운영의 경험[2018] 오픈스택 5년 운영의 경험
[2018] 오픈스택 5년 운영의 경험
 
Cloud Native Bern 05.2023 — Zero Trust Visibility
Cloud Native Bern 05.2023 — Zero Trust VisibilityCloud Native Bern 05.2023 — Zero Trust Visibility
Cloud Native Bern 05.2023 — Zero Trust Visibility
 
OpenStack Architecture
OpenStack ArchitectureOpenStack Architecture
OpenStack Architecture
 
Admission controllers - PSP, OPA, Kyverno and more!
Admission controllers - PSP, OPA, Kyverno and more!Admission controllers - PSP, OPA, Kyverno and more!
Admission controllers - PSP, OPA, Kyverno and more!
 
08 04 mise en place d'un serveur mandataire (proxy)
08 04 mise en place d'un serveur mandataire (proxy)08 04 mise en place d'un serveur mandataire (proxy)
08 04 mise en place d'un serveur mandataire (proxy)
 
Kubernetes Problem-Solving
Kubernetes Problem-SolvingKubernetes Problem-Solving
Kubernetes Problem-Solving
 

Similar to HAProxy as Egress Controller

The new (is it really ) api stack
The new (is it really ) api stackThe new (is it really ) api stack
The new (is it really ) api stack
Red Hat
 
Secure Developer Access at Decisiv
Secure Developer Access at DecisivSecure Developer Access at Decisiv
Secure Developer Access at Decisiv
Teleport
 
KrakenD API Gateway
KrakenD API GatewayKrakenD API Gateway
KrakenD API Gateway
Albert Lombarte
 
JSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontendJSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontend
Vlad Fedosov
 
SPDY and What to Consider for HTTP/2.0
SPDY and What to Consider for HTTP/2.0SPDY and What to Consider for HTTP/2.0
SPDY and What to Consider for HTTP/2.0
Mike Belshe
 
MySQL X protocol - Talking to MySQL Directly over the Wire
MySQL X protocol - Talking to MySQL Directly over the WireMySQL X protocol - Talking to MySQL Directly over the Wire
MySQL X protocol - Talking to MySQL Directly over the Wire
Simon J Mudd
 
HTTP
HTTPHTTP
Montreal MuleSoft_Meetup_16-Aug.pptx
Montreal MuleSoft_Meetup_16-Aug.pptxMontreal MuleSoft_Meetup_16-Aug.pptx
Montreal MuleSoft_Meetup_16-Aug.pptx
shubhamkalsi2
 
Bringing Learnings from Googley Microservices with gRPC - Varun Talwar, Google
Bringing Learnings from Googley Microservices with gRPC - Varun Talwar, GoogleBringing Learnings from Googley Microservices with gRPC - Varun Talwar, Google
Bringing Learnings from Googley Microservices with gRPC - Varun Talwar, Google
Ambassador Labs
 
JS Fest 2019/Autumn. Влад Федосов. Technology agnostic microservices at SPA f...
JS Fest 2019/Autumn. Влад Федосов. Technology agnostic microservices at SPA f...JS Fest 2019/Autumn. Влад Федосов. Technology agnostic microservices at SPA f...
JS Fest 2019/Autumn. Влад Федосов. Technology agnostic microservices at SPA f...
JSFestUA
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2
 
202107 - Orion introduction - COSCUP
202107 - Orion introduction - COSCUP202107 - Orion introduction - COSCUP
202107 - Orion introduction - COSCUP
Ronald Hsu
 
Zero Downtime JEE Architectures
Zero Downtime JEE ArchitecturesZero Downtime JEE Architectures
Zero Downtime JEE Architectures
Alexander Penev
 
FreeBSD and Hardening Web Server
FreeBSD and Hardening Web ServerFreeBSD and Hardening Web Server
FreeBSD and Hardening Web Server
Muhammad Moinur Rahman
 
Graphs, parallelism and business cases
 Graphs, parallelism and business cases Graphs, parallelism and business cases
Graphs, parallelism and business cases
Daniel Toader
 
Graphs, parallelism and business cases
Graphs, parallelism and business casesGraphs, parallelism and business cases
Graphs, parallelism and business cases
DanBelibov1
 
CN 6131(15) Module IV.docx
CN 6131(15) Module IV.docxCN 6131(15) Module IV.docx
CN 6131(15) Module IV.docx
AkhilMS30
 
CN 6131(15) Module IV.pdf
CN 6131(15) Module IV.pdfCN 6131(15) Module IV.pdf
CN 6131(15) Module IV.pdf
AsifSalim12
 
Prometheus and Docker (Docker Galway, November 2015)
Prometheus and Docker (Docker Galway, November 2015)Prometheus and Docker (Docker Galway, November 2015)
Prometheus and Docker (Docker Galway, November 2015)
Brian Brazil
 
Meet with Meteor
Meet with MeteorMeet with Meteor
Meet with Meteor
Tahmina Khatoon
 

Similar to HAProxy as Egress Controller (20)

The new (is it really ) api stack
The new (is it really ) api stackThe new (is it really ) api stack
The new (is it really ) api stack
 
Secure Developer Access at Decisiv
Secure Developer Access at DecisivSecure Developer Access at Decisiv
Secure Developer Access at Decisiv
 
KrakenD API Gateway
KrakenD API GatewayKrakenD API Gateway
KrakenD API Gateway
 
JSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontendJSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontend
 
SPDY and What to Consider for HTTP/2.0
SPDY and What to Consider for HTTP/2.0SPDY and What to Consider for HTTP/2.0
SPDY and What to Consider for HTTP/2.0
 
MySQL X protocol - Talking to MySQL Directly over the Wire
MySQL X protocol - Talking to MySQL Directly over the WireMySQL X protocol - Talking to MySQL Directly over the Wire
MySQL X protocol - Talking to MySQL Directly over the Wire
 
HTTP
HTTPHTTP
HTTP
 
Montreal MuleSoft_Meetup_16-Aug.pptx
Montreal MuleSoft_Meetup_16-Aug.pptxMontreal MuleSoft_Meetup_16-Aug.pptx
Montreal MuleSoft_Meetup_16-Aug.pptx
 
Bringing Learnings from Googley Microservices with gRPC - Varun Talwar, Google
Bringing Learnings from Googley Microservices with gRPC - Varun Talwar, GoogleBringing Learnings from Googley Microservices with gRPC - Varun Talwar, Google
Bringing Learnings from Googley Microservices with gRPC - Varun Talwar, Google
 
JS Fest 2019/Autumn. Влад Федосов. Technology agnostic microservices at SPA f...
JS Fest 2019/Autumn. Влад Федосов. Technology agnostic microservices at SPA f...JS Fest 2019/Autumn. Влад Федосов. Technology agnostic microservices at SPA f...
JS Fest 2019/Autumn. Влад Федосов. Technology agnostic microservices at SPA f...
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
202107 - Orion introduction - COSCUP
202107 - Orion introduction - COSCUP202107 - Orion introduction - COSCUP
202107 - Orion introduction - COSCUP
 
Zero Downtime JEE Architectures
Zero Downtime JEE ArchitecturesZero Downtime JEE Architectures
Zero Downtime JEE Architectures
 
FreeBSD and Hardening Web Server
FreeBSD and Hardening Web ServerFreeBSD and Hardening Web Server
FreeBSD and Hardening Web Server
 
Graphs, parallelism and business cases
 Graphs, parallelism and business cases Graphs, parallelism and business cases
Graphs, parallelism and business cases
 
Graphs, parallelism and business cases
Graphs, parallelism and business casesGraphs, parallelism and business cases
Graphs, parallelism and business cases
 
CN 6131(15) Module IV.docx
CN 6131(15) Module IV.docxCN 6131(15) Module IV.docx
CN 6131(15) Module IV.docx
 
CN 6131(15) Module IV.pdf
CN 6131(15) Module IV.pdfCN 6131(15) Module IV.pdf
CN 6131(15) Module IV.pdf
 
Prometheus and Docker (Docker Galway, November 2015)
Prometheus and Docker (Docker Galway, November 2015)Prometheus and Docker (Docker Galway, November 2015)
Prometheus and Docker (Docker Galway, November 2015)
 
Meet with Meteor
Meet with MeteorMeet with Meteor
Meet with Meteor
 

More from Julien Pivotto

The O11y Toolkit
The O11y ToolkitThe O11y Toolkit
The O11y Toolkit
Julien Pivotto
 
What's New in Prometheus and Its Ecosystem
What's New in Prometheus and Its EcosystemWhat's New in Prometheus and Its Ecosystem
What's New in Prometheus and Its Ecosystem
Julien Pivotto
 
Prometheus: What is is, what is new, what is coming
Prometheus: What is is, what is new, what is comingPrometheus: What is is, what is new, what is coming
Prometheus: What is is, what is new, what is coming
Julien Pivotto
 
What's new in Prometheus?
What's new in Prometheus?What's new in Prometheus?
What's new in Prometheus?
Julien Pivotto
 
Introduction to Grafana Loki
Introduction to Grafana LokiIntroduction to Grafana Loki
Introduction to Grafana Loki
Julien Pivotto
 
Why you should revisit mgmt
Why you should revisit mgmtWhy you should revisit mgmt
Why you should revisit mgmt
Julien Pivotto
 
Observing the HashiCorp Ecosystem From Prometheus
Observing the HashiCorp Ecosystem From PrometheusObserving the HashiCorp Ecosystem From Prometheus
Observing the HashiCorp Ecosystem From Prometheus
Julien Pivotto
 
Monitoring in a fast-changing world with Prometheus
Monitoring in a fast-changing world with PrometheusMonitoring in a fast-changing world with Prometheus
Monitoring in a fast-changing world with Prometheus
Julien Pivotto
 
5 tips for Prometheus Service Discovery
5 tips for Prometheus Service Discovery5 tips for Prometheus Service Discovery
5 tips for Prometheus Service Discovery
Julien Pivotto
 
Prometheus and TLS - an Introduction
Prometheus and TLS - an IntroductionPrometheus and TLS - an Introduction
Prometheus and TLS - an Introduction
Julien Pivotto
 
Powerful graphs in Grafana
Powerful graphs in GrafanaPowerful graphs in Grafana
Powerful graphs in Grafana
Julien Pivotto
 
YAML Magic
YAML MagicYAML Magic
YAML Magic
Julien Pivotto
 
Improved alerting with Prometheus and Alertmanager
Improved alerting with Prometheus and AlertmanagerImproved alerting with Prometheus and Alertmanager
Improved alerting with Prometheus and Alertmanager
Julien Pivotto
 
SIngle Sign On with Keycloak
SIngle Sign On with KeycloakSIngle Sign On with Keycloak
SIngle Sign On with Keycloak
Julien Pivotto
 
Monitoring as an entry point for collaboration
Monitoring as an entry point for collaborationMonitoring as an entry point for collaboration
Monitoring as an entry point for collaboration
Julien Pivotto
 
Incident Resolution as Code
Incident Resolution as CodeIncident Resolution as Code
Incident Resolution as Code
Julien Pivotto
 
Monitor your CentOS stack with Prometheus
Monitor your CentOS stack with PrometheusMonitor your CentOS stack with Prometheus
Monitor your CentOS stack with Prometheus
Julien Pivotto
 
Monitor your CentOS stack with Prometheus
Monitor your CentOS stack with PrometheusMonitor your CentOS stack with Prometheus
Monitor your CentOS stack with Prometheus
Julien Pivotto
 
An introduction to Ansible
An introduction to AnsibleAn introduction to Ansible
An introduction to Ansible
Julien Pivotto
 
Jsonnet
JsonnetJsonnet

More from Julien Pivotto (20)

The O11y Toolkit
The O11y ToolkitThe O11y Toolkit
The O11y Toolkit
 
What's New in Prometheus and Its Ecosystem
What's New in Prometheus and Its EcosystemWhat's New in Prometheus and Its Ecosystem
What's New in Prometheus and Its Ecosystem
 
Prometheus: What is is, what is new, what is coming
Prometheus: What is is, what is new, what is comingPrometheus: What is is, what is new, what is coming
Prometheus: What is is, what is new, what is coming
 
What's new in Prometheus?
What's new in Prometheus?What's new in Prometheus?
What's new in Prometheus?
 
Introduction to Grafana Loki
Introduction to Grafana LokiIntroduction to Grafana Loki
Introduction to Grafana Loki
 
Why you should revisit mgmt
Why you should revisit mgmtWhy you should revisit mgmt
Why you should revisit mgmt
 
Observing the HashiCorp Ecosystem From Prometheus
Observing the HashiCorp Ecosystem From PrometheusObserving the HashiCorp Ecosystem From Prometheus
Observing the HashiCorp Ecosystem From Prometheus
 
Monitoring in a fast-changing world with Prometheus
Monitoring in a fast-changing world with PrometheusMonitoring in a fast-changing world with Prometheus
Monitoring in a fast-changing world with Prometheus
 
5 tips for Prometheus Service Discovery
5 tips for Prometheus Service Discovery5 tips for Prometheus Service Discovery
5 tips for Prometheus Service Discovery
 
Prometheus and TLS - an Introduction
Prometheus and TLS - an IntroductionPrometheus and TLS - an Introduction
Prometheus and TLS - an Introduction
 
Powerful graphs in Grafana
Powerful graphs in GrafanaPowerful graphs in Grafana
Powerful graphs in Grafana
 
YAML Magic
YAML MagicYAML Magic
YAML Magic
 
Improved alerting with Prometheus and Alertmanager
Improved alerting with Prometheus and AlertmanagerImproved alerting with Prometheus and Alertmanager
Improved alerting with Prometheus and Alertmanager
 
SIngle Sign On with Keycloak
SIngle Sign On with KeycloakSIngle Sign On with Keycloak
SIngle Sign On with Keycloak
 
Monitoring as an entry point for collaboration
Monitoring as an entry point for collaborationMonitoring as an entry point for collaboration
Monitoring as an entry point for collaboration
 
Incident Resolution as Code
Incident Resolution as CodeIncident Resolution as Code
Incident Resolution as Code
 
Monitor your CentOS stack with Prometheus
Monitor your CentOS stack with PrometheusMonitor your CentOS stack with Prometheus
Monitor your CentOS stack with Prometheus
 
Monitor your CentOS stack with Prometheus
Monitor your CentOS stack with PrometheusMonitor your CentOS stack with Prometheus
Monitor your CentOS stack with Prometheus
 
An introduction to Ansible
An introduction to AnsibleAn introduction to Ansible
An introduction to Ansible
 
Jsonnet
JsonnetJsonnet
Jsonnet
 

Recently uploaded

find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
Edge AI and Vision Alliance
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
CEPTES Software Inc
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 

Recently uploaded (20)

find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 

HAProxy as Egress Controller

  • 1. Julien Pivotto Inuits.eu Open Source Consultant @roidelapluie roidelapluie@inuits.eu HAProxy as Egress Controller HAProxyConf - November 12
  • 3. HAProxy, the other way around We use HAProxy in a quite unusual way... ● send requests to the external world ● initialize TLS with the external world ● throttle requests to the external world
  • 4. Context ● Healthcare services in Belgium ● Transmitting millions of messages everyday between different parties ○ Thousands of users ○ Dozens of partners ● Dozens of services: Monolith & Microservices ● Long lived services & technologies (> 10 years) ● SOAP-XML & REST-JSON
  • 5. Challenges ● Ensure that transactions are successful ● Monitor and react upon failure at partners ● Provide a unified view over calls to the outside world ● Use modern technology (latest TLS versions, SNI), even with old apps ● Authenticate requests ● Make it easy for application owners to interact with the outside world
  • 7. ● HAProxy is isolated from Apps ● Only HAProxy has Internet Access
  • 8. How HTTPS forward proxies work ● HTTPS forward proxies just open TCP sockets and pass them to clients ● Clients are in charge of all the TLS connection ● Proxies does not see the content of requests
  • 9. Initiating TLS requests from HAProxy ● Client connects to HAProxy in TLS ● HAProxy connects to external partner in TLS
  • 10. Identifying requests Instead of calling: https://www.example.com/helloworld Application “myback” will call: https://proxy.inuits.eu/myback/prod/example/prod/high/helloworld
  • 11. Wait … What? Instead of calling: https://www.example.com/helloworld Application “myback” will call: https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld
  • 13. What the URL tells us ... https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld The application myback in production is calling the URI /helloworld of the service www of the partner example in production and expects an quick answer (high sla) If you can read one, you can read all of them.
  • 14. First remarks ● Use HTTPS internally: ○ Before the HAProxy, direct HTTPS connections were made from the apps. ○ Everything that was encrypted stays encrypted in the new model. ● Applications need to change the URL they use to contact partners. ● This method “cuts” tls; there are two https connections (one to the HAProxy and one from the HAProxy).
  • 16. Easy Access control: IP-Based We use HAProxy’s ACL’s to define who are our clients. frontend proxy acl client:myback:prod src 172.21.132.0/25 acl client:myback:dev src 172.21.131.0/25 acl client:myback:acc src 172.21.130.0/25 acl client:legback:dev src 172.21.132.2 172.21.132.4 acl client:3rdapp:prod src 172.21.132.0/25 ACL Name = client:<application-name>:<application-env>
  • 17. Who access what? Remember: https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld frontend proxy acl client:myback:prod src 172.21.132.0/25 acl partner:myback:prod:example:prod:www:high path_beg /myback/prod/example/prod/www/high use_backend example:prod:www:high if partner:myback:prod:example:prod:www:high client:myback:prod Use specific backend if: URL matches a known backend and comes from the app’s IP address
  • 18. Where are we? ● The client identifies itself in the URL ● HAProxy checks that app is correct with the source IP address ○ Monitoring purpose ○ IP-Based ACL is not security ● The client identifies the partner, env, app it wants to reach ● A “SLA” is defined that redirects to a correctly configured backend
  • 19. HAProxy features used so far... ● ACL with source IP address ● ACL with path_beg to match the start of the URI ● use_backend to specify the backend to use depending on conditions Note: in our case, “backend” is an external partner.
  • 21. SLA’s are simply: setting timeouts Timeouts are set per backend in HAProxy. Some transactions are expected to last several minutes, other a few milliseconds. Defining those timeouts in each application is not practical, but you want safe values to avoid blocking your app because partners respond slowly.
  • 22. Our “SLA” levels towards partners 1. Asynchronous calls: low - posting big files a. 301 s (client, server) b. 5 s (connect) 2. Normal calls: medium a. 31s (server) b. 5s (client) c. 1s (connect) 3. Synchronous calls: high - an end-user is waiting behind their screen a. 11s (server) b. 5s (client) c. 1s (connect) 4. Specific SLA for specific apps (3s up to 3000s)
  • 23. 1 backend / partner / sla backend example:prod:www:high timeout connect 1000 timeout client 5000 timeout server 11000 timeout http-request 5000 timeout queue 0s Each “SLA” requires a backend. We disable queuing.
  • 25. How to make the request we want. Instead of calling: https://proxy.inuits.eu/myback/prod/example/prod/www/high/helloworld We want to call: https://www.example.com/helloworld What needs to change? ● Hostname ● SNI ● Path
  • 26. Altering the query backend example:prod:www:high balance first http-request set-header Host www.example.com reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2 fullconn 20 server www www.example.com:443 maxconn 20 sni str(www.example.com) ssl ca-file /etc/ssl/certs/ca-bundle.crt resolvers mydns resolve-prefer ipv4
  • 27. Step by step: changing URI From /myback/prod/example/prod/www/high/helloworld to /helloworld backend example:prod:www:high reqrep ^([^ ]* )/[a-zA-Z0-9-]+/[a-z]+/example/prod/www/high[/]?(.*) 1/2 reqrep will replace the http request line. 1 will be the METHOD and 2 the actual URI. From … POST /myback/prod/example/prod/www/high/helloworld To … POST /helloworld
  • 28. Step by step: changing the hostname 2 different things: the HTTP host header + the SNI TCP header. SNI - TLS extension to specify hostname upon TLS negotiation. backend example:prod:www:high http-request set-header Host www.example.com server www www.example.com:443 sni str(www.example.com) ssl ca-file /etc/ssl/certs/ca-bundle.crt We validate partners certificate with OS CA bundle.
  • 29. HAProxy features used... ● reqrep to alter request line and change URI ● http-request set-header to change/add a header ● The str() function to work with strings ● The sni instruction to tell HAProxy to do SNI with the backends Note: in our case, “backend” is not a “backend”, it is an external partner.
  • 30. A word about DNS ...
  • 31. Remember our backend? backend example:prod:www:high resolvers mydns resolve-prefer ipv4 resolvers mydns nameserver dns1 172.21.16.6:53 nameserver dns2 172.21.16.34:53 timeout resolve 1s timeout retry 1s resolve_retries 5 hold other 10s hold refused 10s hold nx 10s hold timeout 10s hold valid 300s hold obsolete 10s
  • 32. Lessons learned about DNS ● When DNS resolution fails, error message in the logs in unclear ● HAProxy uses OS DNS resolution at startup, not resolvers ○ If a hostname is in /etc/hosts, HAProxy will accept the config, but the backend won’t work ○ Same can happen if local resolver (/etc/resolv.conf) != HAProxy resolvers section
  • 33. Who needs DNS anyway? Real world scenario: ● Partner does not publish DNS entries ● Partner does not publish DNS entries … yet ● Partner uses the same hostname but with different IP addresses for different environments (don’t ask why...)
  • 34. NO-DNS Scenario backend example:prod:www:high http-request set-header Host www.example.com server www 93.184.216.34:443 sni str(www.example.com) ssl ca-file /etc/ssl/certs/ca-bundle.crt With this configuration, no DNS entry is required. HAProxy will still alter the query to set hostname and do correct SNI.
  • 36. Canary releases Objective: redirect X % of requests to a new service at partner (requests stay the same) frontend proxy http-request set-path /myback/prod/example/prod/www2/high if { path /myback/prod/example/prod/www/high } { rand(100) lt 10 } If that is set before the ACL with use_backend, then this is the URI that those ACL will use, redirecting 10% of the traffic from www to www2.
  • 37. Point in time roll out Objective: Partner informs us that on Sunday 10AM they will change URL/URI. Before: putting someone oncall to change all the apps at 10AM. Now: frontend proxy http-request set-path /myback/prod/example/prod/www2/high if { path /myback/prod/example/prod/www/high } { date() ge 1571558400 }
  • 38. Advanced SSL Interesting SSL keywords: ● 2-way SSL with client certificate: crt <path to the crt file> ● Force a TLS version: force-tlsv12 ensures that we talk to backend only on TLS 1.2
  • 40. Configuration Management ● This setup produces a big file (4895 lines) ● But the input is minimal: ○ Who are the clients ○ Who are the partners ○ What are the SLA ● Then, we use ansible to mix them all ● Achievements: ○ Decouple the data from the config ○ Abstract HAProxy knowledge from developers, who just need to alter high level YAML files
  • 41. Monitoring ● Make HAProxy log to a file ● Read the file, you will see: ○ client/env ○ partner/env ○ backend actually used (useful for canaries etc...) ○ status ○ duration ● We use: prometheus, grafana, HAProxy_exporter, mtail
  • 42. mtail metrics Parsing HAProxy log file to get Prometheus metrics that match our URL model. sum(rate(http_requests_duration_ms_count{ partner="exemple",partner_env="prod",partner_service="www", client="myback",client_env="prod" }[5m])) by(code) github.com/roidelapluie/haproxy-egress
  • 44. How we dit if ● Abstract configuration in simple concepts (client, partner, sla) for cfgmgmt ○ App maintainers provide simple input ○ Config management tools turn the input in a haproxy config file ● Putting correct monitoring in place (analyzing log files) ● Using advanced HAProxy features
  • 45. The benefits ● Full understanding of egresses of our applications ● Detailed metrics about connectivity and response time of partners ● Quick alerts when partners are not responding ○ Identification of the apps ○ Quick evaluation of business impact ● Egress with a modern TLS stack (TLS 1.2) ● Unified timeouts / tcp retries rules ● Delegated 2-way-ssl ● DNS bypass, canary releases, date-triggered URL changes… ● Flexibility over requests without restarting the client apps!
  • 47. Julien Pivotto Inuits.eu Open Source Consultant @roidelapluie roidelapluie@inuits.eu Thank you