Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Проксирование HTTP-запросов web-акселератором / Александр Крижановский (Tempesta Technologies)


Published on

РИТ++ 2017, HighLoad Junior
Зал Сингапур, 6 июня, 11:00


Вы поставили HTTP-акселератор перед вашим web-сервером для ускорения отдачи контента, но запросы пользователей по-прежнему отдаются с большой задержкой, а ресурсы сервера кажутся незагруженными. А, может, после того, как поставили
web-акселератор, web-приложение сломалось, да еще и так, что проблема воспроизводится редко, хуже того, о ней могут знать ваши пользователи, но не вы.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Проксирование HTTP-запросов web-акселератором / Александр Крижановский (Tempesta Technologies)

  1. 1. HTTP Requests Proxying Alexander Krizhanovsky Tempesta Technologies, Inc.
  2. 2. Who am I? CEO & CTO at Tempesta Technologies Developing Tempesta FW – open source Linux Application Delivery Controller (ADC) Custom software development in: ● high performance network traffic processing e.g. WAF mentioned in Gartner magic quadrant ● Databases e.g. MariaDB SQL System Versioning ng-asof
  3. 3. HTTP requests proxying Load balancing Backend connections management SSL/TLS termination Protocol downgrade
  4. 4. HTTP proxying issues How many server connections? What to do if an upstream resets a connection? How to manage HTTP message queues? Is it safe to resend a request to another upstream? TLS is expensive Sockets proxying is expensive HTTP multiplexing - HTTP/2?
  5. 5. Server connections New connections or persistent connections HTTP keep-alive connections Keep-Alive: timeout=5, max=10000 Reestablish closed KA connection New connections if all are busy Tempesta FW: connections provisioning for DDoS and flash crowds
  6. 6. How many connections? There is optimal connections number for particular Web server and hardware Large connections number: Larger latency Smaller RPS
  7. 7. Connection resets & server failure Default for Apache HTTPD & Nginx: reset each 100 requests Dirty hack for messy dynamics Server failure ● Connection reset or I/O timeout ● HTTP error responses ● Backend healthcheck • TCP connection • HTTP response code • HTTP response body
  8. 8. HTTP message pipelining Mostly unused by proxies Squid, Tempesta FW, Polipo Messages multiplexing Forwarding and reforwarding issues Security issues ● Breaking authentication ● HTTP Response splitting
  9. 9. Connection closing and Keep-Alive Client Connection: close + TCP active close (RFC 7230 3.3.3) no Content-Length no Transfer-Encoding: chunked Backend ● Connection: keep-alive ● Add Content-Length ● Now the message can be pipelined
  10. 10. HTTP Response Splitting attack (Web cache poisoning) /redir_lang.jsp?lang=foobar%0d%0aContent-Length:%200%0d%0a%0d%0a HTTP/1.1%20200%20OK%0d%0aContent-Type:%20text/html%0d%0a Content-Length:%2019%0d%0a%0d%0a<html>Shazam</html> HTTP/1.1 302 Moved Temporarily Date: Wed, 24 Dec 2003 15:26:41 GMT Location: Content-Length: 0 HTTP/1.1 200 OK Content-Type: text/html Content-Length: 19 <html>Shazam</html> Proxy must count number of requests and responses (later detection!) Validate allowed URI alphabet (also for injection attacks)
  11. 11. (Non-)indempotent requests RFC 7231 4.2.2: Nonidempotent request changes server state Idempotent (safe) methods: GET, HEAD, TRACE, OPTIONS ● Admin responsibility: GET /action?do=delete ● Can be POST (e.g. web-search) Let a user to specify idempotence of methods and resources: nonidempotent GET prefix “/action”
  12. 12. Pipeline queues Tempesta FW: switch off pipelining server_queue_size 1 Use a queue with NI-request only if all connections are busy
  13. 13. Resend HTTP requests? Tainted requests: server_forward_retries and server_forward_timeout Request-killers – RFC 7230 6.3.2: “a client MUST NOT pipeline immediately after connection establishment” ● Connection with re-forwarded requests is non-schedulable Nonidempotent requests aren’t reforwarded server_retry_nonidempotent Error messages keep order of responses Resending HTTP messages + TCP retransmission Copying is required… ...unless TCP send queue integreated into a proxy
  14. 14. Requests reforwarding (not for persistent sessions!)
  15. 15. SSL/TLS: no zero-copying (no sendfile()!) User-kernel space copying ● Copy network data to user space ● Encrypt/decrypt it ● Copy the data to kernel for transmission Kernel-mode TLS ● Facebook, RedHat: ● Netflix: ● DDoS: TLS handshake is still an issue ● Tempesta:
  16. 16. Buffering vs streaming Buffering ● Seems everyone by default ● Performance degradation on large messages ● 200 means OK, no incomplete response Streaming ● Tengine (patched Nginx) w/ proxy_request_buffering & fastcgi_request_buffering ● More performance, but 200 doesn't mean full response
  17. 17. HTTP messages adjustment Add/remove/update HTTP headers Build a new HTTP message
  18. 18. User space HTTP proxying 1. Receive request at CPU1 2. Copy request to user space 3. Update headers 4. Copy request to kernel space 5. Send the request from CPU2 3 data copies Access TCP control blocks and data buffers from different CPUs
  19. 19. HTTP messages adjustment: zero-copy Add/remove/update HTTP headers w/o copies skb and its head are allocated in the same page fragment or a compound page
  20. 20. Zero-copy sockets proxying Kernel space or user space TCP/IP stack Socket callbacks call TLS and HTTP processing Everything is processing in softirq (while the data is hot) No receive & accept queues No file descriptors Less locking
  21. 21. Tempesta FW: kernel HTTPS/TCP/IP stack Alternative to user space TCP/IP stacks HTTPS is built into TCP/IP stack L7 DDoS mitigation and simple WAF out of the box Very fast HTTP parser and strings processing using AVX2 TODO HTTP QoS for asymmetric DDoS mitigation DSL for multi-layer filter rules
  22. 22. Tempesta FW: zero-copy proxying Socket callbacks call TLS and HTTP processing Everything is processing in softirq (while the data is hot) No receive & accept queues No file descriptors Less locking Lock-free inter-CPU transport => faster socket reading => lower latency
  23. 23. HTTP/2 Pros ● Responses are sent in parallel and in any order (no head-of-line blocking) ● Compression Cons ● Zero copy techniques aren’t applicable => For client connections (slow network), not for LAN (fast network)
  24. 24. Thanks! Web-site: (Powered by Tempesta FW) Availability: Blog: E-mail: