This document summarizes a talk given at ApacheCon 2015 about replacing Squid with Apache Traffic Server (ATS) as the proxy server at Yahoo. It discusses the history of using Squid at Yahoo, limitations with Squid that led to considering ATS, key differences in configuration between the two, how features like caching, logging, and peering are implemented in each, and lessons learned from the migration process.
October 14 2009 New York Web Performance Group Session.
Rusty Conover is talking about his experience at InfoGears building Content Delivery Network (CDN) on top of Amazon EC2
October 14 2009 New York Web Performance Group Session.
Rusty Conover is talking about his experience at InfoGears building Content Delivery Network (CDN) on top of Amazon EC2
Load Balancing MySQL with HAProxy - SlidesSeveralnines
Agenda:
* What is HAProxy?
* SQL Load balancing for MySQL
* Failure detection using MySQL health checks
* High Availability with Keepalived and Virtual IP
* Use cases: MySQL Cluster, Galera Cluster and MySQL Replication
* Alternative methods: Database drivers with inbuilt cluster support, MySQL proxy, MaxScale, ProxySQL
How To Set Up SQL Load Balancing with HAProxy - SlidesSeveralnines
We continuously see great interest in MySQL load balancing and HAProxy, so we thought it was about time we organised a live webinar on the topic! Here is the replay of that webinar!
As most of you will know, database clusters and load balancing go hand in hand.
Once your data is distributed and replicated across multiple database nodes, a load balancing mechanism helps distribute database requests, and gives applications a single database endpoint to connect to.
Instance failures or maintenance operations like node additions/removals, reconfigurations or version upgrades can be masked behind a load balancer. This provides an efficient way of isolating changes in the database layer from the rest of the infrastructure.
In this webinar, we cover the concepts around the popular open-source HAProxy load balancer, and show you how to use it with your SQL-based database clusters. We also discuss HA strategies for HAProxy with Keepalived and Virtual IP.
Agenda:
* What is HAProxy?
* SQL Load balancing for MySQL
* Failure detection using MySQL health checks
* High Availability with Keepalived and Virtual IP
* Use cases: MySQL Cluster, Galera Cluster and MySQL Replication
* Alternative methods: Database drivers with inbuilt cluster support, MySQL proxy, MaxScale, ProxySQL
Joined by Rick Nelson, Technical Solutions architect from NGINX Server Density take you though the do's and don'ts of monitoring NGINX. Critical and non critical metrics to monitor, important alerts to configure and the best monitoring tools available.
Using HAProxy to Scale MySQL: growing from a single database server to a master-master with multiple slaves replication topology using HAProxy to maintain high-availability and ease maintenance tasks.
Scaling out on the cloud is easy. Especially, if you have a software provisioning system that helps you to deploy your environment wherever you want. This session will give you an overview of the fantastic new features of HAProxy V 1.5, and how you can integrate it into your environment to build a high available environment, using open source software. Starting with a single-webserver + mysql setup provisioned via chef, we will deploy an HA Proxy Cluster in front and scale out your nginx and mysql database backend.
Learn how to load balance your applications following best practices with NGINX and NGINX Plus.
Join this webinar to learn:
- How to configure basic HTTP load balancing features
- The essential elements of load balancing: session persistence, health checks, and SSL termination
- How to load balance MySQL, DNS, and other common TCP/UDP applications
- How to have NGINX Plus automatically discover new service instances in an auto-scaling or microservices environment
A brief overview of caching mechanisms in a web application. Taking a look at the different layers of caching and how to utilize them in a PHP code base. We also compare Redis and MemCached discussing their advantages and disadvantages.
Load Balancing MySQL with HAProxy - SlidesSeveralnines
Agenda:
* What is HAProxy?
* SQL Load balancing for MySQL
* Failure detection using MySQL health checks
* High Availability with Keepalived and Virtual IP
* Use cases: MySQL Cluster, Galera Cluster and MySQL Replication
* Alternative methods: Database drivers with inbuilt cluster support, MySQL proxy, MaxScale, ProxySQL
How To Set Up SQL Load Balancing with HAProxy - SlidesSeveralnines
We continuously see great interest in MySQL load balancing and HAProxy, so we thought it was about time we organised a live webinar on the topic! Here is the replay of that webinar!
As most of you will know, database clusters and load balancing go hand in hand.
Once your data is distributed and replicated across multiple database nodes, a load balancing mechanism helps distribute database requests, and gives applications a single database endpoint to connect to.
Instance failures or maintenance operations like node additions/removals, reconfigurations or version upgrades can be masked behind a load balancer. This provides an efficient way of isolating changes in the database layer from the rest of the infrastructure.
In this webinar, we cover the concepts around the popular open-source HAProxy load balancer, and show you how to use it with your SQL-based database clusters. We also discuss HA strategies for HAProxy with Keepalived and Virtual IP.
Agenda:
* What is HAProxy?
* SQL Load balancing for MySQL
* Failure detection using MySQL health checks
* High Availability with Keepalived and Virtual IP
* Use cases: MySQL Cluster, Galera Cluster and MySQL Replication
* Alternative methods: Database drivers with inbuilt cluster support, MySQL proxy, MaxScale, ProxySQL
Joined by Rick Nelson, Technical Solutions architect from NGINX Server Density take you though the do's and don'ts of monitoring NGINX. Critical and non critical metrics to monitor, important alerts to configure and the best monitoring tools available.
Using HAProxy to Scale MySQL: growing from a single database server to a master-master with multiple slaves replication topology using HAProxy to maintain high-availability and ease maintenance tasks.
Scaling out on the cloud is easy. Especially, if you have a software provisioning system that helps you to deploy your environment wherever you want. This session will give you an overview of the fantastic new features of HAProxy V 1.5, and how you can integrate it into your environment to build a high available environment, using open source software. Starting with a single-webserver + mysql setup provisioned via chef, we will deploy an HA Proxy Cluster in front and scale out your nginx and mysql database backend.
Learn how to load balance your applications following best practices with NGINX and NGINX Plus.
Join this webinar to learn:
- How to configure basic HTTP load balancing features
- The essential elements of load balancing: session persistence, health checks, and SSL termination
- How to load balance MySQL, DNS, and other common TCP/UDP applications
- How to have NGINX Plus automatically discover new service instances in an auto-scaling or microservices environment
A brief overview of caching mechanisms in a web application. Taking a look at the different layers of caching and how to utilize them in a PHP code base. We also compare Redis and MemCached discussing their advantages and disadvantages.
Linux HTTPS/TCP/IP Stack for the Fast and Secure WebAll Things Open
Presented at All Things Open 2018
Presented by Alexander Krizhanovsky with Tempesta Technologies INC
10/23/18 - 2:00 PM - Networking/Infrastructure Track
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTanel Poder
Troubleshooting Complex Oracle Performance Problems hacking session & presentation by Tanel Poder.
This presentation is about a complex performance issue where the initial symptoms pointed somewhere else than the root cause. Only when systematically following through the troubleshooting drilldown method, we get to the root cause of the problem. This session aims to help you understand (and reason about) the Oracle’s multi-process & multi-layer system behavior, preparing you for independent troubleshooting of such complex performance issues in the future.
Video recordings of this presentation are in my YouTube channel:
1) Hacking Session: https://www.youtube.com/watch?v=INQewGJMdCI
2) Presentation: https://www.youtube.com/watch?v=aaHZ8A8Ygdg
Tanel's blog and training information: https://blog.tanelpoder.com/seminar
Practical non blocking microservices in java 8Michal Balinski
How to write application in Java 8 that do not waste resources and which can maximize effective utilization of CPU/RAM. Comparison of blocking and non-blocking approach for I/O and application services. Based on microservices implementing simple business logic in security/cryptography/payments domain. Demonstration of following aspects:
* NIO at all edges of application
* popular libraries that support NIO
* single instance scalability
* performance metrics (incl. throughput and latency)
* resources utilization
* code readability with CompletableFuture
* application maintenance and debugging
All above based on our experiences gathered during development of software platforms at Oberthur Technologies R&D Poland.
В докладе рассказывается о расширении для стека протоколов TCP/IP в ОС Linux, которое необходимо для того, чтобы HTTPS работал в том же стеке, что TCP и IP. DDoS-атаки такого типа как HTTP-флуд на уровне приложений, как правило, подавляются HTTP-акселераторами или балансировщиками нагрузки HTTP. Однако интерфейс сокетов Linux, используемый программным обеспечением, не дает той продуктивности, которая необходима при предельных нагрузках, вызванных DDoS-атаками. HTTP-серверы на базе стеков TCP/IP в пространстве пользователя становятся популярными в связи с увеличением их эффективности, но стеки TCP/IP представляют собой масштабный и сложный код, поэтому неблагоразумно реализовывать и исполнять его дважды — в пространстве пользователя и пространстве ядра. Стек TCP/IP в пространстве ядра хорошо интегрирован со многими мощными инструментами, например IPTables, IPVS, tc, tcpdump, которые недоступны для стека TCP/IP в пространстве пользователя или требуют сложных интерфейсов. Докладчик представит решение Tempesta FW, которое передает обработку HTTPS ядру. HTTPS встроен в стек TCP/IP Linux. Исполняя функцию межсетевого экрана HTTP, Tempesta FW устанавливает набор ограничений по скорости передачи и набор эвристических правил для защиты от таких атак как HTTPS-флуд и Slow HTTP.
These slides show how to reduce latency on websites and reduce bandwidth for improved user experience.
Covering network, compression, caching, etags, application optimisation, sphinxsearch, memcache, db optimisation
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTanel Poder
From Tanel Poder's Troubleshooting Complex Performance Issues series - an example of Oracle SEG$ internal segment contention due to some direct path insert activity.
Caching and tuning fun for high scalabilityWim Godden
Caching has been a 'hot' topic for a few years. But caching takes more than merely taking data and putting it in a cache : the right caching techniques can improve performance and reduce load significantly. But we'll also look at some major pitfalls, showing that caching the wrong way can bring down your site. If you're looking for a clear explanation about various caching techniques and tools like Memcached, Nginx and Varnish, as well as ways to deploy them in an efficient way, this talk is for you.
We are using Elasticsearch to power the search feature of our public frontend, serving 10k queries per hour across 8 markets in SEA.
Here we are sharing our experiences of running Elasticsearch on Kubernetes, presenting our general setup, configuration tweaks and possible pitfalls.
Как Web-акселератор акселерирует ваш сайт / Александр Крижановский (Tempesta ...Ontico
В докладе я расскажу, что такое Web-акселератор, он же reverse proxy и он же - фронтенд. Как следует из названия, он ускоряет сайт. Но за счет чего он это делает? Какие они, вообще, бывают? Что они умеют, а что нет? В чем особенности каждого из решений? И, вообще, постараюсь рассказать о них вглубь и вширь.
Еще я расскажу про еще один Open Source Web-акселератор - Tempesta FW. Уникальность проекта в том, что это гибрид Web-акселератора и файервола, разрабатываемый специально для обработки и фильтрации больших объемов HTTP трафика. Основные сценарии использования системы — это защита от DDoS прикладного уровня и просто доставка больших объемов HTTP трафика малыми затратами на оборудование.
- Что такое Web-акселератор, зачем он был придуман и как понять когда он нужен;
- Типичный функционал reverse proxy, его отличия от Web-сервера;
- Упомянем про SSL акселераторы;
- Заглянем вглубь HTTP, и как он управляет кэшированием и проксированием, что может быть закэшированно, а что - нет;
- Мы сравним наиболее популярные акселераторы (Nginx, Varnish, Apache Traffic Server, Apache HTTPD, Squid) по фичам и внутренностям;
- Зачем нужен еще один Web-акселератор Tempesta FW, и в чем его отличие от других акселераторов.
SecZone 2011 - Cali, Colombia
(29th Nov. 2011)
SAP (in)security:
Scrubbing SAP clean with SOAP
------
Note
------
This is a slightly updated version of my Hashdays 2011 talk.
----------
Abstract:
----------
At the heart of any large enterprise, lies a platform misunderstood and feared by all but the bravest systems administrators. Home to a wealth of information, and key to infinite wisdom. This platform is SAP. For years this system has been amongst the many "red pen" items on penetration tests and audits alike... but no more! We will no longer accept the cries of "Business critical, out-of-scope". The time for SAP has come, the cross-hairs of attackers are firmly focused on the soft underbelly that is ERM, and it's our duty to follow suit. Join me as we take the first steps into exploring SAP, extracting information and popping shells. Leave your Nessus license at the door! It's time to scrub this SAP system clean with SOAP!
----------
4. Squid in Yahoo
The year is 2006
● Dawn of SOA/Web Service in Yahoo!
● Squid improves performance through
caching
○ Other benefits - routing & ACL
5. Squid in Yahoo
● Mark Nottingham was the
Champion of Squid in
Yahoo!
● Collapsed Forwarding added
to Squid 2.6
● SWR/SIE added to Squid
2.7
https://www.mnot.net/personal/mn.jpg
6. Squid in Yahoo
● Squid 3.0 - Rewrite of Squid in C++
○ ESI
○ ICAP
● Squid 3.2 - multiple worker support
● Backward Incompatibilities
○ No Collapse Forwarding till 3.5+
○ No SIE till 3.2+
○ Still no SWR
○ BLOCKERS!!!
7. ATS in Yahoo
Inktomi
● TS 2.0 - 1998, 3.0 - 1999, 4.0 - 2000
● Customers - AOL, @Home
● e.g. - Transcoding images to smaller sizes
for AOL dialup users
8. ATS in Yahoo
YTS
● Inktomi Acquired by Yahoo - late 2002 /
early 2003
● Renamed to YTS - Efforts resumed around
late 2005
● By Feb 2010, served 30 billion objects, 400
terabytes a day for Yahoo
9. ATS in Yahoo
ASF
● Preparation - 700K lines of code change, 9
Months
● Apache Incubator in July 2009
● TLP in April 21, 2010
10. Cost of maintenance
Unresolved Blockers to Upgrade
Performance Limitation in 2.7
Why?
http://www.jillianney.com/wp-content/uploads/2012/01/why.jpg
12. Details - Configuration
ATS - records.config
CONFIG proxy.config.http.server_ports STRING 3128
# ATS requires disk cache to be set up through storage.config as well
CONFIG proxy.config.cache.ram_cache.size INT 2147483648
CONFIG proxy.config.http.negative_caching_enabled INT 1
CONFIG proxy.config.http.negative_caching_lifetime INT 15
13. Details - Configuration
ATS - records.config (cont)
CONFIG proxy.config.http.connect_attempts_timeout INT 15
CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 15
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 30
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 30
CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 30
CONFIG proxy.config.http.transaction_active_timeout_in INT 30
CONFIG proxy.config.http.transaction_active_timeout_out INT 30
CONFIG proxy.config.http.accept_no_activity_timeout INT 12
21. Details - SWR/SIE/Collapsed
Forwarding
ATS
● Collapse Forwarding - check out wiki page
● experimental stale_while_revalidate plugin
● TS-1463
○ once the fetch of the object is initiated, the original
object is not allowed to be served from cache
● TS-1996
○ Deprecated API - TSHttpTxnNewCacheLookupDo()
22. Details - Fixing the Plugin (SWR)
ATS Origin
1. client req in swr
2. serve stale
3. async req (?swr=1)
4. valid async resp
5. post async content to
original URL to update
the cache
23. Details - Fixing the Plugin (SIE)
ATS Origin
1. client req in sie
4. intercept return with async resp
2. async req (?swr=1)
3. valid async resp
ATS Origin
1. client req in sie
4. serve stale
2. async req (?swr=1)
3. 5xx for async resp
24. Details - ACL control (Squid)
Based on src, dst, time, regex, schema etc.
Squid.conf
# Example 1 (give access only to certain clients)
acl myclients src 172.16.5.0/24
http_access allow myclients
http_access deny all
# Example 2 (disable cache for responses from a domain)
acl someserver dstdomain .someserver.com
cache deny someserver
25. Details - ACL control (ATS)
remap.config
ip_allow.config (Similar to Example 1)
cache.config (Similar to Example 2)
map http://www.x.com/ http://server.hoster.com/
src_ip=123.12.3.000-123.12.3.123 action=ip_allow
dest_domain=mydomain.com action=never-cache
26. Details - Extensibility (Squid)
● Helper program for ACL, URL Manipulation,
DNS lookup
○ Rigid and limited
○ e.g. controlling ACL in squid.conf
external_acl_type yca_helper cache=5000 concurrency=1000 children=2 grace=1
%SRC %{App-Auth} %DATA /usr/local/libexec/squid/yca_acl.pl
acl yca external yca_helper
acl yca_appids ext_user REQUIRED
deny_info YCA_AUTH_REQ yca
deny_info YCA_WRONG_APPID yca_appids
30. Details - Peering (ATS)
Type
1. Sibling (ICP*)
2. Parent
* Our Squid ICP peering use case (Example 1) is not needed when we deploy with
hierarchical caching + consistent hashing
31. Details - Peering (ATS)
remap.config (Similar to Squid Example 2)
parent.config (Similar to Squid Example 3)
dest_domain=. method=get parent="p1.x.com:8080; p2.y.com:8080"
round_robin=true
dest_domain=. method=get parent="p1.x.com:8080|1.0; p2.y.com:8080|2.0"
round_robin=consistent_hash
map http://www.x.com/ http://server1.com/
32. Use Cases - Forward Proxy
https://docs.trafficserver.apache.org/en/latest/_images/cache_miss.jpg
33. squid.conf
Use Cases - Forward Proxy (Squid)
http_port 80 vhost
# protecting proxy by only allowing clients to connect to port 80
acl Safe_ports port 80
http_access deny !Safe_ports
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
http_access allow localhost
http_access allow localnet
34. Use Cases - Forward Proxy (ATS)
records.config
remap.config
CONFIG proxy.config.reverse_proxy.enabled INT 0
CONFIG proxy.config.http.server_ports 80
# for security purpose require remap (Optional)
CONFIG proxy.config.url_remap.remap_required INT 1
# map all origin servers for which we need forward proxy
map http://server1.com/ http://server1.com/
35. Use Cases - Reverse Proxy
https://docs.trafficserver.apache.org/en/latest/_images/revproxy.jpg
37. Use Cases - Reverse Proxy (ATS)
1. records.config
2. remap.config
CONFIG proxy.config.http.server_ports 80
CONFIG proxy.config.reverse_proxy.enabled INT 1
CONFIG proxy.config.url_remap.remap_required INT 1
map http://www.server1.com/ http://www.endpoint.com/
map http://www.server2.com/ http://www.endpoint.com/
38. Learnings
● Managing an open source project
○ Backward compatibility
● Squid configuration
○ config can be confusing when they exist in same file
○ allow/deny pattern confusing
● ATS configuration
○ ATS configuration complicated/disjoint, but clean
○ ATS plugins more flexible
● Improved Performance
○ CPU/Latency/RPS
● Migration
○ (Automated) Testing is important
39. Automated Testing
● TSQA
○ integration/functional testing framework
○ Not just for ATS, can be used for other proxy server
(e.g. squid)
○ To be used heavily during migration