2012-10-25
                         Caché acelerador de contenido

                             Derechos reservados © 2012 Sandino Araico Sánchez
                             <sandino@1101.mx>

                             Se permite ilimitadamente el uso, copia, redistribución con o sin
                             modificaciones siempre y cuando se mantenga el aviso de
                             derecho de autor y se anoten al final de la presentación todas las
                             modificaciones que se llevan a cabo conservando la historia de
                             las modificaciones que hagan las demás personas e indicando la
                             fecha de cada modificación y el nombre de la persona que la
                             llevó a cabo.




Thursday, October 25, 2012
Netcraft web server survey




            Fuente: netcraft.com

Thursday, October 25, 2012
Servidores de páginas




                             Apache

                             Nginx

                             Cherokee

                             Lighttpd

                             Tomcat




Thursday, October 25, 2012
Aceleración
                                        Primer ejemplo




                             Nginx

                             Cherokee

                             php-fcgi

                             php-fpm




Thursday, October 25, 2012
Aceleración
                                                          Primer ejemplo
                  server {
                        listen         192.168.182.170:80;
                        server_name        hhh.example.com example.com;
                        access_log       /var/log/nginx/example/hhh.access.log main;
                        error_log        /var/log/nginx/example/hhh.error.log;
                        root    /var/www/example/hhh/public_html;
                        location / {
                               index index.php index.html;
                               if (!-e $request_filename) {
                                       rewrite ^(.*)$ /index.php?q=$1 last;
                                       break;
                               }
                        }
                        location ^~ ^/nota/ {
                               rewrite ^(.*)$ /index.php?q=$1 last;
                               break;
                        }



Thursday, October 25, 2012
Aceleración
                                                    Primer ejemplo
                        location ~* ^.+.(jpg|jpeg|gif|png|ico|js|css)$ {
                               #access_log off;
                               expires 30d;
                        }
                        location ~ "^/([0-9]+)$" {
                               rewrite ^/([0-9]+)$ /index.php?q=/node/$1 last;
                        }
                        location ~ .php$ {
                               fastcgi_pass     127.0.218.170:8000;
                               fastcgi_index index.php;
                               fastcgi_read_timeout       300;
                               fastcgi_param SCRIPT_FILENAME /var/www/example/
                  hhh/public_html$fastcgi_script_name;
                               include /etc/nginx/fastcgi.conf;
                        }
                  }




Thursday, October 25, 2012
Aceleración
                                                   Segundo ejemplo




                             Nginx

                             Cherokee

                             Apache en el puerto 8080

                             PHP en el Apache




Thursday, October 25, 2012
Aceleración
                                                          Segundo ejemplo

                         upstream indio {
                                server 127.0.0.1:8880;
                         }
                         upstream hosting {
                                server 192.168.182.226:80;
                         }
                  server {
                         listen       200.57.34.35:80 default_server ;
                         listen       [2607:f0d0:1002:f2:200:57:34:35]:80
                  default_server ;
                         server_name example.com www.example.com ;
                         root          /var/www/example.com/www/web ;
                         access_log      /var/log/nginx/example.access.log main;
                         access_log      /var/log/nginx/everyone.log main;
                         error_log       /var/log/nginx/example.error.log ;
                         #charset         ISO-8859-1;



Thursday, October 25, 2012
Aceleración
                                                      Segundo ejemplo


                  location / {
                         index index.php index.html;
                         proxy_set_header X-Real-IP $remote_addr;
                         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                         proxy_set_header X-FORWARDED_PROTO https;
                         proxy_set_header Host $http_host;
                         proxy_redirect off;
                         if (!-e $request_filename) {
                                 proxy_pass http://hosting;
                                 break;
                         }

                  }




Thursday, October 25, 2012
Aceleración
                                                          Segundo ejemplo


                     location ~ .php {
                            proxy_set_header X-Real-IP $remote_addr;
                            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                            proxy_set_header X-FORWARDED_PROTO https;
                            proxy_set_header Host $http_host;
                            proxy_redirect off;

                             proxy_pass http://hosting;
                             break;
                     }
        }




Thursday, October 25, 2012
Servidores de caché




                             Squid

                             Varnish

                             Nginx

                             Cherokee




Thursday, October 25, 2012
Caché
                                                    Primer ejemplo


             user           nginx;
             worker_processes 1;
             worker_rlimit_nofile 250000;
             events {
                worker_connections 65536;
             }
             ! proxy_cache_path /data/nginx/cache levels=1:2    keys_zone=STATIC:16m
                                             inactive=24h max_size=15g;
             ! proxy_temp_path ! /data/nginx/tmp ;! !
             ! open_file_cache max=10000 inactive=200s;
             ! open_file_cache_valid    300s;
             ! open_file_cache_min_uses 5;
             ! open_file_cache_errors on;




Thursday, October 25, 2012
Caché
                                                                  Primer ejemplo
                             log_format accounting
                                   '$msec $time_local '
                                   '| $remote_addr '
                                   '| $status '
                                   '| $bytes_sent '
                                   '| $request_time '
                                   '| $pipe '
                                   '| $upstream_cache_status '
                                   '| $upstream_status '
                                   '| $upstream_response_time '
                                   '| $host '
                                   '| $uri '
                                   '| $query_string ' ;




Thursday, October 25, 2012
Caché
                                                      Primer ejemplo

      1351186862.566 25/Oct/2012:12:41:02 -0500 | 69.93.147.213 | 200 | 569 | 0.000 | . |
      HIT | - | - | gpl.la | / | -
      1351186867.326 25/Oct/2012:12:41:07 -0500 | 69.93.147.213 | 301 | 425 | 0.000 | . | - |
      - | - | gpl.la | /gran/peda/linuxera/ | -
      1351186869.260 25/Oct/2012:12:41:09 -0500 | 69.93.147.213 | 200 | 10476 | 0.000 | . |
      HIT | - | - | gpl.la | /gran/peda/linuxera/main.php | -
      1351186869.380 25/Oct/2012:12:41:09 -0500 | 69.93.147.213 | 404 | 464 | 0.120 | . |
      MISS | 404 | 0.054 | gpl.la | /favicon.ico | -
      1351186872.651 25/Oct/2012:12:41:12 -0500 | 69.93.147.213 | 200 | 3314 | 0.000 | . |
      HIT | - | - | gpl.la | /gran/peda/linuxera/modules/core/data/gallery.css | -
      1351186873.198 25/Oct/2012:12:41:13 -0500 | 201.158.247.57 | 301 | 407 | 0.000 | . | -
      | - | - | enli.org.mx | / | -
      1351186876.225 25/Oct/2012:12:41:16 -0500 | 69.93.147.213 | 200 | 838 | 0.000 | . |
      HIT | - | - | gpl.la | /gran/peda/linuxera/modules/albumselect/dtree.css | -




Thursday, October 25, 2012
Caché
                                                           Primer ejemplo



              upstream local ! !   { server 127.0.0.1:8080; }

              upstream mendozaaaa ! server 200.57.34.35:80;
                                    {
                                server 200.57.34.45:80; } #c.w.h
              upstream mozillamexico ! { server 200.57.34.35:80;
                                server 200.57.34.45:80; } #c.w.h
              upstream enli! ! ! { server 200.57.34.35:80;
                                server 200.57.34.45:80; } #c.w.h




Thursday, October 25, 2012
Caché
                                                            Primer ejemplo
                     server {
                     ! listen 80;
                     ! server_name ! www.enli.org.mx ;
                     ! access_log      /var/log/nginx/accounting.log accounting;
                     ! error_log       /var/log/nginx/error.log;
                     ! location / {
                     ! ! include /etc/nginx/proxy-common.conf ;
                     ! ! proxy_pass http:/   /enli;
                     ! ! break;
                     ! }
                     ! location = /programa.html {
                     ! ! include /etc/nginx/proxy-micro.conf ;
                     ! ! expires 60;
                     ! ! proxy_pass http:/   /enli;
                     ! ! break;
                     ! }




Thursday, October 25, 2012
Caché
                                                         Primer ejemplo

                 !     location ~ .php {
                 !     ! include /etc/nginx/proxy-micro.conf ;
                 !     ! proxy_pass http:/   /enli;
                 !     ! expires 60;
                 !     ! break;
                 !     }
                 !     location ~* .(jpg|jpeg|gif|png|ico|css|js|ico)$ {
                 !     ! include /etc/nginx/proxy-common.conf ;
                 !     ! proxy_ignore_headers "Expires" "Cache-Control" "Set-Cookie" ;
                 !     ! proxy_pass http:/   /enli;
                 !     ! break;
                 !     }
                 !     include /etc/nginx/common-attack.conf;
                 }




Thursday, October 25, 2012
Caché
                                                              proxy-common

              !     !        proxy_set_header X-Real-IP $remote_addr;
              !     !        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
              !     !        proxy_set_header Host $http_host;
              !     !        proxy_redirect off;
                                proxy_cache           STATIC;
                                proxy_cache_valid      1d;
                                proxy_cache_use_stale error timeout invalid_header updating
                                                   http_500 http_502 http_503 http_504;
              !     !        proxy_buffering on;
              !     !        proxy_buffer_size 8k;
              !     !        proxy_buffers 8 8k;
              !     !        proxy_busy_buffers_size 16k;




Thursday, October 25, 2012
Caché
                                                               proxy-micro

              !     !        proxy_set_header X-Real-IP $remote_addr;
              !     !        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
              !     !        proxy_set_header Host $http_host;
              !     !        proxy_redirect off;

                                proxy_cache           STATIC;
                                proxy_cache_valid      15;
                                proxy_cache_valid      200 5;
                                proxy_cache_use_stale error timeout invalid_header updating
                                                   http_500 http_502 http_503 http_504;
              !     !        proxy_buffering on;
              !     !        proxy_buffer_size 8k;
              !     !        proxy_buffers 8 8k;
              !     !        proxy_busy_buffers_size 16k;




Thursday, October 25, 2012
Caché
                                                   CDN por DNS


              ;$TTL 86400! 1 day
                          ;
              $TTL 3600! ; 1 hour
              $ORIGIN enli.org.mx.

              ;@!! A! 200.57.34.35!; hosting
              ;www!CNAME! @! ! ;
              @! ! A! 50.30.46.179 ; raicilla
              www! ! ! CNAME! a.cdn.1-1-0-1.net.
              origen!! CNAME! b.w.h.srvr.mx.




Thursday, October 25, 2012
Caché
                                                        Primer ejemplo


                             server {
                             ! listen 80;
                             ! server_name ! enli.org.mx ;
                             ! access_log     /var/log/nginx/accounting.log accounting;
                             ! location / {
                             ! ! rewrite ^/(.*)$ http:/   /www.$server_name/$1 permanent;
                             ! }
                             ! include /etc/nginx/common-attack.conf;
                             }




Thursday, October 25, 2012
Caché
                                                     CDN por DNS Round Robin


                             ;$TTL 3600! ; 1 hour
                             $TTL 300! 5 min
                                      ;
                             $ORIGIN cdn.1-1-0-1.net.
                             ;A
                             ;a! ! A! 50.30.46.179 ; raicilla
                             ;a! ! A! 99.90.100.1 ; Nodo A
                             a! ! A! 99.90.100.2 ; Nodo D
                             ;a! ! A! 200.57.48.12 ; Nodo C
                             ;a! ! A! 200.57.48.13 ; Nodo E
                             a! ! A! 74.86.74.194 ; Nodo F




Thursday, October 25, 2012
Caché
                                                       Provisionamiento de nodos



                             for i in `cat hosts | sed “s/#.*$/ ` ; do
                                                               /”
                                ssh $i < scripts/install-key
                                ./scripts/install-node $i
                             done

                             ./sync-nginx-all




Thursday, October 25, 2012
Caché
                                           sync-nginx-all


      RSYNC_PARAMS='-vacH --progress '

      for i in `cat ./hosts | sed 's/#.*$/ ; do
                                          /'`
      ! echo "===> rsync to: $i"
      ! rsync $RSYNC_PARAMS ./etc/nginx/*conf root@$i:/etc/nginx/
      ! rsync $RSYNC_PARAMS --delete ./etc/nginx/vhosts/ root@$i:/etc/nginx/vhosts/
      ! ssh root@$i service nginx reload
      done




Thursday, October 25, 2012
Caché
                                            common-attack.conf
      #Apache .htaccess
      location ~ /.ht {
      ! deny all;
      }
      ## Only allow these request methods ##
          if ($request_method !~ ^(GET|HEAD|POST)$ ) {
              return 444;
          }
      ## Do not accept DELETE, SEARCH and other methods ##
      #Directory trasversal ../../../../../
      ! if ($query_string ~ ../../../../../ ) {
      ! ! return 404;
      ! }
      location ~ url(data:image/png;base64, {
      ! return 404;
      }




Thursday, October 25, 2012
Referencias




                             http://wiki.nginx.org/HttpProxyModule


 Sandino Araico Sánchez
 <sandino@1101.mx>
 @KBrown                                                    #mendozaaaa
Thursday, October 25, 2012

Caché acelerador de contenido

  • 1.
    2012-10-25 Caché acelerador de contenido Derechos reservados © 2012 Sandino Araico Sánchez <sandino@1101.mx> Se permite ilimitadamente el uso, copia, redistribución con o sin modificaciones siempre y cuando se mantenga el aviso de derecho de autor y se anoten al final de la presentación todas las modificaciones que se llevan a cabo conservando la historia de las modificaciones que hagan las demás personas e indicando la fecha de cada modificación y el nombre de la persona que la llevó a cabo. Thursday, October 25, 2012
  • 2.
    Netcraft web serversurvey Fuente: netcraft.com Thursday, October 25, 2012
  • 3.
    Servidores de páginas Apache Nginx Cherokee Lighttpd Tomcat Thursday, October 25, 2012
  • 4.
    Aceleración Primer ejemplo Nginx Cherokee php-fcgi php-fpm Thursday, October 25, 2012
  • 5.
    Aceleración Primer ejemplo server { listen 192.168.182.170:80; server_name hhh.example.com example.com; access_log /var/log/nginx/example/hhh.access.log main; error_log /var/log/nginx/example/hhh.error.log; root /var/www/example/hhh/public_html; location / { index index.php index.html; if (!-e $request_filename) { rewrite ^(.*)$ /index.php?q=$1 last; break; } } location ^~ ^/nota/ { rewrite ^(.*)$ /index.php?q=$1 last; break; } Thursday, October 25, 2012
  • 6.
    Aceleración Primer ejemplo location ~* ^.+.(jpg|jpeg|gif|png|ico|js|css)$ { #access_log off; expires 30d; } location ~ "^/([0-9]+)$" { rewrite ^/([0-9]+)$ /index.php?q=/node/$1 last; } location ~ .php$ { fastcgi_pass 127.0.218.170:8000; fastcgi_index index.php; fastcgi_read_timeout 300; fastcgi_param SCRIPT_FILENAME /var/www/example/ hhh/public_html$fastcgi_script_name; include /etc/nginx/fastcgi.conf; } } Thursday, October 25, 2012
  • 7.
    Aceleración Segundo ejemplo Nginx Cherokee Apache en el puerto 8080 PHP en el Apache Thursday, October 25, 2012
  • 8.
    Aceleración Segundo ejemplo upstream indio { server 127.0.0.1:8880; } upstream hosting { server 192.168.182.226:80; } server { listen 200.57.34.35:80 default_server ; listen [2607:f0d0:1002:f2:200:57:34:35]:80 default_server ; server_name example.com www.example.com ; root /var/www/example.com/www/web ; access_log /var/log/nginx/example.access.log main; access_log /var/log/nginx/everyone.log main; error_log /var/log/nginx/example.error.log ; #charset ISO-8859-1; Thursday, October 25, 2012
  • 9.
    Aceleración Segundo ejemplo location / { index index.php index.html; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-FORWARDED_PROTO https; proxy_set_header Host $http_host; proxy_redirect off; if (!-e $request_filename) { proxy_pass http://hosting; break; } } Thursday, October 25, 2012
  • 10.
    Aceleración Segundo ejemplo location ~ .php { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-FORWARDED_PROTO https; proxy_set_header Host $http_host; proxy_redirect off; proxy_pass http://hosting; break; } } Thursday, October 25, 2012
  • 11.
    Servidores de caché Squid Varnish Nginx Cherokee Thursday, October 25, 2012
  • 12.
    Caché Primer ejemplo user nginx; worker_processes 1; worker_rlimit_nofile 250000; events { worker_connections 65536; } ! proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=STATIC:16m inactive=24h max_size=15g; ! proxy_temp_path ! /data/nginx/tmp ;! ! ! open_file_cache max=10000 inactive=200s; ! open_file_cache_valid 300s; ! open_file_cache_min_uses 5; ! open_file_cache_errors on; Thursday, October 25, 2012
  • 13.
    Caché Primer ejemplo log_format accounting '$msec $time_local ' '| $remote_addr ' '| $status ' '| $bytes_sent ' '| $request_time ' '| $pipe ' '| $upstream_cache_status ' '| $upstream_status ' '| $upstream_response_time ' '| $host ' '| $uri ' '| $query_string ' ; Thursday, October 25, 2012
  • 14.
    Caché Primer ejemplo 1351186862.566 25/Oct/2012:12:41:02 -0500 | 69.93.147.213 | 200 | 569 | 0.000 | . | HIT | - | - | gpl.la | / | - 1351186867.326 25/Oct/2012:12:41:07 -0500 | 69.93.147.213 | 301 | 425 | 0.000 | . | - | - | - | gpl.la | /gran/peda/linuxera/ | - 1351186869.260 25/Oct/2012:12:41:09 -0500 | 69.93.147.213 | 200 | 10476 | 0.000 | . | HIT | - | - | gpl.la | /gran/peda/linuxera/main.php | - 1351186869.380 25/Oct/2012:12:41:09 -0500 | 69.93.147.213 | 404 | 464 | 0.120 | . | MISS | 404 | 0.054 | gpl.la | /favicon.ico | - 1351186872.651 25/Oct/2012:12:41:12 -0500 | 69.93.147.213 | 200 | 3314 | 0.000 | . | HIT | - | - | gpl.la | /gran/peda/linuxera/modules/core/data/gallery.css | - 1351186873.198 25/Oct/2012:12:41:13 -0500 | 201.158.247.57 | 301 | 407 | 0.000 | . | - | - | - | enli.org.mx | / | - 1351186876.225 25/Oct/2012:12:41:16 -0500 | 69.93.147.213 | 200 | 838 | 0.000 | . | HIT | - | - | gpl.la | /gran/peda/linuxera/modules/albumselect/dtree.css | - Thursday, October 25, 2012
  • 15.
    Caché Primer ejemplo upstream local ! ! { server 127.0.0.1:8080; } upstream mendozaaaa ! server 200.57.34.35:80; { server 200.57.34.45:80; } #c.w.h upstream mozillamexico ! { server 200.57.34.35:80; server 200.57.34.45:80; } #c.w.h upstream enli! ! ! { server 200.57.34.35:80; server 200.57.34.45:80; } #c.w.h Thursday, October 25, 2012
  • 16.
    Caché Primer ejemplo server { ! listen 80; ! server_name ! www.enli.org.mx ; ! access_log /var/log/nginx/accounting.log accounting; ! error_log /var/log/nginx/error.log; ! location / { ! ! include /etc/nginx/proxy-common.conf ; ! ! proxy_pass http:/ /enli; ! ! break; ! } ! location = /programa.html { ! ! include /etc/nginx/proxy-micro.conf ; ! ! expires 60; ! ! proxy_pass http:/ /enli; ! ! break; ! } Thursday, October 25, 2012
  • 17.
    Caché Primer ejemplo ! location ~ .php { ! ! include /etc/nginx/proxy-micro.conf ; ! ! proxy_pass http:/ /enli; ! ! expires 60; ! ! break; ! } ! location ~* .(jpg|jpeg|gif|png|ico|css|js|ico)$ { ! ! include /etc/nginx/proxy-common.conf ; ! ! proxy_ignore_headers "Expires" "Cache-Control" "Set-Cookie" ; ! ! proxy_pass http:/ /enli; ! ! break; ! } ! include /etc/nginx/common-attack.conf; } Thursday, October 25, 2012
  • 18.
    Caché proxy-common ! ! proxy_set_header X-Real-IP $remote_addr; ! ! proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; ! ! proxy_set_header Host $http_host; ! ! proxy_redirect off; proxy_cache STATIC; proxy_cache_valid 1d; proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; ! ! proxy_buffering on; ! ! proxy_buffer_size 8k; ! ! proxy_buffers 8 8k; ! ! proxy_busy_buffers_size 16k; Thursday, October 25, 2012
  • 19.
    Caché proxy-micro ! ! proxy_set_header X-Real-IP $remote_addr; ! ! proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; ! ! proxy_set_header Host $http_host; ! ! proxy_redirect off; proxy_cache STATIC; proxy_cache_valid 15; proxy_cache_valid 200 5; proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; ! ! proxy_buffering on; ! ! proxy_buffer_size 8k; ! ! proxy_buffers 8 8k; ! ! proxy_busy_buffers_size 16k; Thursday, October 25, 2012
  • 20.
    Caché CDN por DNS ;$TTL 86400! 1 day ; $TTL 3600! ; 1 hour $ORIGIN enli.org.mx. ;@!! A! 200.57.34.35!; hosting ;www!CNAME! @! ! ; @! ! A! 50.30.46.179 ; raicilla www! ! ! CNAME! a.cdn.1-1-0-1.net. origen!! CNAME! b.w.h.srvr.mx. Thursday, October 25, 2012
  • 21.
    Caché Primer ejemplo server { ! listen 80; ! server_name ! enli.org.mx ; ! access_log /var/log/nginx/accounting.log accounting; ! location / { ! ! rewrite ^/(.*)$ http:/ /www.$server_name/$1 permanent; ! } ! include /etc/nginx/common-attack.conf; } Thursday, October 25, 2012
  • 22.
    Caché CDN por DNS Round Robin ;$TTL 3600! ; 1 hour $TTL 300! 5 min ; $ORIGIN cdn.1-1-0-1.net. ;A ;a! ! A! 50.30.46.179 ; raicilla ;a! ! A! 99.90.100.1 ; Nodo A a! ! A! 99.90.100.2 ; Nodo D ;a! ! A! 200.57.48.12 ; Nodo C ;a! ! A! 200.57.48.13 ; Nodo E a! ! A! 74.86.74.194 ; Nodo F Thursday, October 25, 2012
  • 23.
    Caché Provisionamiento de nodos for i in `cat hosts | sed “s/#.*$/ ` ; do /” ssh $i < scripts/install-key ./scripts/install-node $i done ./sync-nginx-all Thursday, October 25, 2012
  • 24.
    Caché sync-nginx-all RSYNC_PARAMS='-vacH --progress ' for i in `cat ./hosts | sed 's/#.*$/ ; do /'` ! echo "===> rsync to: $i" ! rsync $RSYNC_PARAMS ./etc/nginx/*conf root@$i:/etc/nginx/ ! rsync $RSYNC_PARAMS --delete ./etc/nginx/vhosts/ root@$i:/etc/nginx/vhosts/ ! ssh root@$i service nginx reload done Thursday, October 25, 2012
  • 25.
    Caché common-attack.conf #Apache .htaccess location ~ /.ht { ! deny all; } ## Only allow these request methods ## if ($request_method !~ ^(GET|HEAD|POST)$ ) { return 444; } ## Do not accept DELETE, SEARCH and other methods ## #Directory trasversal ../../../../../ ! if ($query_string ~ ../../../../../ ) { ! ! return 404; ! } location ~ url(data:image/png;base64, { ! return 404; } Thursday, October 25, 2012
  • 26.
    Referencias http://wiki.nginx.org/HttpProxyModule Sandino Araico Sánchez <sandino@1101.mx> @KBrown #mendozaaaa Thursday, October 25, 2012