Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Extending Piwik At

How we deployed Piwik web analytics system to handle a huge amount of unpredicted traffic, adding some cloud and modern scalability techniques. files:

  • Login to see the comments

Extending Piwik At

  1. 1. Extending Piwik at Phase 1 – Collecting data Adding some cloud and modern scalability to a traditional LAMP stack leonardo lorieri, system architect, 'lorieri at', feb/2012
  2. 2. Why Piwik ?   - Open Source = flexible, understandable, free!   - Great interface   - Mobile app   - REST API   - Developers knows the market needs   - Efficient in small machines   - Lots of possible improvements   - Lots of improvements already in the roadmap   - Great and supportive community (Thank you all!)
  3. 3. Our Plan, goals and trade-offs   - Don't change original code       - reduces development and maintenance costs   - Count only visits and page views       - to be fast and focused ( even though you still can use the .js tracker,         it is easy to get lost in the UI's beauty and all its functionalities)   - Handle odd unexpected traffic peaks       - from tv announcements    - Count not only websites       - media delivery, internal searches, debugs   - At least 99% of accuracy   - Have numbers to compare with other analytics tools   - We've lost P3P for now
  4. 4. Our big problem - The TV Effect <ul><li>from Gaiser's presentation at </li></ul><ul><li>Traffic peak during a TV Show </li></ul>
  5. 5. Regular Piwik Setup based on Rodrigo Campos presentation   - Apache/Nginx   - Php   - Mysql
  6. 6. Bigger Piwik Setup based on Rodrigo Campos presentation   - Apache/Nginx   - Php   - MySql
  7. 7. Regular Php Scaling Piwik Setup based on Rodrigo Campos presentation   - Apache/Nginx   - Php   - MySql Replication       (slave for backup only,       piwik is not &quot;slave ready&quot;) Load balancer/Nginx
  8. 8. Two problems, one easy solution <ul><li>  Problem: Data Collection for the TV Effect </li></ul><ul><li>  Easy solution: make it asynchronous </li></ul><ul><li>  </li></ul><ul><li>  Problem: Data processing </li></ul><ul><li>  Hard solution: huge ($$$) servers and complex tunings </li></ul>
  9. 9. Asynchronous Piwik Setup based on Rodrigo Campos presentation   - Nginx   - NOT even Php   - MySql Master   - Apache+Php for Admin UI   - Archive cron Load balancer/Nginx   - MySql Slave   - Perl/Python worker to     process logs (manages user cookies) (user cookie) - accesses logs Visits REST API <img src=> request Admin/ Reports
  10. 10. Nginx (more details later) <ul><li>  - Small virtual machines can handle thousands requests per second </li></ul><ul><li>  - Visits divided in logs by virtual hosts </li></ul><ul><li>  - HttpUserIdModule </li></ul><ul><li>    - automatically creates and handles user id cookies </li></ul><ul><li>  - HttpLogModule </li></ul><ul><li>    - formats log as NSCA combined (logging cookies and referrers) </li></ul><ul><li>  - HttpEmptyGifModule </li></ul><ul><li>    - respond an empty gif  </li></ul><ul><li>  - HttpHeadersModule </li></ul><ul><li>    - expires -1; </li></ul><ul><li>  (all modules available in ubuntu's nginx-extras package) </li></ul><ul><li>  - Logrotate </li></ul><ul><li>    - unix tool to rotate logs </li></ul><ul><li>    - each 5 minutes for us </li></ul><ul><li>  </li></ul>
  11. 11. Log processing &quot;Worker&quot; (more details later) <ul><li>  - Copy and uncompress available logs </li></ul><ul><li>  - Format them as a REST API request </li></ul><ul><li>    - force date visit </li></ul><ul><li>    - force client ip </li></ul><ul><li>    - force idVisitor </li></ul><ul><li>    - force User Agent </li></ul><ul><li>    - User log's Referrer as URL </li></ul><ul><li>    - Use referrer as page tittle (useful to log multiple hostnames) </li></ul><ul><li>  - Send request to Piwik server in parallel </li></ul><ul><li>    - we are using 270 concurrent requests that </li></ul><ul><li>      makes 1300 requests per second </li></ul>
  12. 12. Mysql Master (more details later) <ul><li>  </li></ul><ul><li>  - The only machine that has to be huge, saving money  </li></ul><ul><li>  - Piwik admin and reports interface is here </li></ul><ul><li>    - could be somewhere else, but the machine is huge anyway </li></ul><ul><li>  - Mysql tuning  </li></ul><ul><li>  - Raid tuning </li></ul><ul><li>  - Linux networking tuning  </li></ul><ul><li>    - same in all machines, to handle too many tcp concurrent connections </li></ul><ul><li>   </li></ul>
  13. 13. Php tuning (more details later) <ul><li>  - Max execution time </li></ul><ul><li>  - Max INPUT TIME   <--- bug in php reports it as max execution time </li></ul><ul><li>  - Max memory limit </li></ul><ul><li>  - Apc </li></ul><ul><li>  - Apc shm_size </li></ul><ul><li>  </li></ul><ul><li>Some problems: </li></ul><ul><li>  - Consider Apache, it is slower than nginx, but more stable and </li></ul><ul><li>    much easier to debug, easier to control concurrency </li></ul><ul><li>  - MYSQLi is more stable and has better debugging than Mysql_pdo </li></ul><ul><li>  - mod_php is more stable and easier to debug than fastcgi </li></ul>
  14. 14. Piwik tuning <ul><li>Follow the rules: </li></ul><ul><li>  - disable unused plugins </li></ul><ul><li>  - since the cookies comes from nginx, you can set in the config.ini: </li></ul><ul><li>[Tracker] </li></ul><ul><li>trust_visitors_cookies=1 </li></ul><ul><li>  </li></ul>
  15. 15. Handling TV Effect <ul><li>nginx requests/second > maximum requests on peaks of traffic </li></ul><ul><li>  - autoscaling guarantees it </li></ul><ul><li>  - autoscaling provides scheduled capacity changes </li></ul><ul><li>total requests in a day  <  (rest api requests/second) * (all seconds in a day) </li></ul><ul><li>  - even though the peaks requests vary in 1000% in a short time, the total amount of traffic is easily handled when it comes in a queue with fixed requests/second rate, it will only take some more time to catch up </li></ul><ul><li>maximum apache concurrent requests  > maximum concurrent worker connections </li></ul><ul><li>  - the program that process the logs cannot make more requests than the apache can handle, we configure apache to 1000 concurrent requests and configure the worker to input 260 concurrent requets. so apache has some free slots to other admin tasks. </li></ul><ul><li>mysql max connections > apache concurrent requests </li></ul><ul><li>  - otherwise you will get &quot;too many connections&quot; </li></ul><ul><li>archive.php performance > rest api input rate </li></ul><ul><li>  - you can't input more that than archieve.php will be able to handle, otherwise you will endup with logs that you will never be able to process </li></ul>
  16. 16. Our real setup, how we deployed it <ul><li>AWS autoscaling for the Nginx machines </li></ul><ul><li>    - easy high availability, increases and decrease collecting machines automatically </li></ul><ul><li>      saving money. </li></ul><ul><li>    - logrotate runs when a machine is &quot;terminated&quot;, to make sure none requests were lost </li></ul><ul><li>AWS SNS </li></ul><ul><li>    - To easily notifies when a new log files is ready to use, making it easy to synchronize </li></ul><ul><li>      the files processing </li></ul><ul><li>    - Notifies to multiple queues </li></ul><ul><li>    - Having multiple queues, we can use same logs to multiple analysis tools. We use </li></ul><ul><li>      one for web analysis and another for flash player debug </li></ul><ul><li>AWS SQS </li></ul><ul><li>    - Easy queue service, so we don't need expensive and complex high availability setups for it </li></ul><ul><li>AWS S3 </li></ul><ul><li>    - Cheap and virtually unlimited storage </li></ul><ul><li>    - Very easy access to files </li></ul><ul><li>    - Durable, Amazon guarantees better durability than regular data centers </li></ul><ul><li>Nginx </li></ul><ul><li>    - embedded perl script to get real IP on amazon (perl module is also include in ubuntu's </li></ul><ul><li>      package) </li></ul><ul><li>Logrotate </li></ul><ul><li>    - Added a s3cmd command (package also available on ubuntu) to upload the log to a S3 bucket, </li></ul><ul><li>      and added an AWS CLI command to send a notification to SNS once it is finished. </li></ul>
  17. 17. Our setup diagram Visits ELB Elastic Load Balancer nginx autoscaling pool S3 bucket SNS Notifications SQS queues Other workers/processors for other projects worker BigAss MySql mysql connection mysql slave, apache, piwik api, python-boto, python-twisted mysql master, piwik Piwik Users one file per virtualhost per machine, for each 5 minutes one notification per s3 file Datacenter
  18. 18. Our Worker - Part 1 <ul><li>Our choice for the REST API was based in same PHP scaling philosophy: Small standalone processes easy to multiply. Also, as in the MySQL replication, it is easier and healthy to process lot of small pieces than to freeze the servers with huge processes. </li></ul><ul><li>To input the requests in parallel we used python twisted as shown in this blog post: </li></ul><ul><li> </li></ul><ul><li>We installed apache and piwik in the mysql slave machine (of course the php is connecting in the master's mysql), then we tuned apache, mysql and tcp connections (as shown before). We access the REST API using </li></ul><ul><li>From the twisted blog post mentioned early, we changed the maxRun to 260, added some logging and error handling (we check if a gif was returned and its size, otherwise we log the failed request to be reprocessed later), and we implemented the callLater mentioned in the blog's comments for 0.03 seconds. </li></ul><ul><li>To get messages and files from Amazon, we are using python-boto </li></ul>
  19. 19. Our Worker - Part 2 <ul><li>Work Flow: </li></ul><ul><li>    - check for new messages in the AWS SQS queue </li></ul><ul><li>    - if there is an message, it means a new file is available, the message </li></ul><ul><li>      contains a s3 file path </li></ul><ul><li>    - with the s3 file path, download it, uncompress it </li></ul><ul><li>    - transform the NSCA log into a REST API request URL </li></ul><ul><li>    - put the URL in an array </li></ul><ul><li>    - delete the message in the queue </li></ul><ul><li>    - run twisted reactor for that array, making requests in the Piwik server in parallel </li></ul><ul><li>    - if a request fails, log it to be reprocessed later, </li></ul><ul><li>      alarm it in the monitoring system </li></ul><ul><li>      (we use zabbix btw, for more information: </li></ul><ul><li>  </li></ul><ul><li>Note: It is good to have one SNS and SQS for each virtual host if you have too many. </li></ul><ul><li>  </li></ul><ul><li>Python details later </li></ul>
  20. 20. Better costs management <ul><li>- Contributing to Piwik sharing our ideas, brings more ideas and more improvements, and  </li></ul><ul><li>    one of its consequences is to reduce costs </li></ul><ul><li>- CPU on Amazon is cheap and you pay as you use by time </li></ul><ul><li>- Traffic on Amazon is cheap, and you pay as you use, no long term contracts </li></ul><ul><li>- By dividing work we can better manage resources, like having only one or two huge machines  </li></ul><ul><li>  for the MySQLs and lots of small virtual nginx on autoscaling setup. It is easy to decouple the workers  </li></ul><ul><li>  processing to other machines </li></ul><ul><li>- Not changing Piwik's code reduces maintenance and development costs </li></ul><ul><li>- High Availability on Amazon is easy and cheap </li></ul><ul><li>- Storage durability on Amazon is automatic and cheap </li></ul><ul><li>- Storage retrieval and management on Amazon is very easy and fast </li></ul><ul><li>- Distribution control on Amazon is easy and cheap </li></ul><ul><li>- Having an easy way to access the logs, makes it simple to replay traffic, so you can run tests as </li></ul><ul><li>much as you need, and test as many tools as you want, improving resources usage and reducing costs </li></ul><ul><li>- Amazon reduces their prices and improve services all the time </li></ul>
  21. 21. Not only web analytics <ul><li>We are also using Piwik to log video plays </li></ul><ul><li>Once an user hits the play button in the flash player, it triggers a </li></ul><ul><li>GET request similiar to this: </li></ul><ul><li> </li></ul><ul><li>And we use the Video name as the Action's Page Tittle </li></ul><ul><li>It will appear in the Piwik's Actions interface divided by category, </li></ul><ul><li>and by Videos Name in the actions page tittles </li></ul>
  22. 22. Real Numbers <ul><li>Sorry, we can't provide real numbers, but we can do tests and show how far we can go. </li></ul><ul><li>  - Collecting data </li></ul><ul><li>    Nginx: as much requests per second as we need, just a matter of adding more nginx </li></ul><ul><li>    cheap virtual machines </li></ul><ul><li>  - REST API </li></ul><ul><li>    Running outside the master machine we've got 1500 requests/s, our Mysql Master </li></ul><ul><li>    has 2 quad core cpus, 64GB memory and Raid 10 </li></ul><ul><li>  - Download of logs </li></ul><ul><li>    If you run inside amazon, the traffic is free, the bandwidth is huge </li></ul><ul><li>    and the latency is small. We download the logs outside amazon and it is not our  </li></ul><ul><li>    bottleneck yet </li></ul><ul><li>  - Distribution tasks control </li></ul><ul><li>    SNS and SQS do it for us, not a bottleneck yet </li></ul><ul><li>  - We are still testing how many data we can archive for a month or two, it is already </li></ul><ul><li>    possible to archive one hour of 1000 requests/s in 30 minutes (considering S3 download  </li></ul><ul><li>    and uncompress), enough to log 50 million a day. But the tests are in too early stages. </li></ul>
  23. 23. CODE OR GTFO! <ul><li>It is hard to show all the code. </li></ul><ul><li>Most of tools are regular tools from Ubuntu and Amazon, and some others are relevant only for us. But some code and few links can help a lot. </li></ul><ul><li>Unfortunately I can't teach everything and how to use all Amazon tools, but the key points will be shown, like how to get the real user ip address on Amazon and some of the linux and mysql tuning. </li></ul>
  24. 24. MySql tuning details - Raid <ul><li>Raid: </li></ul><ul><li>Our Raid: </li></ul><ul><li>Our commands: </li></ul><ul><li>Check battery status: </li></ul><ul><li>/usr/sbin/megacli -AdpBbuCmd -GetBbuStatus -a0 | grep -e '^isSOHGood'|grep ': Yes' </li></ul><ul><li>Turn on the write cache: </li></ul><ul><li>/usr/sbin/megacli -LDInfo -LAll -aAll|tee /tmp/chefraidstatus |grep 'Default Cache Policy: WriteBack' </li></ul><ul><li>  </li></ul><ul><li>Turn on the cache: </li></ul><ul><li>/usr/sbin/megacli -LDSetProp Cached -LALL -aALL </li></ul><ul><li>  </li></ul><ul><li>Turn off the cache in case the battery is not good </li></ul><ul><li>/usr/sbin/megacli -LDSetProp Cached -LALL -aALL </li></ul><ul><li>  </li></ul><ul><li>Turn on the HDD cache </li></ul><ul><li>/usr/sbin/megacli -LDSetProp EnDskCache -LAll -aAll </li></ul><ul><li>Turn on the adaptive cache </li></ul><ul><li>/usr/sbin/megacli -LDSetProp ADRA -LALL -aALL </li></ul><ul><li>DO NOT FORGET TO MONITOR THE RAID: There are tool for it in the website above </li></ul><ul><li>  </li></ul>
  25. 25. MySql tuning details - Innodb <ul><li>all mysql tunings you can find here: </li></ul><ul><li> </li></ul><ul><li>Our tuning: </li></ul><ul><li>  </li></ul><ul><li>table_cache=1024 tmp_table_size=6G max_heap_table_size=6G thread_cache=16 query_cache_size=1G query_cache_limit=4M </li></ul><ul><li>default-storage-engine = InnoDB expire_logs_days = 5 </li></ul><ul><li>ignore-builtin-innodb max_binlog_size = 1024M skip-name-resolve innodb_flush_log_at_trx_commit=2 innodb_thread_concurrency=32  # we have 16 cpu threads </li></ul><ul><li>innodb_buffer_pool_size = 40G # we have 64G of memory innodb_flush_method=O_DIRECT innodb_additional_mem_pool_size=100M innodb_log_buffer_size = 18M innodb_log_file_size = 300M </li></ul><ul><li>interactive_timeout = 999999 </li></ul><ul><li>wait_timeout = 999999 </li></ul>
  26. 26. Linux tuning <ul><li>/etc/sysctl.conf: </li></ul><ul><li>vm.swappiness = 0 </li></ul><ul><li>net.core.somaxconn = 1024 </li></ul><ul><li>net.ipv4.tcp_rmem = 4096 4096 16777216 net.ipv4.tcp_wmem = 4096 4096 16777216 net.ipv4.tcp_timestamps = 0 net.ipv4.tcp_sack = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_fin_timeout = 20 net.ipv4.tcp_keepalive_intvl = 30 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_tw_reuse = 1 net.core.netdev_max_backlog = 5000 net.ipv4.ip_local_port_range = 2000 65535 fs.file-max=999999 </li></ul><ul><li>/etc/security/limits.conf </li></ul><ul><li>  </li></ul><ul><li># max open files *               -       nofile         999999 </li></ul><ul><li>/etc/default/nginx </li></ul><ul><li>ULIMIT=&quot;-n 999999&quot; </li></ul>
  27. 27. Nginx confs - Getting Real User IP on AWS ELB <ul><li># apt-get install nginx-extras </li></ul><ul><li>To get real user IP in a Elastic Load Balancer setup, added those lines inside the http context on /etc/nginx/nginx.conf: </li></ul><ul><li>        perl_set  $ip 'sub {                 my $r=shift;                 local $_ = $r->header_in(&quot;X-Forwarded-For&quot;);                 # XXX only works well because we know the AWS network uses 10.x.x.x ip addresses </li></ul><ul><li>                # Thanks Zed9h                 my $ip0 = m{.*b(                         (?:                                 d|                                 1[1-9]|                                 [2-9]d|                                 [12]d{2}                         ).d+.d+.d+                 )b}xo && $1;                 # $ip0 ne $ip1 && &quot;$ip0 ne $ip1tt$_&quot;; # debug                 $ip0 || $r->remote_addr         }'; </li></ul><ul><li>(Thanks Zed for the Perl script) </li></ul>
  28. 28. Nginx confs - Adding a virtual host (1/2) <ul><li>create a file on /etc/nginx/sites-available/VHOST.conf </li></ul><ul><li>server {         listen   80; ## listen for ipv4; this line is default and implied </li></ul><ul><li>        server_name;         root /usr/share/nginx/www;         index index.html index.htm;         userid on;         userid_name uid;         userid_domain;         userid_expires max;         set $myid $uid_got;         location = /crossdomain.xml {                 echo &quot;<?xml version=&quot;1.0&quot;?><!DOCTYPE cross-domain-policy SYSTEM &quot;;><cross-domain-policy><allow-access-from domain=&quot;*&quot; /></cross-domain-policy>&quot;;                   expires       modified +24h;                   access_log off;                   error_log /var/log/nginx/error.log;         } </li></ul><ul><li>        location / {                 if ($uid_got = &quot;&quot;){                         set $myid $uid_set;                 }                 expires -1; #               return 204;  #use this if you want an empty response                 empty_gif;   #use this if you want an empty gif response         } </li></ul>
  29. 29. <ul><li>        location /healthcheck {                     try_files $uri $uri $uri =404;                     access_log off;                     error_log /var/log/nginx/error.log;         }                           location /nginx_status {           stub_status on;           access_log   off;           allow;           deny all;           access_log off;           error_log /var/log/nginx/error.log;         }         # !!!!!!!!!!!!!!!!!!!!         # the log format is for Amazon AWS only, if you have the real IP, change         # the ip variable to $remote_addr         log_format VHOST        '$ip - $remote_user [$time_local]  '                                 '&quot;$request&quot; $status $body_bytes_sent '                                 '&quot;$http_referer&quot; &quot;$http_user_agent&quot; '                                 '&quot;$myid&quot;';         # /mnt is the AWS's fastest partition         access_log /mnt/log/nginx/VHOST.access.log VHOST;         error_log /mnt/log/nginx/VHOST.error.log; } </li></ul>Nginx confs - Adding a virtual host (2/2)
  30. 30. Testing Nginx <ul><li>$ curl localhost/ </li></ul><ul><li>result must be a gif </li></ul><ul><li>$ curl -I localhost/ </li></ul><ul><li>cookie must be set </li></ul>
  31. 31. Php tuning details <ul><li># apt-get install php-apc </li></ul><ul><li>  </li></ul><ul><li>  </li></ul><ul><li>create a file /etc/php5/conf.d/piwik.ini </li></ul><ul><li>memory_limit = 15G max_execution_time = 0 max_input_time = 0 apc.shm_size = 64 </li></ul><ul><li>* Piwik tuning on previous slides </li></ul>
  32. 32. AWS SNS <ul><li>  </li></ul><ul><li>It is out of the scope to teach how to create an Autoscaling group, a SNS, a S3 bucket and SQS queue. </li></ul><ul><li>We will only show we using them. </li></ul><ul><li>  </li></ul><ul><li>Create a SNS topic on Amazon &quot;MYTOPIC&quot;, and attach a SQS queue on it &quot;MYQUEUE&quot; </li></ul><ul><li>Install SNS client and unzip it somewhere, let's say /usr/local/bin: </li></ul><ul><li>Download from Amazon the file: </li></ul><ul><li>Install JDK: </li></ul><ul><li># apt-get install openjdk-6-jdk </li></ul><ul><li>Create a .conf file with a key and secret to access the SNS. let's say /usr/local/sns.conf: </li></ul><ul><li>AWSAccessKeyId=XXXXXXXXXXXX AWSSecretKey=XXXXXXXXX </li></ul><ul><li>Create a source file on /usr/local/sns_env.source: </li></ul><ul><li>export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bin/SimpleNotificationServiceCli-   </li></ul><ul><li>export AWS_SNS_HOME=/usr/local/bin/SimpleNotificationServiceCli- export EC2_REGION=us-east-1 export AWS_CREDENTIAL_FILE=/usr/local/sns.conf </li></ul>
  33. 33. AWS S3 and s3cmd <ul><li>Create a S3 bucket on Amazon </li></ul><ul><li>$ apt-get install s3cmd </li></ul><ul><li>$ s3cmd --configure </li></ul><ul><li>$ cp ~/.s3cmd.cfg /usr/local/s3cmd.cfg </li></ul>
  34. 34. Log rotate (1/3) <ul><li>Script was ripped from Ubuntu's init and logrotate and put in a cronjob. You can use logrotate though. </li></ul><ul><li>  </li></ul><ul><li>in the crontab: </li></ul><ul><li>*/5 * * * *  nice /bin/bash /usr/loca/bin/ >> /mnt/log/VHOST.send.log 2>&1 </li></ul><ul><li>  </li></ul><ul><li> </li></ul><ul><li>  </li></ul><ul><li>#!/bin/bash date #print date to the log DEBUGS3=`mktemp` atexit() {         rm -f $DEBUGS3 } trap atexit 0 BUCKET=&quot;MYBUCKET&quot; PROJECT=&quot;MYVHOST&quot; ARCHIVEDIR=&quot;/mnt/MYVHOST/&quot; S3CMD_CONF=&quot;/usr/local/s3cmd.cfg&quot; ORIGINPATH=&quot;/mnt/log/nginx/VHOST.access.log&quot; SNS_ENV=&quot;/usr/local/sns_env.source&quot; SNS_TOPIC=&quot;MYTOPIC&quot;   </li></ul><ul><li>HOST=`hostname` #we use instance-id on amazon DATE=$(date --utc +%Y%m%d_%H%M%S) DATEDIR=$(date --utc +%Y/%m/%d) POSTPATH=&quot;$PROJECT/$DATEDIR/$PROJECT-$DATE-$HOST.log&quot; LOCALPATH=&quot;$ARCHIVEDIR/$POSTPATH&quot; GZLOCALPATH=&quot;$LOCALPATH.gz&quot; REMOTEPATH=&quot;s3://$BUCKET/$POSTPATH.gz&quot; </li></ul>
  35. 35. Log rotate (2/3) <ul><li>echo &quot;->Trying file: $REMOTEPATH&quot; LOCALDIR=&quot;$(dirname &quot;$LOCALPATH&quot;)&quot; #sleep 1 recomended by nginx's wiki mkdir -p &quot;$LOCALDIR&quot; && mv &quot;$ORIGINPATH&quot; &quot;$LOCALPATH&quot; && { [ ! -f /var/run/ ] || kill -USR1 `cat /var/run/` ; }  && sleep 1 && gzip &quot;$LOCALPATH&quot; && { MD5=$(/usr/bin/md5sum &quot;$GZLOCALPATH&quot; | awk '{ print $1 }') ; } #try 3 times if [ -z &quot;$MD5&quot; ] then         echo &quot;ERROR ON MD5&quot;         OK=1 else         OK=$(/usr/bin/s3cmd -d --no-progress -c &quot;$S3CMD_CONF&quot; put &quot;$GZLOCALPATH&quot; &quot;$REMOTEPATH&quot; 2>&1 |grep -q &quot;DEBUG: MD5 sums: computed=$MD5, received=&quot;$MD5&quot;&quot;;echo $?)         if [ &quot;$OK&quot; -eq &quot;1&quot; ]         then                 OK=$(/usr/bin/s3cmd -d --no-progress -c &quot;$S3CMD_CONF&quot; put &quot;$GZLOCALPATH&quot; &quot;$REMOTEPATH&quot; 2>&1 |grep -q &quot;DEBUG: MD5 sums: computed=$MD5, received=&quot;$MD5&quot;&quot;;echo $?)                 if [ &quot;$OK&quot; -eq &quot;1&quot; ]                 then                         /usr/bin/s3cmd -d --no-progress -c &quot;$S3CMD_CONF&quot; put &quot;$GZLOCALPATH&quot; &quot;$REMOTEPATH&quot; 2>&1|tee &quot;$DEBUGS3&quot;                         OK=$(grep -q &quot;DEBUG: MD5 sums: computed=$MD5, received=&quot;$MD5&quot;&quot; &quot;$DEBUGS3&quot;; echo $?)                 fi         fi fi </li></ul>
  36. 36. Log rotate (3/3) <ul><li>  </li></ul><ul><li># if ok, publish a message on SNS </li></ul><ul><li>  </li></ul><ul><li>if [ &quot;$OK&quot; = &quot;0&quot; ] then         source &quot;$SNS_ENV&quot;         echo -n '-> Message: '         sns-publish &quot;$TOPIC&quot; --message &quot;$REMOTEPATH&quot;         OK=${PIPESTATUS[0]} fi echo &quot;OK=$OK&quot; #for monitoring #/usr/bin/zabbix_sender -s &quot;$HOST&quot; -z -k XXXXXX -o &quot;$OK&quot; </li></ul>
  37. 37. Rotating and uploading logs on reboot and shutdown <ul><li>This is a protection for the Autoscaling group where machines are created and </li></ul><ul><li>terminated all the time </li></ul><ul><li>create a file on /etc/init.d/ </li></ul><ul><li>#!/bin/bash </li></ul><ul><li>/bin/echo TERMINATED `date --utc` >> /mnt/log/nginx/VHOST.access.log </li></ul><ul><li>/usr/bin/nice -20 /bin/bash /usr/local/bin/ </li></ul><ul><li>Then execute: </li></ul><ul><li># update-rc.d stop 21 0 6 . </li></ul><ul><li>(the dot in the end of the line is required) </li></ul>
  38. 38. Worker details (1/3) <ul><li>I'm not a developer, my worker python code is too ugly to be shown. It is very similar to the blog post mentioned early, the only addition is download S3 files and ready messages on SQS, the functions are similar as those: </li></ul><ul><li>Connect to S3 and SQS </li></ul><ul><li>  </li></ul><ul><li>  </li></ul><ul><li>from boto.sqs.connection import SQSConnection from boto.s3.connection import S3Connection </li></ul><ul><li>from boto.sqs.message import RawMessage # for SNS messages </li></ul><ul><li>import json </li></ul><ul><li>print &quot;connecting to sqs&quot;;connecting to sqs&quot;) connsqs = SQSConnection('xxxxxxxxxxxx', 'xxxxxxxxxxxxx') </li></ul><ul><li>print &quot;connecting to s3&quot;;connection to s3&quot;) conns3 = S3Connection('xxxxxxxxxxxxxxxx', 'xxxxxxxxxxxxx') </li></ul>
  39. 39. Worker details (2/3) <ul><li>Reading and deleting SQS messages, and putting results in an array: </li></ul><ul><li>print &quot;getting queue&quot;;getting queue&quot;) my_queue = connsqs.get_queue('MYQUEUE') my_queue.set_message_class(RawMessage) #raw messages from SNS </li></ul><ul><li>maxmsgs = 10 msgs = [] msg = while msg:;getting message&quot;)         msgsingle = json.loads(msg.get_body())['Message']         msgs.append(msgsingle) </li></ul><ul><li>  </li></ul><ul><li>;deleting message&quot;)         my_queue.delete_message(msg)         if len(msgs) < maxmsg :       ;getting more messages&quot;)                 msg =         else:                 msg = False </li></ul>
  40. 40. Worker details (3/3) <ul><li>Getting files from S3 and putting lines in an array: </li></ul><ul><li>msg_data[] lines = [] filename = '/tmp/tmppiwikpy.%s.txt' % os.getpid() for msg_data in msgs:         llog = &quot;trying file &quot;+msg_data         if &quot;s3://MYBUCKET/&quot; in msg_data :                 s3obj = msg_data.replace(&quot;s3://MYBUCKET/&quot;,&quot;&quot;)                 llog = &quot;downloading &quot;+msg_data                        key = conns3.get_bucket('MYBUCKET').get_key(s3obj)                 key.get_contents_to_filename(filename)                 llog = &quot;decompressing file&quot;+msg_data                        fgz =, 'r');                 line = fgz.readline()                 while line:                         lines.append(line)                         line = fgz.readline()   </li></ul><ul><li>llog = &quot;closing and deleting temporary file&quot; fgz.close() os.remove(filename) </li></ul>
  41. 41. Piwik REST API <ul><li>Check it here: </li></ul><ul><li> </li></ul><ul><li>Your worker script has to create an url like this: </li></ul><ul><li>  </li></ul><ul><li> </li></ul><ul><li>  </li></ul><ul><li>The cookie from Nginx is the first 16 characters from its md5 sum, as the same as Piwik does internally </li></ul><ul><li>  </li></ul><ul><li>Date of visit must be in UTC </li></ul>
  42. 42. Others / Next steps <ul><li>  - If you deploy it not in Amazon, it makes sense to send the log lines to a queue </li></ul><ul><li>    and have lots of small workers reading and replicating them into Piwik. It is </li></ul><ul><li>    easier to handle, skip or reprocess a failed line than an entire log file </li></ul><ul><li>  - We still have window to improve, we are not using SSDs cards, we didn't do </li></ul><ul><li>    any partition or sharding </li></ul><ul><li>  - Visit logs and Action logs will need to be changed in order to make the database  </li></ul><ul><li>    cheaper and more scalable. </li></ul><ul><li>  - Our next step will be try to improve the archives, probably our next bottleneck. </li></ul>
  43. 43. What is missing on Piwik <ul><li>  - split read and write connections to mysql database, so we can have  </li></ul><ul><li>    the benefits of mysql replications, like run a dedicated slave to the </li></ul><ul><li>    archive.php selects and a dedicated slave to non-admin users. </li></ul><ul><li>  - create a database per website. It is easier to maintain and reduces </li></ul><ul><li>    the mysql indexes sizes to fit them in memory. You can partition the  </li></ul><ul><li>    tables by idsite, it helps. </li></ul><ul><li>  - people read this presentation and send feedbacks :) </li></ul><ul><li>    please use Piwik's forums for this </li></ul><ul><li>  - feature request: optionally have a Mysql connection per website or be </li></ul><ul><li>    able to configure Piwik's interface to import data from other Piwik  </li></ul><ul><li>    installations, having all websites on a single place. Doing that we can  </li></ul><ul><li>    have smaller databases for each website. (Zabbix have simliar feature) </li></ul><ul><li>- feature request: have mysql optional connections profiles, so we can </li></ul><ul><li>  set smaller buffers for smaller tasks, improving the memory usage </li></ul>
  44. 44. <ul><li>Thanks Piwik ! </li></ul><ul><li>Now we have modern analytics for old problems </li></ul><ul><li>and a modern scaling setup for a traditional LAMP stack </li></ul><ul><li>Thanks Zed (aka Carlo) for all programming support, </li></ul><ul><li>Gaiser for all Amazon tips, Matt for all Piwik tips. </li></ul><ul><li>And thanks to R7 Managers Denis and Vechiato to believe and provide time and </li></ul><ul><li>resources to make it happen and R7 Director Brandi to review this and allow us to share. </li></ul>