Extending Piwik At R7.com

Extending Piwik at r7.com Phase 1 – Collecting data Adding some cloud and modern scalability to a traditional LAMP stack leonardo lorieri, r7.com system architect, 'lorieri at gmail.com', feb/2012

Why Piwik ? - Open Source = flexible, understandable, free! - Great interface - Mobile app - REST API - Developers knows the market needs - Efficient in small machines - Lots of possible improvements - Lots of improvements already in the roadmap - Great and supportive community (Thank you all!)

Our Plan, goals and trade-offs - Don't change original code - reduces development and maintenance costs - Count only visits and page views - to be fast and focused ( even though you still can use the .js tracker, it is easy to get lost in the UI's beauty and all its functionalities) - Handle odd unexpected traffic peaks - from tv announcements - Count not only websites - media delivery, internal searches, debugs - At least 99% of accuracy - Have numbers to compare with other analytics tools - We've lost P3P for now

Our big problem - The TV Effect ,[object Object],[object Object]

Regular Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes - Apache/Nginx - Php - Mysql

Bigger Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes - Apache/Nginx - Php - MySql

Regular Php Scaling Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes - Apache/Nginx - Php - MySql Replication (slave for backup only, piwik is not "slave ready") Load balancer/Nginx

Two problems, one easy solution ,[object Object],[object Object],[object Object],[object Object],[object Object]

Asynchronous Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes - Nginx - NOT even Php - MySql Master - Apache+Php for Admin UI - Archive cron Load balancer/Nginx - MySql Slave - Perl/Python worker to process logs (manages user cookies) (user cookie) - accesses logs Visits REST API <img src=> request Admin/ Reports

Nginx (more details later) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Log processing "Worker" (more details later) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Mysql Master (more details later) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Php tuning (more details later) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Piwik tuning ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Handling TV Effect ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Our real setup, how we deployed it ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Our setup diagram Visits ELB Elastic Load Balancer nginx autoscaling pool S3 bucket SNS Notifications SQS queues Other workers/processors for other projects worker BigAss MySql mysql connection mysql slave, apache, piwik api, python-boto, python-twisted mysql master, piwik Piwik Users one file per virtualhost per machine, for each 5 minutes one notification per s3 file Datacenter

Our Worker - Part 1 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Our Worker - Part 2 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Better costs management ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Not only web analytics ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Real Numbers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

CODE OR GTFO! ,[object Object],[object Object],[object Object]

MySql tuning details - Raid ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

MySql tuning details - Innodb ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Linux tuning ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Nginx confs - Getting Real User IP on AWS ELB ,[object Object],[object Object],[object Object],[object Object],[object Object]

Nginx confs - Adding a virtual host (1/2) ,[object Object],[object Object],[object Object],[object Object]

[object Object],Nginx confs - Adding a virtual host (2/2)

Testing Nginx ,[object Object],[object Object],[object Object],[object Object]

Php tuning details ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

AWS SNS ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

AWS S3 and s3cmd ,[object Object],[object Object],[object Object],[object Object]

Log rotate (1/3) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Log rotate (2/3) ,[object Object]

Log rotate (3/3) ,[object Object],[object Object],[object Object],[object Object]

Rotating and uploading logs on reboot and shutdown ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Worker details (1/3) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Worker details (2/3) ,[object Object],[object Object],[object Object],[object Object],[object Object]

Worker details (3/3) ,[object Object],[object Object],[object Object]

Piwik REST API ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Others / Next steps ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What is missing on Piwik ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Extending Piwik At R7.com

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (13)

Similar to Extending Piwik At R7.com

Similar to Extending Piwik At R7.com (20)

More from Leo Lorieri

More from Leo Lorieri (7)

Recently uploaded

Recently uploaded (20)

Extending Piwik At R7.com