SlideShare a Scribd company logo
1 of 44
Download to read offline
Extending Piwik at r7.com   Phase 1 – Collecting data Adding some cloud and modern scalability to a traditional LAMP stack leonardo lorieri, r7.com system architect, 'lorieri at gmail.com', feb/2012
Why Piwik ?   - Open Source = flexible, understandable, free!   - Great interface   - Mobile app   - REST API   - Developers knows the market needs   - Efficient in small machines   - Lots of possible improvements   - Lots of improvements already in the roadmap   - Great and supportive community (Thank you all!)
Our Plan, goals and trade-offs   - Don't change original code        - reduces development and maintenance costs   - Count only visits and page views        - to be fast and focused ( even though you still can use the .js tracker,          it is easy to get lost in the UI's beauty and all its functionalities)   - Handle odd unexpected traffic peaks        - from tv announcements    - Count not only websites        - media delivery, internal searches, debugs   - At least 99% of accuracy   - Have numbers to compare with other analytics tools   - We've lost P3P for now
Our big problem - The TV Effect ,[object Object],[object Object]
Regular Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes   - Apache/Nginx   - Php   - Mysql
Bigger Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes   - Apache/Nginx   - Php   - MySql
Regular Php Scaling Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes   - Apache/Nginx   - Php   - MySql Replication       (slave for backup only,        piwik is not "slave ready") Load balancer/Nginx
Two problems, one easy solution ,[object Object],[object Object],[object Object],[object Object],[object Object]
Asynchronous Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes   - Nginx   -  NOT even  Php   - MySql Master   - Apache+Php for Admin UI   - Archive cron Load balancer/Nginx   - MySql Slave   - Perl/Python worker to     process logs (manages user cookies) (user cookie) - accesses logs Visits REST API <img src=> request Admin/ Reports
Nginx  (more details later) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log processing &quot;Worker&quot;  (more details later) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Mysql Master  (more details later) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Php tuning  (more details later) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Piwik tuning ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Handling TV Effect ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Our real setup, how we deployed it ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Our setup diagram Visits ELB Elastic Load Balancer nginx autoscaling pool S3 bucket SNS Notifications SQS queues Other workers/processors for other projects worker BigAss MySql mysql connection mysql slave, apache, piwik api, python-boto, python-twisted mysql master, piwik Piwik Users one file per virtualhost per machine, for each 5 minutes one notification per s3 file Datacenter
Our Worker - Part 1 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Our Worker - Part 2 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Better costs management ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Not only web analytics ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Real Numbers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
CODE OR GTFO! ,[object Object],[object Object],[object Object]
MySql tuning details - Raid ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
MySql tuning details - Innodb ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Linux tuning ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Nginx confs - Getting Real User IP on AWS ELB ,[object Object],[object Object],[object Object],[object Object],[object Object]
Nginx confs - Adding a virtual host (1/2) ,[object Object],[object Object],[object Object],[object Object]
[object Object],Nginx confs - Adding a virtual host (2/2)
Testing Nginx ,[object Object],[object Object],[object Object],[object Object]
Php tuning details ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
AWS SNS ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
AWS S3 and s3cmd ,[object Object],[object Object],[object Object],[object Object]
Log rotate (1/3) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log rotate (2/3) ,[object Object]
Log rotate (3/3) ,[object Object],[object Object],[object Object],[object Object]
Rotating and uploading logs on reboot and shutdown ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Worker details (1/3) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Worker details (2/3) ,[object Object],[object Object],[object Object],[object Object],[object Object]
Worker details (3/3) ,[object Object],[object Object],[object Object]
Piwik REST API ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Others / Next steps ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What is missing on Piwik ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

More Related Content

What's hot

A (XPages) developers guide to Cloudant - MeetIT
A (XPages) developers guide to Cloudant - MeetITA (XPages) developers guide to Cloudant - MeetIT
A (XPages) developers guide to Cloudant - MeetITFrank van der Linden
 
Collab365 Global Summit Slides
Collab365 Global Summit SlidesCollab365 Global Summit Slides
Collab365 Global Summit SlidesRick Van Rousselt
 
SPSNL17 - Getting notified by SharePoint with the webhook functionality - Eli...
SPSNL17 - Getting notified by SharePoint with the webhook functionality - Eli...SPSNL17 - Getting notified by SharePoint with the webhook functionality - Eli...
SPSNL17 - Getting notified by SharePoint with the webhook functionality - Eli...DIWUG
 
Tabtale story: Building a publishing and monitoring mobile games architecture...
Tabtale story: Building a publishing and monitoring mobile games architecture...Tabtale story: Building a publishing and monitoring mobile games architecture...
Tabtale story: Building a publishing and monitoring mobile games architecture...Tikal Knowledge
 
Write Serverless Applications with PowerShell and Azure Functions
Write Serverless Applications with PowerShell and Azure FunctionsWrite Serverless Applications with PowerShell and Azure Functions
Write Serverless Applications with PowerShell and Azure FunctionsJaap Brasser
 
Vincent biret azure functions and flow (ottawa)
Vincent biret azure functions and flow (ottawa)Vincent biret azure functions and flow (ottawa)
Vincent biret azure functions and flow (ottawa)Vincent Biret
 
Vincent biret azure functions and flow (toronto)
Vincent biret azure functions and flow (toronto)Vincent biret azure functions and flow (toronto)
Vincent biret azure functions and flow (toronto)Vincent Biret
 
Blazing fast sites using Blaze, Hybrid CMS NYC
Blazing fast sites using Blaze, Hybrid CMS NYCBlazing fast sites using Blaze, Hybrid CMS NYC
Blazing fast sites using Blaze, Hybrid CMS NYCJesus Manuel Olivas
 
How to automate the SharePoint Provisioning
How to automate the SharePoint Provisioning How to automate the SharePoint Provisioning
How to automate the SharePoint Provisioning Knut Relbe-Moe [MVP, MCT]
 
App Services - Connecting the dots of Web Mobile and Integration_published
App Services - Connecting the dots of Web Mobile and Integration_publishedApp Services - Connecting the dots of Web Mobile and Integration_published
App Services - Connecting the dots of Web Mobile and Integration_publishedWagner Silveira
 
Open source web services
Open source web servicesOpen source web services
Open source web servicesRTigger
 
Playing in the Sandbox by Wictor Wilén
Playing in the Sandbox by Wictor WilénPlaying in the Sandbox by Wictor Wilén
Playing in the Sandbox by Wictor WilénWictor Wilén
 
SharePoint Saturday Belgium 2016
SharePoint Saturday Belgium 2016SharePoint Saturday Belgium 2016
SharePoint Saturday Belgium 2016Rick Van Rousselt
 
SharePoint + Silverlight - new BFF's by Wictor Wilén
SharePoint + Silverlight - new BFF's by Wictor WilénSharePoint + Silverlight - new BFF's by Wictor Wilén
SharePoint + Silverlight - new BFF's by Wictor WilénWictor Wilén
 
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...camunda services GmbH
 
Developing Cross-Platform Web Apps with ASP.NET Core1.0
Developing Cross-Platform Web Apps with ASP.NET Core1.0Developing Cross-Platform Web Apps with ASP.NET Core1.0
Developing Cross-Platform Web Apps with ASP.NET Core1.0EastBanc Tachnologies
 
SPSNL17 - Be more effective with the PnP Provisioning Engine - Erwin van Hunen
SPSNL17 - Be more effective with the PnP Provisioning Engine - Erwin van HunenSPSNL17 - Be more effective with the PnP Provisioning Engine - Erwin van Hunen
SPSNL17 - Be more effective with the PnP Provisioning Engine - Erwin van HunenDIWUG
 
Part 3 of the REAL Webinars on Oracle Cloud Native Application Development (J...
Part 3 of the REAL Webinars on Oracle Cloud Native Application Development (J...Part 3 of the REAL Webinars on Oracle Cloud Native Application Development (J...
Part 3 of the REAL Webinars on Oracle Cloud Native Application Development (J...Lucas Jellema
 
Navigate around the edge with PowerShell
Navigate around the edge with PowerShellNavigate around the edge with PowerShell
Navigate around the edge with PowerShellJaap Brasser
 

What's hot (20)

A (XPages) developers guide to Cloudant - MeetIT
A (XPages) developers guide to Cloudant - MeetITA (XPages) developers guide to Cloudant - MeetIT
A (XPages) developers guide to Cloudant - MeetIT
 
Collab365 Global Summit Slides
Collab365 Global Summit SlidesCollab365 Global Summit Slides
Collab365 Global Summit Slides
 
SPSNL17 - Getting notified by SharePoint with the webhook functionality - Eli...
SPSNL17 - Getting notified by SharePoint with the webhook functionality - Eli...SPSNL17 - Getting notified by SharePoint with the webhook functionality - Eli...
SPSNL17 - Getting notified by SharePoint with the webhook functionality - Eli...
 
Tabtale story: Building a publishing and monitoring mobile games architecture...
Tabtale story: Building a publishing and monitoring mobile games architecture...Tabtale story: Building a publishing and monitoring mobile games architecture...
Tabtale story: Building a publishing and monitoring mobile games architecture...
 
Write Serverless Applications with PowerShell and Azure Functions
Write Serverless Applications with PowerShell and Azure FunctionsWrite Serverless Applications with PowerShell and Azure Functions
Write Serverless Applications with PowerShell and Azure Functions
 
Vincent biret azure functions and flow (ottawa)
Vincent biret azure functions and flow (ottawa)Vincent biret azure functions and flow (ottawa)
Vincent biret azure functions and flow (ottawa)
 
Vincent biret azure functions and flow (toronto)
Vincent biret azure functions and flow (toronto)Vincent biret azure functions and flow (toronto)
Vincent biret azure functions and flow (toronto)
 
Blazing fast sites using Blaze, Hybrid CMS NYC
Blazing fast sites using Blaze, Hybrid CMS NYCBlazing fast sites using Blaze, Hybrid CMS NYC
Blazing fast sites using Blaze, Hybrid CMS NYC
 
How to automate the SharePoint Provisioning
How to automate the SharePoint Provisioning How to automate the SharePoint Provisioning
How to automate the SharePoint Provisioning
 
App Services - Connecting the dots of Web Mobile and Integration_published
App Services - Connecting the dots of Web Mobile and Integration_publishedApp Services - Connecting the dots of Web Mobile and Integration_published
App Services - Connecting the dots of Web Mobile and Integration_published
 
Open source web services
Open source web servicesOpen source web services
Open source web services
 
Playing in the Sandbox by Wictor Wilén
Playing in the Sandbox by Wictor WilénPlaying in the Sandbox by Wictor Wilén
Playing in the Sandbox by Wictor Wilén
 
SharePoint Saturday Belgium 2016
SharePoint Saturday Belgium 2016SharePoint Saturday Belgium 2016
SharePoint Saturday Belgium 2016
 
SharePoint + Silverlight - new BFF's by Wictor Wilén
SharePoint + Silverlight - new BFF's by Wictor WilénSharePoint + Silverlight - new BFF's by Wictor Wilén
SharePoint + Silverlight - new BFF's by Wictor Wilén
 
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
 
Developing Cross-Platform Web Apps with ASP.NET Core1.0
Developing Cross-Platform Web Apps with ASP.NET Core1.0Developing Cross-Platform Web Apps with ASP.NET Core1.0
Developing Cross-Platform Web Apps with ASP.NET Core1.0
 
Firebase
FirebaseFirebase
Firebase
 
SPSNL17 - Be more effective with the PnP Provisioning Engine - Erwin van Hunen
SPSNL17 - Be more effective with the PnP Provisioning Engine - Erwin van HunenSPSNL17 - Be more effective with the PnP Provisioning Engine - Erwin van Hunen
SPSNL17 - Be more effective with the PnP Provisioning Engine - Erwin van Hunen
 
Part 3 of the REAL Webinars on Oracle Cloud Native Application Development (J...
Part 3 of the REAL Webinars on Oracle Cloud Native Application Development (J...Part 3 of the REAL Webinars on Oracle Cloud Native Application Development (J...
Part 3 of the REAL Webinars on Oracle Cloud Native Application Development (J...
 
Navigate around the edge with PowerShell
Navigate around the edge with PowerShellNavigate around the edge with PowerShell
Navigate around the edge with PowerShell
 

Viewers also liked

Piwik in enterprise munich 2014
Piwik in enterprise munich 2014Piwik in enterprise munich 2014
Piwik in enterprise munich 2014piwik
 
Piwik Analytics Platform
Piwik Analytics PlatformPiwik Analytics Platform
Piwik Analytics Platformpiwik
 
Learning Analytics and Serious Games: Trends and Considerations
Learning Analytics and Serious Games: Trends and ConsiderationsLearning Analytics and Serious Games: Trends and Considerations
Learning Analytics and Serious Games: Trends and ConsiderationsLaila Shoukry
 
AMIMOTO WordPress + Amazon Web Services Hands-on
AMIMOTO WordPress + Amazon Web Services Hands-on AMIMOTO WordPress + Amazon Web Services Hands-on
AMIMOTO WordPress + Amazon Web Services Hands-on Kel
 
소셜미디어 모니터링 툴 리스트 Via hoyalog
소셜미디어 모니터링 툴 리스트 Via hoyalog소셜미디어 모니터링 툴 리스트 Via hoyalog
소셜미디어 모니터링 툴 리스트 Via hoyalogHoya Kim
 
Introduction to piwik analytics platform 2015
Introduction to piwik analytics platform 2015Introduction to piwik analytics platform 2015
Introduction to piwik analytics platform 2015Matthieu Aubry
 
Capacity Planning for Linux Systems
Capacity Planning for Linux SystemsCapacity Planning for Linux Systems
Capacity Planning for Linux SystemsRodrigo Campos
 
Piwik elasticsearch kibana at OSC Tokyo 2016 Spring
Piwik elasticsearch kibana at OSC Tokyo 2016 SpringPiwik elasticsearch kibana at OSC Tokyo 2016 Spring
Piwik elasticsearch kibana at OSC Tokyo 2016 SpringTakashi Yamamoto
 
Privacy Regulations and Your Digital Setup
Privacy Regulations and Your Digital SetupPrivacy Regulations and Your Digital Setup
Privacy Regulations and Your Digital SetupPiwik PRO
 
[C1]웹서비스, 빠를수록 좋다
[C1]웹서비스, 빠를수록 좋다[C1]웹서비스, 빠를수록 좋다
[C1]웹서비스, 빠를수록 좋다NAVER D2
 
Web Analytics and Privacy
Web Analytics and Privacy Web Analytics and Privacy
Web Analytics and Privacy Piwik PRO
 
A Comparison of Analytics and Tag Management Suites by Piwik PRO and Google
A Comparison of Analytics and Tag Management Suites by Piwik PRO and GoogleA Comparison of Analytics and Tag Management Suites by Piwik PRO and Google
A Comparison of Analytics and Tag Management Suites by Piwik PRO and GooglePiwik PRO
 
Javascript Tracking or Web Log Analytics?
Javascript Tracking or Web Log Analytics? Javascript Tracking or Web Log Analytics?
Javascript Tracking or Web Log Analytics? Piwik PRO
 

Viewers also liked (13)

Piwik in enterprise munich 2014
Piwik in enterprise munich 2014Piwik in enterprise munich 2014
Piwik in enterprise munich 2014
 
Piwik Analytics Platform
Piwik Analytics PlatformPiwik Analytics Platform
Piwik Analytics Platform
 
Learning Analytics and Serious Games: Trends and Considerations
Learning Analytics and Serious Games: Trends and ConsiderationsLearning Analytics and Serious Games: Trends and Considerations
Learning Analytics and Serious Games: Trends and Considerations
 
AMIMOTO WordPress + Amazon Web Services Hands-on
AMIMOTO WordPress + Amazon Web Services Hands-on AMIMOTO WordPress + Amazon Web Services Hands-on
AMIMOTO WordPress + Amazon Web Services Hands-on
 
소셜미디어 모니터링 툴 리스트 Via hoyalog
소셜미디어 모니터링 툴 리스트 Via hoyalog소셜미디어 모니터링 툴 리스트 Via hoyalog
소셜미디어 모니터링 툴 리스트 Via hoyalog
 
Introduction to piwik analytics platform 2015
Introduction to piwik analytics platform 2015Introduction to piwik analytics platform 2015
Introduction to piwik analytics platform 2015
 
Capacity Planning for Linux Systems
Capacity Planning for Linux SystemsCapacity Planning for Linux Systems
Capacity Planning for Linux Systems
 
Piwik elasticsearch kibana at OSC Tokyo 2016 Spring
Piwik elasticsearch kibana at OSC Tokyo 2016 SpringPiwik elasticsearch kibana at OSC Tokyo 2016 Spring
Piwik elasticsearch kibana at OSC Tokyo 2016 Spring
 
Privacy Regulations and Your Digital Setup
Privacy Regulations and Your Digital SetupPrivacy Regulations and Your Digital Setup
Privacy Regulations and Your Digital Setup
 
[C1]웹서비스, 빠를수록 좋다
[C1]웹서비스, 빠를수록 좋다[C1]웹서비스, 빠를수록 좋다
[C1]웹서비스, 빠를수록 좋다
 
Web Analytics and Privacy
Web Analytics and Privacy Web Analytics and Privacy
Web Analytics and Privacy
 
A Comparison of Analytics and Tag Management Suites by Piwik PRO and Google
A Comparison of Analytics and Tag Management Suites by Piwik PRO and GoogleA Comparison of Analytics and Tag Management Suites by Piwik PRO and Google
A Comparison of Analytics and Tag Management Suites by Piwik PRO and Google
 
Javascript Tracking or Web Log Analytics?
Javascript Tracking or Web Log Analytics? Javascript Tracking or Web Log Analytics?
Javascript Tracking or Web Log Analytics?
 

Similar to Extending Piwik At R7.com

High Performance Web Sites
High Performance Web SitesHigh Performance Web Sites
High Performance Web SitesRavi Raj
 
Joomla! Performance on Steroids
Joomla! Performance on SteroidsJoomla! Performance on Steroids
Joomla! Performance on SteroidsSiteGround.com
 
Serial Data from Arduino to Raspberry Pi to MySQL using CoAP Protocol
Serial Data from Arduino to Raspberry Pi to MySQL using CoAP ProtocolSerial Data from Arduino to Raspberry Pi to MySQL using CoAP Protocol
Serial Data from Arduino to Raspberry Pi to MySQL using CoAP ProtocolSanjay Kumar
 
AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09Chris Purrington
 
HTTP Plugin for MySQL!
HTTP Plugin for MySQL!HTTP Plugin for MySQL!
HTTP Plugin for MySQL!Ulf Wendel
 
Magento scalability from the trenches (Meet Magento Sweden 2016)
Magento scalability from the trenches (Meet Magento Sweden 2016)Magento scalability from the trenches (Meet Magento Sweden 2016)
Magento scalability from the trenches (Meet Magento Sweden 2016)Divante
 
ApacheCon NA 2010 - High Performance Cloud-enabled SCA Runtimes
ApacheCon NA 2010 - High Performance Cloud-enabled SCA RuntimesApacheCon NA 2010 - High Performance Cloud-enabled SCA Runtimes
ApacheCon NA 2010 - High Performance Cloud-enabled SCA RuntimesJean-Sebastien Delfino
 
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky  "How to manage hundreds of pipelines for processing da...Yaroslav Nedashkovsky  "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...Lviv Startup Club
 
Roy foubister (hosting high traffic sites on a tight budget)
Roy foubister (hosting high traffic sites on a tight budget)Roy foubister (hosting high traffic sites on a tight budget)
Roy foubister (hosting high traffic sites on a tight budget)WordCamp Cape Town
 
Scale Apache with Nginx
Scale Apache with NginxScale Apache with Nginx
Scale Apache with NginxBud Siddhisena
 
PHP is the King, nodejs is the Prince and Lua is the fool
PHP is the King, nodejs is the Prince and Lua is the foolPHP is the King, nodejs is the Prince and Lua is the fool
PHP is the King, nodejs is the Prince and Lua is the foolAlessandro Cinelli (cirpo)
 
murakumo Cloud Controller
murakumo Cloud Controllermurakumo Cloud Controller
murakumo Cloud ControllerShingo Kawano
 
Ceph Day Taipei - Bring Ceph to Enterprise
Ceph Day Taipei - Bring Ceph to EnterpriseCeph Day Taipei - Bring Ceph to Enterprise
Ceph Day Taipei - Bring Ceph to EnterpriseCeph Community
 
The next step from Microsoft - Vnext (Srdjan Poznic)
The next step from Microsoft - Vnext (Srdjan Poznic)The next step from Microsoft - Vnext (Srdjan Poznic)
The next step from Microsoft - Vnext (Srdjan Poznic)Geekstone
 
Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)
Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)
Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)Nicolas Brousse
 
uWSGI - Swiss army knife for your Python web apps
uWSGI - Swiss army knife for your Python web appsuWSGI - Swiss army knife for your Python web apps
uWSGI - Swiss army knife for your Python web appsTomislav Raseta
 
Paris.rb – 07/19 – Sidekiq scaling, workers vs processes
Paris.rb – 07/19 – Sidekiq scaling, workers vs processesParis.rb – 07/19 – Sidekiq scaling, workers vs processes
Paris.rb – 07/19 – Sidekiq scaling, workers vs processesMaxence Haltel
 
Ceph Day Bring Ceph To Enterprise
Ceph Day Bring Ceph To EnterpriseCeph Day Bring Ceph To Enterprise
Ceph Day Bring Ceph To EnterpriseAlex Lau
 

Similar to Extending Piwik At R7.com (20)

High Performance Web Sites
High Performance Web SitesHigh Performance Web Sites
High Performance Web Sites
 
Joomla! Performance on Steroids
Joomla! Performance on SteroidsJoomla! Performance on Steroids
Joomla! Performance on Steroids
 
Serial Data from Arduino to Raspberry Pi to MySQL using CoAP Protocol
Serial Data from Arduino to Raspberry Pi to MySQL using CoAP ProtocolSerial Data from Arduino to Raspberry Pi to MySQL using CoAP Protocol
Serial Data from Arduino to Raspberry Pi to MySQL using CoAP Protocol
 
AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09
 
HTTP Plugin for MySQL!
HTTP Plugin for MySQL!HTTP Plugin for MySQL!
HTTP Plugin for MySQL!
 
SFDCP – sfdc project
SFDCP – sfdc projectSFDCP – sfdc project
SFDCP – sfdc project
 
Magento scalability from the trenches (Meet Magento Sweden 2016)
Magento scalability from the trenches (Meet Magento Sweden 2016)Magento scalability from the trenches (Meet Magento Sweden 2016)
Magento scalability from the trenches (Meet Magento Sweden 2016)
 
ApacheCon NA 2010 - High Performance Cloud-enabled SCA Runtimes
ApacheCon NA 2010 - High Performance Cloud-enabled SCA RuntimesApacheCon NA 2010 - High Performance Cloud-enabled SCA Runtimes
ApacheCon NA 2010 - High Performance Cloud-enabled SCA Runtimes
 
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky  "How to manage hundreds of pipelines for processing da...Yaroslav Nedashkovsky  "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
 
Roy foubister (hosting high traffic sites on a tight budget)
Roy foubister (hosting high traffic sites on a tight budget)Roy foubister (hosting high traffic sites on a tight budget)
Roy foubister (hosting high traffic sites on a tight budget)
 
Scale Apache with Nginx
Scale Apache with NginxScale Apache with Nginx
Scale Apache with Nginx
 
PHP is the King, nodejs is the Prince and Lua is the fool
PHP is the King, nodejs is the Prince and Lua is the foolPHP is the King, nodejs is the Prince and Lua is the fool
PHP is the King, nodejs is the Prince and Lua is the fool
 
slides (PPT)
slides (PPT)slides (PPT)
slides (PPT)
 
murakumo Cloud Controller
murakumo Cloud Controllermurakumo Cloud Controller
murakumo Cloud Controller
 
Ceph Day Taipei - Bring Ceph to Enterprise
Ceph Day Taipei - Bring Ceph to EnterpriseCeph Day Taipei - Bring Ceph to Enterprise
Ceph Day Taipei - Bring Ceph to Enterprise
 
The next step from Microsoft - Vnext (Srdjan Poznic)
The next step from Microsoft - Vnext (Srdjan Poznic)The next step from Microsoft - Vnext (Srdjan Poznic)
The next step from Microsoft - Vnext (Srdjan Poznic)
 
Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)
Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)
Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)
 
uWSGI - Swiss army knife for your Python web apps
uWSGI - Swiss army knife for your Python web appsuWSGI - Swiss army knife for your Python web apps
uWSGI - Swiss army knife for your Python web apps
 
Paris.rb – 07/19 – Sidekiq scaling, workers vs processes
Paris.rb – 07/19 – Sidekiq scaling, workers vs processesParis.rb – 07/19 – Sidekiq scaling, workers vs processes
Paris.rb – 07/19 – Sidekiq scaling, workers vs processes
 
Ceph Day Bring Ceph To Enterprise
Ceph Day Bring Ceph To EnterpriseCeph Day Bring Ceph To Enterprise
Ceph Day Bring Ceph To Enterprise
 

More from Leo Lorieri

Deis, a PaaS built with Docker, Docker Meetup Sao Paulo #3 @Wayra
Deis, a PaaS built with Docker,  Docker Meetup Sao Paulo #3 @WayraDeis, a PaaS built with Docker,  Docker Meetup Sao Paulo #3 @Wayra
Deis, a PaaS built with Docker, Docker Meetup Sao Paulo #3 @WayraLeo Lorieri
 
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis OverviewLeo Lorieri
 
Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
Ceph, Docker, Heroku Slugs, CoreOS and Deis OverviewCeph, Docker, Heroku Slugs, CoreOS and Deis Overview
Ceph, Docker, Heroku Slugs, CoreOS and Deis OverviewLeo Lorieri
 
DevOps com Exemplos Práticos - QConRio 2014
DevOps com Exemplos Práticos - QConRio 2014DevOps com Exemplos Práticos - QConRio 2014
DevOps com Exemplos Práticos - QConRio 2014Leo Lorieri
 
História dos Sistemas - 3a Semana de SI
História dos Sistemas - 3a Semana de SIHistória dos Sistemas - 3a Semana de SI
História dos Sistemas - 3a Semana de SILeo Lorieri
 
História dos Sistemas - Por que estamos fazendo desse jeito
História dos Sistemas - Por que estamos fazendo desse jeitoHistória dos Sistemas - Por que estamos fazendo desse jeito
História dos Sistemas - Por que estamos fazendo desse jeitoLeo Lorieri
 
History of Computer Systems - Why we are doing it that way
History of Computer Systems - Why we are doing it that wayHistory of Computer Systems - Why we are doing it that way
History of Computer Systems - Why we are doing it that wayLeo Lorieri
 

More from Leo Lorieri (7)

Deis, a PaaS built with Docker, Docker Meetup Sao Paulo #3 @Wayra
Deis, a PaaS built with Docker,  Docker Meetup Sao Paulo #3 @WayraDeis, a PaaS built with Docker,  Docker Meetup Sao Paulo #3 @Wayra
Deis, a PaaS built with Docker, Docker Meetup Sao Paulo #3 @Wayra
 
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
 
Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
Ceph, Docker, Heroku Slugs, CoreOS and Deis OverviewCeph, Docker, Heroku Slugs, CoreOS and Deis Overview
Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
 
DevOps com Exemplos Práticos - QConRio 2014
DevOps com Exemplos Práticos - QConRio 2014DevOps com Exemplos Práticos - QConRio 2014
DevOps com Exemplos Práticos - QConRio 2014
 
História dos Sistemas - 3a Semana de SI
História dos Sistemas - 3a Semana de SIHistória dos Sistemas - 3a Semana de SI
História dos Sistemas - 3a Semana de SI
 
História dos Sistemas - Por que estamos fazendo desse jeito
História dos Sistemas - Por que estamos fazendo desse jeitoHistória dos Sistemas - Por que estamos fazendo desse jeito
História dos Sistemas - Por que estamos fazendo desse jeito
 
History of Computer Systems - Why we are doing it that way
History of Computer Systems - Why we are doing it that wayHistory of Computer Systems - Why we are doing it that way
History of Computer Systems - Why we are doing it that way
 

Recently uploaded

COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 

Recently uploaded (20)

COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 

Extending Piwik At R7.com

  • 1. Extending Piwik at r7.com Phase 1 – Collecting data Adding some cloud and modern scalability to a traditional LAMP stack leonardo lorieri, r7.com system architect, 'lorieri at gmail.com', feb/2012
  • 2. Why Piwik ?   - Open Source = flexible, understandable, free!   - Great interface   - Mobile app   - REST API   - Developers knows the market needs   - Efficient in small machines   - Lots of possible improvements   - Lots of improvements already in the roadmap   - Great and supportive community (Thank you all!)
  • 3. Our Plan, goals and trade-offs   - Don't change original code       - reduces development and maintenance costs   - Count only visits and page views       - to be fast and focused ( even though you still can use the .js tracker,         it is easy to get lost in the UI's beauty and all its functionalities)   - Handle odd unexpected traffic peaks       - from tv announcements    - Count not only websites       - media delivery, internal searches, debugs   - At least 99% of accuracy   - Have numbers to compare with other analytics tools   - We've lost P3P for now
  • 4.
  • 5. Regular Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes   - Apache/Nginx   - Php   - Mysql
  • 6. Bigger Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes   - Apache/Nginx   - Php   - MySql
  • 7. Regular Php Scaling Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes   - Apache/Nginx   - Php   - MySql Replication       (slave for backup only,       piwik is not &quot;slave ready&quot;) Load balancer/Nginx
  • 8.
  • 9. Asynchronous Piwik Setup based on Rodrigo Campos presentation http://www.slideshare.net/xinu/capacity-planning-for-linux-systes   - Nginx   - NOT even Php   - MySql Master   - Apache+Php for Admin UI   - Archive cron Load balancer/Nginx   - MySql Slave   - Perl/Python worker to     process logs (manages user cookies) (user cookie) - accesses logs Visits REST API <img src=> request Admin/ Reports
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17. Our setup diagram Visits ELB Elastic Load Balancer nginx autoscaling pool S3 bucket SNS Notifications SQS queues Other workers/processors for other projects worker BigAss MySql mysql connection mysql slave, apache, piwik api, python-boto, python-twisted mysql master, piwik Piwik Users one file per virtualhost per machine, for each 5 minutes one notification per s3 file Datacenter
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.