SlideShare a Scribd company logo
1 of 63
Download to read offline
Take my logs. Please.

Mike Brittain
Director of Engineering, Infrastructure
Etsy.com

mike@etsy.com      @mikebrittain
(hello?)
This sounds
boooooorrrrring...
No, no... hang in there!
25 MM uniques/month
150 Countries
$300 MM+ sales last year
Apache, PHP, MySQL,
PostgreSQL,
Memcache, Gearman,
Solr, etc.
What’s working?
What’s working?
Performance
What’s working?
Performance
Operability
What’s working?
Performance
Operability
Simplicity
Logging + Trending
App logging
(Apache access and error logs)
“Common”
LogFormat "%h %l %u %t
 "%r" %>s %b
“Combined”
LogFormat "%h %l %u %t
 "%r" %>s %b
 "%{Referer}i"
 "%{User-agent}i""
mod_log_config
  %f         Filename requested

       # of keepalive requests served
  %k         on this connection
       Time taken to serve the request,
  %T              in seconds
mod_log_config
  %f         Filename requested

       # of keepalive requests served
  %k         on this connection



       Time taken to serve the request,
  %D           in microseconds
mod_log_config
   %f              Filename requested

             # of keepalive requests served
   %k              on this connection



             Time taken to serve the request,
   %D                in microseconds
             Contents of “note” foobar from
%{foobar}n           another module
apache_note()
apache_note(“foobar”, $whatever);
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
$GLOBALS['timer'] = microtime(true) * 1000000;
$GLOBALS['timer'] = microtime(true) * 1000000;

register_shutdown_function('pageStats');

function pageStats() {




}
$GLOBALS['timer'] = microtime(true) * 1000000;

register_shutdown_function('pageStats');

function pageStats() {
  $timer_end = microtime(true) * 1000000;
  $diff = $timer_end - $GLOBALS['timer'];




}
$GLOBALS['timer'] = microtime(true) * 1000000;

register_shutdown_function('pageStats');

function pageStats() {
  $timer_end = microtime(true) * 1000000;
  $diff = $timer_end - $GLOBALS['timer'];

    apache_note('php_microsec', $diff);

    apache_note('php_bytes',
                memory_get_peak_usage());
}
What about “%D”?
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n ...

easy_reg=1; personalize_widget=0;
icon_in_cornflower_blue=1;
Coming soon...
%{locale}n     (i18n)

%{platform}n   (desktop vs. mobile)
Coming soon...
%{locale}n     (i18n)

%{platform}n   (desktop vs. mobile)



OPS-1805, OPS-1827
etsy.com/careers
Using something else?
time, http method, request uri,
response code, referer, user-agent,
response time, response memory,
custom segmentation fields...
Quick averages
grep "GET /listing/" access.log | 

awk '{sum=sum+$(NF-1)} END {print sum/NR}'
Quick graphs
grep "GET /listing/" access.log | 

perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" | 

awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat




gives you...
Quick graphs
# /tmp/pagetimes.dat

18:37   251.0
18:38   252.1
18:39   253.5
18:40   251.0
18:45   250.0


and then...
Quick graphs
# GNUPLOT

set terminal png
set output 'listings.png'
set yrange [0:2000]
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot '/tmp/pagetimes.dat' using 1:2 with points
Quick graphs
Error logs
PHP + Apache errors in one file
Simple logging interface
Error logs
Levels: error, info, debug
Namespace: perf, sql, __class__
Logger::error("Query exceeded 5 sec: $query",

             “sql_long_query”);
web0054 [Fri Mar 04 16:27:48 2011] [error]

[sql_long_query] [mk04gw1p71] Query exceeded

5 sec: SELECT * FROM ...
web0054 [Fri Mar 04 16:27:48 2011] [error]

[sql_long_query] [mk04gw1p71] Query exceeded

5 sec: SELECT * FROM ...
$ grep "16:27:48" access.log | wc -l

1527
web0054 [Fri Mar 04 16:27:48 2011] [error]

[sql_long_query] [mk04gw1p71] Query exceeded

5 sec: SELECT * FROM ...
iow
error.log -> request_uid -> access.log
request uri, ab selections, user id, locale,
platform, api key, etc.
Filtering
tail -f error.log | grep -v “sql_long_query” | ...
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Help me, Rhonda.
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Oh noooooo!
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Heeeeeeellllllllllllllppp
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Oh noooooo!
web0001   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201   [04:28:54   2011]   [warning] [client 10.101.x.x] Gaaaaahhh!
web0034   [04:28:54   2011]   [warning] [client 10.101.x.x] Oh nooooooooooo
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201   [04:28:54   2011]   [error] [client 10.101.x.x] You've been eaten by a gr
web0055   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002   [04:28:54   2011]   [warning] [client 10.101.x.x] Sky is falling.
web0089   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020   [04:28:54   2011]   [error] [client 10.101.x.x] Sky is falling.
web1101   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055   [04:28:54   2011]   [warning] [client 10.101.x.x] Gaaaaahhh!
web0001   [04:28:54   2011]   [warning] [client 10.101.x.x] Oh nooooooooooo
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0034   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0087   [04:28:54   2011]   [fatal] [client 10.101.x.x] Sky is falling.
web0002   [04:28:54   2011]   [error] [client 10.101.x.x] Oh noooooo!
web0201   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!
web0077   [04:28:54   2011]   [warning] [client 10.101.x.x] Gaaaaahhh!
web0355   [04:28:54   2011]   [warning] [client 10.101.x.x] Oh nooooooooooo
web0052   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0003   [04:28:54   2011]   [error] [client 10.101.x.x] You've been eaten by a gr
web0066   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0001   [04:28:54   2011]   [warning] [client 10.101.x.x] Sky is falling
web0020   [04:28:54   2011]   [error] [client 10.101.x.x] Sky is falling.
web1101   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055   [04:28:54   2011]   [warning] [client 10.101.x.x] Gaaaaahhh!
web0001   [04:28:54   2011]   [warning] [client 10.101.x.x] Oh nooooooooooo
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
Trending
fatals   errors   warnings
Logster
Run by cron
Maintains a cursor on log files
Simple parsing & aggregation
Output to Ganglia or Graphite

                          github.com/etsy
web0054 [Fri Mar 04 16:27:48 2011] [error]
 [login] [mk04gw1p71] User login failed.
      Reason: wrong password for ...
^.+ [.+] [(?P<log_level>.+)]
if (fields['log_level'] == “fatal”):
   self.fatals += 1

elif (fields['log_level'] == “error”):
   self.errors += 1

elif (fields['log_level'] == “warning”):
   self.warnings += 1

...
MetricObject("fatals",
  (self.fatals / self.duration), "per sec")

MetricObject("errors",
  (self.errors / self.duration), "per sec")

MetricObject("warning",
  (self.warnings / self.duration), "per sec")
fatals   errors   warnings
Logster

          Signed-in vs. Signed-out
github.com/etsy
Log a plethora of data.
Don’t be afraid to use one file.
Use custom fields to segment data.
Correlate errors to specific requests.
Make f#@k!ng graphs.
Convert rates to trend lines.
Take my logs. Please!
Thank you.
                  codeascraft.etsy.com
                  github.com/etsy

Mike Brittain
Director of Engineering, Infrastructure
Etsy.com

mike@etsy.com      @mikebrittain

More Related Content

What's hot

Real World Dependency Injection - oscon13
Real World Dependency Injection - oscon13Real World Dependency Injection - oscon13
Real World Dependency Injection - oscon13
Stephan Hochdörfer
 
Rails 3: Dashing to the Finish
Rails 3: Dashing to the FinishRails 3: Dashing to the Finish
Rails 3: Dashing to the Finish
Yehuda Katz
 
Phing for power users - dpc_uncon13
Phing for power users - dpc_uncon13Phing for power users - dpc_uncon13
Phing for power users - dpc_uncon13
Stephan Hochdörfer
 
Your Business. Your Language. Your Code - dpc13
Your Business. Your Language. Your Code - dpc13Your Business. Your Language. Your Code - dpc13
Your Business. Your Language. Your Code - dpc13
Stephan Hochdörfer
 
The Best (and Worst) of Django
The Best (and Worst) of DjangoThe Best (and Worst) of Django
The Best (and Worst) of Django
Jacob Kaplan-Moss
 

What's hot (20)

Using Task Queues and D3.js to build an analytics product on App Engine
Using Task Queues and D3.js to build an analytics product on App EngineUsing Task Queues and D3.js to build an analytics product on App Engine
Using Task Queues and D3.js to build an analytics product on App Engine
 
Anatomy of an Addon Ecosystem - EmberConf 2019
Anatomy of an Addon Ecosystem - EmberConf 2019Anatomy of an Addon Ecosystem - EmberConf 2019
Anatomy of an Addon Ecosystem - EmberConf 2019
 
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]
 
Oracle APEX Performance
Oracle APEX PerformanceOracle APEX Performance
Oracle APEX Performance
 
PHPUnit Episode iv.iii: Return of the tests
PHPUnit Episode iv.iii: Return of the testsPHPUnit Episode iv.iii: Return of the tests
PHPUnit Episode iv.iii: Return of the tests
 
Real World Dependency Injection - oscon13
Real World Dependency Injection - oscon13Real World Dependency Injection - oscon13
Real World Dependency Injection - oscon13
 
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...
 
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...
 
Rails 3: Dashing to the Finish
Rails 3: Dashing to the FinishRails 3: Dashing to the Finish
Rails 3: Dashing to the Finish
 
KISS Automation.py
KISS Automation.pyKISS Automation.py
KISS Automation.py
 
Phing for power users - dpc_uncon13
Phing for power users - dpc_uncon13Phing for power users - dpc_uncon13
Phing for power users - dpc_uncon13
 
A Journey with React
A Journey with ReactA Journey with React
A Journey with React
 
Incremental Type Safety in React Apollo
Incremental Type Safety in React Apollo Incremental Type Safety in React Apollo
Incremental Type Safety in React Apollo
 
You do not need automation engineer - Sqa Days - 2015 - EN
You do not need automation engineer  - Sqa Days - 2015 - ENYou do not need automation engineer  - Sqa Days - 2015 - EN
You do not need automation engineer - Sqa Days - 2015 - EN
 
Your Business. Your Language. Your Code - dpc13
Your Business. Your Language. Your Code - dpc13Your Business. Your Language. Your Code - dpc13
Your Business. Your Language. Your Code - dpc13
 
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)
 
Bowtie: Interactive Dashboards
Bowtie: Interactive DashboardsBowtie: Interactive Dashboards
Bowtie: Interactive Dashboards
 
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014Hardcore URL Routing for WordPress - WordCamp Atlanta 2014
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014
 
The Best (and Worst) of Django
The Best (and Worst) of DjangoThe Best (and Worst) of Django
The Best (and Worst) of Django
 
GAEO
GAEOGAEO
GAEO
 

Viewers also liked

Simple Log Analysis and Trending
Simple Log Analysis and TrendingSimple Log Analysis and Trending
Simple Log Analysis and Trending
Mike Brittain
 
On Failure and Resilience
On Failure and ResilienceOn Failure and Resilience
On Failure and Resilience
Mike Brittain
 

Viewers also liked (10)

How to Get to Second Base with Your CDN
How to Get to Second Base with Your CDNHow to Get to Second Base with Your CDN
How to Get to Second Base with Your CDN
 
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCContinuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYC
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty Details
 
Simple Log Analysis and Trending
Simple Log Analysis and TrendingSimple Log Analysis and Trending
Simple Log Analysis and Trending
 
On Failure and Resilience
On Failure and ResilienceOn Failure and Resilience
On Failure and Resilience
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty Details
 
From Building a Marketplace to Building Teams
From Building a Marketplace to Building TeamsFrom Building a Marketplace to Building Teams
From Building a Marketplace to Building Teams
 
The Real Life Social Network v2
The Real Life Social Network v2The Real Life Social Network v2
The Real Life Social Network v2
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
 
26 Disruptive & Technology Trends 2016 - 2018
26 Disruptive & Technology Trends 2016 - 201826 Disruptive & Technology Trends 2016 - 2018
26 Disruptive & Technology Trends 2016 - 2018
 

Similar to Take My Logs. Please!

The state of Symfony2 - SymfonyDay 2010
The state of Symfony2 - SymfonyDay 2010The state of Symfony2 - SymfonyDay 2010
The state of Symfony2 - SymfonyDay 2010
Fabien Potencier
 
Joomla! security 101
Joomla! security 101Joomla! security 101
Joomla! security 101
vdrover
 
Action Controller Overview, Season 2
Action Controller Overview, Season 2Action Controller Overview, Season 2
Action Controller Overview, Season 2
RORLAB
 
idea: talk about the Active Cache
idea: talk about the Active Cacheidea: talk about the Active Cache
idea: talk about the Active Cache
Ching Yi Chan
 

Similar to Take My Logs. Please! (20)

The state of Symfony2 - SymfonyDay 2010
The state of Symfony2 - SymfonyDay 2010The state of Symfony2 - SymfonyDay 2010
The state of Symfony2 - SymfonyDay 2010
 
Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기
 
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...
 
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac...
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac..."Full Stack frameworks or a story about how to reconcile Front (good) and Bac...
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac...
 
Joomla! security 101
Joomla! security 101Joomla! security 101
Joomla! security 101
 
Action Controller Overview, Season 2
Action Controller Overview, Season 2Action Controller Overview, Season 2
Action Controller Overview, Season 2
 
Introducing Neo4j 3.1: New Security and Clustering Architecture
Introducing Neo4j 3.1: New Security and Clustering Architecture Introducing Neo4j 3.1: New Security and Clustering Architecture
Introducing Neo4j 3.1: New Security and Clustering Architecture
 
Building Cloud Castles
Building Cloud CastlesBuilding Cloud Castles
Building Cloud Castles
 
OWASP Top 10 - DrupalCon Amsterdam 2019
OWASP Top 10 - DrupalCon Amsterdam 2019OWASP Top 10 - DrupalCon Amsterdam 2019
OWASP Top 10 - DrupalCon Amsterdam 2019
 
Re-Design with Elixir/OTP
Re-Design with Elixir/OTPRe-Design with Elixir/OTP
Re-Design with Elixir/OTP
 
idea: talk about the Active Cache
idea: talk about the Active Cacheidea: talk about the Active Cache
idea: talk about the Active Cache
 
What's new in Rails 4
What's new in Rails 4What's new in Rails 4
What's new in Rails 4
 
Neo4j Network security
Neo4j Network securityNeo4j Network security
Neo4j Network security
 
Brian hogg word camp preparing a plugin for translation
Brian hogg   word camp preparing a plugin for translationBrian hogg   word camp preparing a plugin for translation
Brian hogg word camp preparing a plugin for translation
 
Malware Detection with OSSEC HIDS - OSSECCON 2014
Malware Detection with OSSEC HIDS - OSSECCON 2014Malware Detection with OSSEC HIDS - OSSECCON 2014
Malware Detection with OSSEC HIDS - OSSECCON 2014
 
Symfony 1, mi viejo amigo
Symfony 1, mi viejo amigoSymfony 1, mi viejo amigo
Symfony 1, mi viejo amigo
 
Preparing a WordPress Plugin for Translation
Preparing a WordPress Plugin for TranslationPreparing a WordPress Plugin for Translation
Preparing a WordPress Plugin for Translation
 
WebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerWebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan Wintermeyer
 
Review unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphpReview unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphp
 
Workshop quality assurance for php projects tek12
Workshop quality assurance for php projects tek12Workshop quality assurance for php projects tek12
Workshop quality assurance for php projects tek12
 

Recently uploaded

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 

Recently uploaded (20)

How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistan
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهله
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 

Take My Logs. Please!

  • 1. Take my logs. Please. Mike Brittain Director of Engineering, Infrastructure Etsy.com mike@etsy.com @mikebrittain
  • 4.
  • 5. 25 MM uniques/month 150 Countries $300 MM+ sales last year
  • 12. App logging (Apache access and error logs)
  • 13. “Common” LogFormat "%h %l %u %t "%r" %>s %b
  • 14. “Combined” LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i""
  • 15. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %T in seconds
  • 16. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %D in microseconds
  • 17. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %D in microseconds Contents of “note” foobar from %{foobar}n another module
  • 19. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 20. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 22. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { }
  • 23. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { $timer_end = microtime(true) * 1000000; $diff = $timer_end - $GLOBALS['timer']; }
  • 24. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { $timer_end = microtime(true) * 1000000; $diff = $timer_end - $GLOBALS['timer']; apache_note('php_microsec', $diff); apache_note('php_bytes', memory_get_peak_usage()); }
  • 26. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 27. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 28. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 29. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n ... easy_reg=1; personalize_widget=0; icon_in_cornflower_blue=1;
  • 30. Coming soon... %{locale}n (i18n) %{platform}n (desktop vs. mobile)
  • 31. Coming soon... %{locale}n (i18n) %{platform}n (desktop vs. mobile) OPS-1805, OPS-1827 etsy.com/careers
  • 32. Using something else? time, http method, request uri, response code, referer, user-agent, response time, response memory, custom segmentation fields...
  • 33. Quick averages grep "GET /listing/" access.log | awk '{sum=sum+$(NF-1)} END {print sum/NR}'
  • 34. Quick graphs grep "GET /listing/" access.log | perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" | awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat gives you...
  • 35. Quick graphs # /tmp/pagetimes.dat 18:37 251.0 18:38 252.1 18:39 253.5 18:40 251.0 18:45 250.0 and then...
  • 36. Quick graphs # GNUPLOT set terminal png set output 'listings.png' set yrange [0:2000] set xdata time set timefmt "%d/%B/%Y:%H:%M:%S" set format x "%H:%M" plot '/tmp/pagetimes.dat' using 1:2 with points
  • 38. Error logs PHP + Apache errors in one file Simple logging interface
  • 39. Error logs Levels: error, info, debug Namespace: perf, sql, __class__
  • 40. Logger::error("Query exceeded 5 sec: $query", “sql_long_query”);
  • 41. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
  • 42. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
  • 43. $ grep "16:27:48" access.log | wc -l 1527
  • 44. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
  • 45. iow error.log -> request_uid -> access.log request uri, ab selections, user id, locale, platform, api key, etc.
  • 46. Filtering tail -f error.log | grep -v “sql_long_query” | ...
  • 47. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppp web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!! web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling. web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling. web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling. web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling. web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
  • 48. Trending fatals errors warnings
  • 49. Logster Run by cron Maintains a cursor on log files Simple parsing & aggregation Output to Ganglia or Graphite github.com/etsy
  • 50. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] User login failed. Reason: wrong password for ...
  • 52. if (fields['log_level'] == “fatal”): self.fatals += 1 elif (fields['log_level'] == “error”): self.errors += 1 elif (fields['log_level'] == “warning”): self.warnings += 1 ...
  • 53. MetricObject("fatals", (self.fatals / self.duration), "per sec") MetricObject("errors", (self.errors / self.duration), "per sec") MetricObject("warning", (self.warnings / self.duration), "per sec")
  • 54. fatals errors warnings
  • 55. Logster Signed-in vs. Signed-out
  • 57. Log a plethora of data. Don’t be afraid to use one file.
  • 58. Use custom fields to segment data.
  • 59. Correlate errors to specific requests.
  • 61. Convert rates to trend lines.
  • 62. Take my logs. Please!
  • 63. Thank you. codeascraft.etsy.com github.com/etsy Mike Brittain Director of Engineering, Infrastructure Etsy.com mike@etsy.com @mikebrittain