Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Achieving
Common Ground via
Logging
Second Class Citizens
Why?
Why?
(?:(?:(?:b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|
Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct...
All good?
Yes. But...
Dev
Ops
Support
Products
Where to begin?
“Request Completed”

Masterzen - 10 Commandments of Logging
Context
Request (f67a) completed for user
George-Michael (token: weqeqweq)
to fakeblock in 3.16275 seconds.
Context
{

}

"msg": "Request completed",
"runtime_s": 3.16275,
"request": {
"action": "fakeblock",
"accessToken": "yjfhak...
Context + Correlation
{

{
...
"host": "load-balancer",
"status": "200",
"request": "POST /url/ HTTP/1.1",
"http_user_agen...
Ship them!
level : ‘error’
... and class : ‘mailer’
... and host : ‘i-8904’
action : ‘fakeblock’
... and runtime > 2.1
level : ‘error’
... and ...
... and ...
is count > threshold
Critical, Warn, OK?
Wait a sec...
{

}

"msg": "Request completed",
"runtime_s": 3.16275,
"request": {
"action": "fakeblock",
"accessToken": "...
type : ‘user_log’
and
user : ‘George-Michael’
“invalid identifier returned”
But we are a small
team...
Defaults work well
• 40 GB index/day
• 3k events/sec indexed
• m1.xlarge x 2
• Chef Cookbook available
• pyes (ES Api in p...
Bonus
Thanks
Manas Gupta
@ngrep_q
Upcoming SlideShare
Loading in …5
×

5 manas gupta - achieving common ground via logging

1,905 views

Published on

Achieving Common Ground via Logging

Slides from - http://devopsdays.org/events/2013-newyork/proposals/Achieving%20common%20ground%20via%20Logging/

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

5 manas gupta - achieving common ground via logging

  1. 1. Achieving Common Ground via Logging
  2. 2. Second Class Citizens
  3. 3. Why?
  4. 4. Why? (?:(?:(?:b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?| Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)b) +(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?:(?![0-9])(?:(?:2[0123]|[01] [0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?:(?:(?:b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-] {0,62}))*(.?|b))|(?(?![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?: 25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}) [.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?(?(?:[w._/%-]+))(?: [(?b(?:[1-9][0-9]*)b)])?): (?(?![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9] {1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]? [0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?(?:[+-]?(?: [0-9]+))) [(?(?(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?b(?:Jan(?:uary)?| Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?| Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)b)/(?[0-9]+):(?(?![0-9]) (?(?:2[0123]|[01][0-9])):(?(?:[0-5][0-9]))(?::(?(?:(?:[0-5][0-9]|60)(?:[.,] [0-9]+)?)))(?![0-9])).(?(?:[+-]?(?:[0-9]+))))] (?S+) (?S+)/(?S+) (?(?:[+-]?(?: [0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?S+) (?S+) (?S+) (?(?:[+-]?(?:[0-9]+)))/(?(?: [+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?S+) (?(?:[+-]? (?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))) Thank you @jordansissel
  5. 5. All good? Yes. But... Dev Ops Support Products
  6. 6. Where to begin? “Request Completed” Masterzen - 10 Commandments of Logging
  7. 7. Context Request (f67a) completed for user George-Michael (token: weqeqweq) to fakeblock in 3.16275 seconds.
  8. 8. Context { } "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "fakeblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a"
  9. 9. Context + Correlation { { ... "host": "load-balancer", "status": "200", "request": "POST /url/ HTTP/1.1", "http_user_agent": "iphone", "x_requestId": "f67a", "request_time": 3.9 .... "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "faceblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a", "host": "i-90zas2", } }
  10. 10. Ship them!
  11. 11. level : ‘error’ ... and class : ‘mailer’ ... and host : ‘i-8904’
  12. 12. action : ‘fakeblock’ ... and runtime > 2.1
  13. 13. level : ‘error’ ... and ... ... and ... is count > threshold Critical, Warn, OK?
  14. 14. Wait a sec... { } "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "fakeblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a"
  15. 15. type : ‘user_log’ and user : ‘George-Michael’
  16. 16. “invalid identifier returned”
  17. 17. But we are a small team...
  18. 18. Defaults work well • 40 GB index/day • 3k events/sec indexed • m1.xlarge x 2 • Chef Cookbook available • pyes (ES Api in py)
  19. 19. Bonus
  20. 20. Thanks Manas Gupta @ngrep_q

×