Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Monitoring at section.io
Operational visibility for both the platform and our users
• Runs on your local machine and pre-production
• Configuration and deployment via git
• Fast global cache management
• HT...
• Integrates with popular open-source
• API driven
• Near real-time log access
• Consistent operational interface
Open pla...
• Delivery Proxies
• Varnish Cache
• ModSecurity
• Kibana
• Graphite
• Umpire
Containers
• Web access logs, syslog, performance data
• Docker Volumes
• Elastic Beats
• Log rotation
Gathering data
• 600 million web access logs per week
• 60,000 log entries processed per minute
• 7 days of logs are searchable
Log volume
Log flow
Delivery
networks
Logstash
receivers redis
Logstash
processors
Logstash
senders
redis
Ops Elasticsearch
cluster
A...
• Kibana
• Elasticsearch API
• Traces
Log visibility
• Metrics can optimise common log queries
• Metrics retention:
• 1 minute granularity for 1 month
• 1 hour granularity for...
• CPU utilisation, memory usage, disk space
• Traffic: connections, requests, packets, bytes
• By partition, node, geo-reg...
• Cache hit, miss, pass
• By content-type
• Response time (median, mean, upper 95%)
• WAF intercepts
• By rule
• By countr...
• Every staff member does on-call
• Every alert is actionable
• Every incident feeds the product backlog
Internal processes
• Yelp Elastalert
• Custom log fields
• A `tail -f` UI
• Automated anomaly detection
Beyond today
Jason Stangroome
Twitter: @jstangroome
https://blog.stangroome.com
https://www.section.io/blog
Thank you
Upcoming SlideShare
Loading in …5
×

Monitoring at section.io - Operational Intelligence Meetup May 2016

377 views

Published on

http://www.meetup.com/Operational-Intelligence/events/230955872/

Published in: Technology
  • Be the first to comment

Monitoring at section.io - Operational Intelligence Meetup May 2016

  1. 1. Monitoring at section.io Operational visibility for both the platform and our users
  2. 2. • Runs on your local machine and pre-production • Configuration and deployment via git • Fast global cache management • HTTPS and HTTP/2 by default A modern CDN
  3. 3. • Integrates with popular open-source • API driven • Near real-time log access • Consistent operational interface Open platform
  4. 4. • Delivery Proxies • Varnish Cache • ModSecurity • Kibana • Graphite • Umpire Containers
  5. 5. • Web access logs, syslog, performance data • Docker Volumes • Elastic Beats • Log rotation Gathering data
  6. 6. • 600 million web access logs per week • 60,000 log entries processed per minute • 7 days of logs are searchable Log volume
  7. 7. Log flow Delivery networks Logstash receivers redis Logstash processors Logstash senders redis Ops Elasticsearch cluster Apps Elasticsearch cluster StatsD, Carbon Between about 5 seconds and 2 minutes
  8. 8. • Kibana • Elasticsearch API • Traces Log visibility
  9. 9. • Metrics can optimise common log queries • Metrics retention: • 1 minute granularity for 1 month • 1 hour granularity for 13 months • Graphite, Tessera, and Grafana • Heroku Umpire Beyond logs
  10. 10. • CPU utilisation, memory usage, disk space • Traffic: connections, requests, packets, bytes • By partition, node, geo-region, and domain • By HTTP response status code • Log latency, queue depth, processing rate • Message counts, errors, processing time Platform monitoring
  11. 11. • Cache hit, miss, pass • By content-type • Response time (median, mean, upper 95%) • WAF intercepts • By rule • By country Website monitoring
  12. 12. • Every staff member does on-call • Every alert is actionable • Every incident feeds the product backlog Internal processes
  13. 13. • Yelp Elastalert • Custom log fields • A `tail -f` UI • Automated anomaly detection Beyond today
  14. 14. Jason Stangroome Twitter: @jstangroome https://blog.stangroome.com https://www.section.io/blog Thank you

×