Correlate Log Data with Business Metrics Like a Jedi
•Download as PPTX, PDF•
3 likes•914 views
The Logentries and Hosted Graphite integration allows you to connect two of your favorite Ops tools to easily extract important data from your log files, visualize them as metrics, and share them in Hosted Graphite dashboards. • Integrate your systems to extract the metrics you need, from both your applications and log data. • Set-up log metric dashboards based on common use cases (e.g. error tracking, performance, app usage). • Get off the "complexity elevator" of hosting your own in-house logging or graphite solutions. • Delight your team and organization with valuable metrics and performance insights.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Correlate Log Data with Business Metrics Like a Jedi
Similar to Correlate Log Data with Business Metrics Like a Jedi (20)
Recently uploaded
Recently uploaded (20)
Correlate Log Data with Business Metrics Like a Jedi
- 1. by RevelOps How to Correlate Log Data with Business Metrics Like a Jedi Oct 22nd, 2014
- 2. by RevelOps Housekeeping • Webinar is being recorded, the archive will be emailed to you • You can ask questions via the chat pane • Follow us @Logentries and @HostedGraphite
- 3. by RevelOps Meet the Hosts Dave Concannon Hosted Graphite CEO & Co-founder Trevor Parsons Logentries Co-founder and Chief Scientist
- 4. by RevelOps Outline • Introduction: Logentries & Hosted Graphite • Logs and Metrics: Correlation is King! • What vs. Why • Collecting data from every layer in your software stack • Turning Logs into Metrics • Sending Metrics to Graphite • Real Time Metrics with Graphite • Use Case: Application Performance Monitoring w/ Logentries & Hosted Graphite • Rolling your own Logging and Graphite Solutions: Pros and Cons • Q & A
- 5. by RevelOps A Heritage in Advanced Research •Founders hold PhDs in Distributed Systems •Former Scientists at the IBM Center for Advanced Studies •Built Log Management Solution for IBM (RTCE) Global SaaS Log Management Service •More than 1,000 customers in over 100 countries, processing over 10B events daily •Locations in Boston, Dublin, Prague •Raised more than $11M in venture funding About Logentries Leadership Team President & CEO Andrew Burton Proven software executive, with over 16 years of experience in start-ups & enterprise companies. Co-founder, Chief Scientist Trevor Parsons Formerly a scientist at IBM's Center for advanced studies with a PhD in Computer Science. VP, Engineering Brendan Dillon Experienced software engineering & security executive, formerly CTO at Adaptive Mobile. Viliam Holub Over a decade developing enterprise software, with a PhD in Computer Science. Co-founder, CTO VP, Marketing Leigh Merrigan SaaS marketing, communications and branding executive, formerly with AppNeta.
- 6. by RevelOps About Hosted Graphite Team Charlie Von Metzradt Hacker and tinkerer with experience in running very large metric collection systems Rob McAdoo PhD. Computer science Dan Fox Ops Engineer Ex-Amazon ops engineer Cian Synnott Ex-google data wrangler Co-founder Years of measuring things in a variety of companies Co-founder Engineer Data Scientist Dave Concannon Emmanuel Stone Experienced startup engineer Engineer Years of dealing with metric data •Founders worked with high-volume data that measured games, like “Call of Duty” •Broad experience across many industries and company sizes SaaS Metrics Service •Hundreds of customers, hundreds of countries, billions of data points daily •Locations in Dublin and Indiana •Bootstrapped with blood, sweat, and tears :) Hugh Nowlan Security Engineer
- 7. by RevelOps Logs & Metrics: Correlation is King
- 8. by RevelOps Logs & Metrics: Correlation is King Question •What Happened? Answer •Things got slow!
- 9. by RevelOps Logs & Metrics: Correlation is King Question •Why? Answer •Correlation is King!
- 10. by RevelOps Logs & Metrics: Correlation is King • Response Time Increasing AND CPU Load Increasing
- 11. by RevelOps Logs & Metrics: Correlation is King Data point 1 •Number of Users CPU and Response Time Solution •Need more servers
- 12. by RevelOps Logs & Metrics: Correlation is King Data point 2 •Java Exceptions CPU and Response Time Solution •Fix the Code
- 13. by RevelOps Logs & Metrics: Correlation is King Data point 3 •Slow Queries CPU and Response Time Solution •Tune the DB Logs maintain the evidence
- 14. by RevelOps Data Sources What you can typically get from logs • OS Level • Syslog • Windows Event Logs • Server Resource Usage • Middleware: • Web Servers • App Servers • Databases • Message Queues • Application Logs • Routers, Firewalls • APIs: • E.g. Cloudwatch, CloudTrail • Client Side: • Web Browser • Mobile Apps
- 15. by RevelOps Data Sources What you will typically send to Graphite: • Server Level: • CPU Load • Network IO • Disk IO • Memory usage • Application data: • Timing / Measurement of specific functions or functional areas • Overall request time and frequency • How often the out-of-memory killer kills a process • Security information: • Number of SSH login attempts - failed and succeeded • Sensor Data: • Temperature / Pressure / Flow rates • Transmission rates • APIs: • E.g. Cloudwatch • Client Side: • Web Browser
- 16. by RevelOps Getting Metrics from Logs
- 17. by RevelOps Getting Metrics from Logs
- 18. by RevelOps Getting Metrics from Logs
- 19. by RevelOps Getting Metrics from Logs to Hosted Graphite
- 20. by RevelOps How to work with Metrics in Hosted Graphite? StatsD - Pre-aggregator for metric data • We also provide Hosted StatsD! • Support for counts / gauges / percentile data Diamond • A useful server agent to collect system data (CPU/Memory/etc) Aggregating hundreds or thousands of sources into a single metric • Power in percentile data • 90th percentile as an indicator of general health - Averages lie! • Dig in to specific problems after seeing a quick overview • Avoids “twitchy” monitoring Data views • Sums / rates / observations, no need to define views beforehand
- 21. by RevelOps Typical Use Case: DevOps Dashboard Logs Metrics Graphite OS logs & Server Metrics: • SSH Acccess, • CPU, Mem… Web Server Logs: • Response time App Server Logs: • App Exceptions Client Side Logs: • Response time (client side) • Url & Usage tracking
- 22. by RevelOps Typical Use Case: DevOps Dashboard Metrics Graphite Application Metrics: • Code Latency • Query Time to DB • Throughput Diamond: • Collectors… Hadoop, Mongo, Kafka, MySQL, NetApp, RabbitMQ, Redis, AWS S3… • Check out:https://github.com/BrightcoveOS/Diamond/wiki /Collectors
- 23. by RevelOps Typical Use Case: DevOps Dashboard Some questions you can answer: Application Performance Monitoring: •Is User experience ok? • Client side response time • HTTP 404s/500s? • Query time Server Capacity: •CPU vs user load Security: •Failed Logins vs. unusual system behavior
- 24. by RevelOps Rolling your own Logging/Metrics Great open source tools e.g. Logstash & Graphite •They are free… • No license/monthly billing etc. •They are not free… • 250 GB per day, 8 servers on AWS, ~$4k per month • Maintenance, few hours per week, especially when you have a clustered server environment • Downtime can occur as volumes grow, if no autoscaling in place • Log volumes ALWAYS grow… •Some more reading: • https://blog.logentries.com/2014/09/the-pros-and-cons-of-open-source-logging/
- 25. by RevelOps Give it a Try for Free • Get started in minutes with the free Logentries service! • https://logentries.com/quick-start • Free 14 Day trial of Hosted Graphite • https://www.hostedgraphite.com/accounts/register/ • Logentries & Hosted Graphite: https://blog.logentries.com/2014/10/connecting-logs-and-metrics-logentries-and-hosted- graphite/ • Contact us directly! • Trevor.parsons@logentries.com • support@logentries.com • dave@hostedgraphite.com, help@hostedgraphite.com • Let’s take a few questions!
- 26. by RevelOps Oct 2014