Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Scale Your Architecture and DevOps Practices for Big Data Applications

622 views

Published on

by Manoj Chaudhary, CTO & VP of Engineering at Loggly

  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

How to Scale Your Architecture and DevOps Practices for Big Data Applications

  1. 1. 1© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 How to Scale Your Architecture and DevOps Practices for Big Data Applications Manoj Chaudhary CTO and VP of Engineering March 30, 2017
  2. 2. 2© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 •  Cloud-based log management •  Founded in 2009 •  Based in San Francisco •  10,000+ customers •  Startups to Fortune 500
  3. 3. 3© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 How log management starts Management = simple stuff •  Rotate files, compress and delete •  Scan for specific events (not easy) •  Log retention policies evolve over time
  4. 4. 4© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Log volume Self-InflictedPain “…hmmm, our logs are getting a bit bloated” “…let’s spend time managing our log capacity” “…how can I make this someone else’s problem!” As Log Data Grows As log data grows
  5. 5. 5© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Focus on managing application vs. managing logs from the trenches… If you get a disk space alert, first login… % sudo rm –rf /var/log/apache2/* Admit it, we’ve all seen this kind of thing! Do not make this is an either/or proposition! !
  6. 6. 6© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 We believe the future is all about increasingly complex systems, and companies need better ways to be able to understand them.
  7. 7. 7© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 The way we make complex systems understandable is by making it ridiculously easy to reveal the hidden stories in your log data.
  8. 8. 8© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Loggly makes log management easy and simple •  Cloud-based SaaS for easy central log collection, aggregation, management •  Agent-free, easy setup •  Uses open or industry standards and native log transfer capabilities (example: syslog) •  No dependency on / maintenance effort from proprietary agents •  54 logging technologies supported out of the box, list growing •  Setup Wizard or optionally fully manual configuration with full control and transparency
  9. 9. 9© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Dynamic parsing •  Parses data in real-time at ingestion time, auto-recognizes common log formats, but also supports custom parsing rules •  Automatically breaks data down into meaningful groups, fields, values that can be mouse-navigated in a menu- style structure (Dynamic Field Explorer™) •  JSON support / extraction •  Custom parsing and tagging rules allow to segregate dynamically •  Self-documenting data inventory
  10. 10. 10© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Gamut Search™ •  Provides instant results over massive amounts of data and long time periods •  Rather than waiting minutes, hours or days for a query to complete, the new capabilities enable users to begin analyzing the entire range, or the “gamut” of their log data, immediately, in a highly interactive fashion. •  No special query language to learn, common Regular Expressions syntax •  Surround Search shows events context with one single click
  11. 11. 11© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Other key features •  In-depth analytics •  Dashboards, pre-configured and customizable, shareable •  Anomaly Detection •  Alerts that can be sent to HipChat, Slack, PagerDuty, HTTP endpoints, others •  JIRA Software integration, point-and- click ticket creation without leaving Loggly Log management is part of a bigger process! DevOps, Agile, CI/CD, …
  12. 12. 12© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Log management is a big data problem Massive incoming event stream Fundamentally multi-tenant Scalable framework for analysis Near real-time indexing Near real-time search Time-series index management Logs are TLDR; Traffic is unpredictable
  13. 13. 13© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 But can be viewed as simple as Ingest Process Index Search and other Services
  14. 14. 14© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Let’s look under the covers of simplicity Amazon Route 53 Kafka broker Kafka broker Search and analytics engine Ingestion pod Indexing pod S3 ingestion Services on processing pod Services on indexing Index management •  Convert unstructured data to structured •  Data processing happens here •  Checkpoint processed data to Kafka •  Can have multiple indexing clusters •  Time-series index management •  Index and store data in search and analytics engine •  Entry point for logs •  Collect and validate •  Checkpoint logs to Kafka Amazon S3 & AWS CloudTrail Amazon RDS MySQL S3 archiving Amazon S3
  15. 15. 15© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Scalability means not just the ability to operate, but to operate efficiently and with adequate quality of service, over the given range of configurations. Build for scalability developing service Scalability is a feature “ ” + running service governing service+ SaaS engineering =
  16. 16. 16© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 All customers don’t behave as good citizens at all times “Noisy neighbors” with spikes in log volumes •  Application on fire •  Log management configuration problem •  Other human error •  Spikes can last a long time
  17. 17. 17© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 How to mitigate the effects of unpredictable load patterns •  Have governors •  Hawk for your infrastructure •  Processes for managing out-of-policy activity •  Set up metrics and alerts that let you know about unexpected behavior
  18. 18. 18© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Sits on top of platform and “watches” what’s going on – across tenants Governors to segregate tenants Identify out-of- policy behavior Segregate misbehavior Inform the right people
  19. 19. 19© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Search and analytics engine Index management Trusted platform for all customers Amazon Route53 Kafka broker Kafka broker Services on indexing pod Search and analytics engine Index management Kafka broker Ingestion pod Indexing pod Services on ingestion pod like S3 archiving Services on processing pod Governors Amazon ElastiCache Amazon RDS MySQL RDS RDS
  20. 20. 20© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Case study
  21. 21. 21© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Troubleshooting use case Problem •  Running our business involves integrating with more than 100 web-based services and billions of API calls every month. •  There are many places where things can go awry. Segment depends on its log data to troubleshoot operational issues •  Solve operational issues with partner integrations •  Isolate relevant log data from billions of API calls per month Competing solutions •  ELK stack •  Homegrown log management system
  22. 22. 22© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Troubleshooting use case Solution •  Use UUID to trace API calls to and from partner services •  Monitor API volume from specific customers •  Error monitoring with code releases Results •  Faster MTTR •  DevOps team no longer involved in most troubleshooting = Devops efficiency •  Pro-active alerts = Devops efficiency and increase customer satisfaction. •  Integrated with bug tracking system = Devops and engineering team efficiency
  23. 23. 23© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Troubleshooting demo
  24. 24. 24© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  25. 25. 25© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  26. 26. 26© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  27. 27. 27© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  28. 28. 28© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  29. 29. 29© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  30. 30. 30© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  31. 31. 31© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  32. 32. 32© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  33. 33. 33© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  34. 34. 34© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  35. 35. 35© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  36. 36. 36© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  37. 37. 37© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  38. 38. 38© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  39. 39. 39© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  40. 40. 40© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  41. 41. 41© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  42. 42. 42© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Monitoring for errors
  43. 43. 43© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  44. 44. 44© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  45. 45. 45© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  46. 46. 46© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  47. 47. 47© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  48. 48. 48© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  49. 49. 49© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  50. 50. 50© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  51. 51. 51© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  52. 52. 52© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17
  53. 53. 53© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Other common Loggly use cases Proactive log monitoring and alerting Data analysis and optimization Team collaboration
  54. 54. 54© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 “Logfooding” at Loggly
  55. 55. 55© 2017 Loggly, Inc. Confidential & Proprietary. 3/30/17 Manoj Chaudhary email: manoj@loggly.com twittter:twitt_2_manoj Visit us at loggly.com or follow @loggly on Twitter. Try Loggly for Free! → https://www.loggly.com/

×