Download as PDF, PPTX
Uploaded byAdam DesJardin
Dallas Mulesoft Meetup - Log Aggregation and Elastic Stack on Anypoint Platform
The document outlines the agenda and key discussions from a meetup focused on log aggregation and ELK (Elasticsearch, Logstash, Kibana) on the Anypoint platform. It covers topics such as structured logging, log correlation, distributed tracing, and application performance management while providing insights into various logging tools and implementations. Additionally, it highlights the importance of standardized logging practices for effective monitoring and reporting in software development and operations.
More Related Content
Similar to Dallas Mulesoft Meetup - Log Aggregation and Elastic Stack on Anypoint Platform
Dallas Mulesoft Meetup - Log Aggregation and Elastic Stack on Anypoint Platform
- 1. July 31st, 2019:Log Aggregation and ELK on Anypoint Platform Dallas Meetup
- 2. All contents ©MuleSoft Inc. Agenda 2 • Introductions • Log Aggregation and ELK on Anypoint Platform • What’s next • Networking time
- 3. All contents ©MuleSoft Inc. Introductions 3 • About the organizer: – Ruman Khan – Miguel Martinez • About the sponsor: – AVIO Consulting A SHOW OF HANDS: Who is new to this MeetUp?
- 4. by Adam DesJardin(AVIO Consulting) Log Aggregation and ELK on Anypoint Platform
- 5. Structured Logging • Writinglog messages in a defined format so data is both human readable and machine processable Terminology AVIOCONSULTING.COM Log Aggregation • Collecting logs from multiple sources into a single system for storage, searching and reporting Log Correlation • Being able to trace all log messages from a single execution based on some correlation value
- 6. Distributed Tracing • Tracinglogs and execution across multiple distributed systems, such as microservices, to see a full end to end view of the execution Terminology AVIOCONSULTING.COM APM - Application Performance Management • Monitoring and management of application performance, providing a detailed view into real time execution performance
- 7. Why use structuredlogging? • Produce logs that are human and machine readable and in a standard format • Enforce standard data to be logged in all scenarios for better operations • Parse logs to provide a rich data structure instead of a string message • Build reports and dashboards around log data such as errors, exception cases and business events that occur Structured Logging AVIOCONSULTING.COM
- 8. Mulesoft JSON Logger •Open Source by Mulesoft Professional Services • Published into your private exchange • Provides a base set of recommended fields as well as a message field • Direct integration to Anypoint MQ Structured Logging AVIOCONSULTING.COM https://github.com/mulesoft-consulting/json-logger
- 9. AVIO Custom Logger •Open Source by AVIO Consulting • Published into your private exchange • Provides a base set of recommended fields as well as a message field • Provides configurable logger categories for each message • Provides key/value pair data element for application specific data • Uses Log4j MapMessage for flexible formatting • Uses Log4j JSONLayout for conversion to JSON, other layouts work as well Structured Logging AVIOCONSULTING.COM https://github.com/avioconsulting/mule-custom-logger
- 10.
- 11. Cloudhub Logging • OOTBLogging, 100mb or 30 days per application • Available in Anypoint Console or via API • Search within a single application only Anypoint Logging AVIOCONSULTING.COM
- 12. Anypoint Monitoring Logging •Log aggregation across all Mule applications in all environments • Provides predefined filters such as environment and application • Search across multiple applications using simple query language • 200gb per Production core, additional can be purchased • Requires Titanium support subscription Anypoint Logging AVIOCONSULTING.COM
- 13.
- 14. Elasticsearch • Distribute full-textsearch engine based on Lucene • Search across all indexed data using power query language and SQL • Scalable storage backend defined by your needs • Can be hosted on-premise, on a cloud provider or fully managed on Elastic.co • Many advanced features available such as infrastructure metrics, APM and Machine Learning Elastic Stack AVIOCONSULTING.COM
- 15. Logstash • Server-side dataprocessing pipeline • Ingests data from many sources, transforms and enriches it, and stores it in your configured data store (Elasticsearch) • Many input sources supported such as Kafka, S3, SQS, http, beats, etc... Elastic Stack AVIOCONSULTING.COM Filebeats • Ships file based logs to Elasticsearch or Logstash • Understands many common log formats (syslog, apache, nginx)
- 16. Kibana • Web basedUI on top of Elasticsearch • Provides search, management, visualizations and dashboards • Can visualize logs, time series data, location data, etc.. Elastic Stack AVIOCONSULTING.COM
- 17.
- 18. Advantages • Run andkeep your data anywhere you choose • Collect more than just Mule logs, including microservice logs (Java, .NET, node.js) and service metrics • Define alerts on log events using Watchers • Build custom dashboards and reports that can be shared with other users • Flexible archiving options and destinations such as S3 for low cost storage • Leverage advanced features such as Machine Learning Elastic Stack AVIOCONSULTING.COM
- 19. Elastic Stack withMulesoft On Premise AVIOCONSULTING.COM
- 20. Elastic Stack withMulesoft On Premise AVIOCONSULTING.COM Filebeats • Run on each server with a Mule runtime • Collects Mule runtime and application specific logs • Adds cloud provided metadata if applicable • These are shipped directly to logstash
- 21. Elastic Stack withMulesoft On Premise AVIOCONSULTING.COM Logstash • Accepts logs from filebeats using the beats input • Parses and transforms the messages as needed • Override and set standard elastic fields such as message and timestamp
- 22. Elastic Stack withMulesoft On Premise AVIOCONSULTING.COM Elasticsearch • All logs are stored into a single index • Depending on volume the index can be daily or monthly • Single index allows searching across apps • Indexes per environment and separate permissions for who can view them
- 23. Elastic Stack withMulesoft On Premise AVIOCONSULTING.COM Kibana • Users can then search logs and build dashboards/reports • Watchers can be configured through the Kibana UI to send alerts via email, Slack or other channels
- 24. Elastic Stack withMulesoft in Cloudhub AVIOCONSULTING.COM
- 25. Elastic Stack withMulesoft in Cloudhub AVIOCONSULTING.COM Cloudhub Mule Apps • A support ticket is required to enable custom logging on Cloudhub • Standard Cloudhub logging can still be kept with the correct log4j2.xml • Any log4j2 appender can be used to publish messages to other data stores
- 26. Elastic Stack withMulesoft in Cloudhub AVIOCONSULTING.COM Log4j Appenders • Appenders are written in Java and allow messages to be sent anywhere • SQS appender supports multiple threads and batching for high throughput https://github.com/avioconsulting/log4j2-sqs-appender
- 27. Elastic Stack withMulesoft in Cloudhub AVIOCONSULTING.COM Amazon SQS • Can exist in the same region as the Cloudhub applications for low latency • Almost unlimited throughput based on number of threads and latency • Any other queuing system can be used, but must be highly available
- 28. Elastic Stack withMulesoft in Cloudhub AVIOCONSULTING.COM Logstash • Accepts logs from filebeats using the SQS input • One pipeline per SQS Queue is needed, usually one per environment • Parses and transforms the messages as needed • Override and set standard elastic fields such as message and timestamp
- 29. Elastic Stack withMulesoft in Cloudhub AVIOCONSULTING.COM Elasticsearch • All logs are stored into a single index • Depending on volume the index can be daily or monthly • Single index allows searching across apps • Indexes per environment and separate permissions for who can view them
- 30. Elastic Stack withMulesoft in Cloudhub AVIOCONSULTING.COM Kibana • Users can then search logs and build dashboards/reports • Watchers can be configured through the Kibana UI to send alerts via email, Slack or other channels
- 31. • Cloudhub andOn-premise follow very similar patterns • Differences should only be in how the logs get from Mule applications to logstash, and minimal difference in how they are processed in logstash • These logs can be combined into a single Elastic index, allowing searching across apps in a hybrid environment • API Analytics can also be stored in Elastic for a single view into all data Elastic Stack with Mulesoft AVIOCONSULTING.COM
- 32. Elastic Stack withMulesoft AVIOCONSULTING.COM
- 33. Elastic Stack withMulesoft AVIOCONSULTING.COM
- 34. Elastic Stack withMulesoft AVIOCONSULTING.COM
- 35. • Once alllogs are being aggregated into a single index distributed tracing becomes much easier • The remaining challenge is passing a tracing id through all API’s and flows that can be used to correlate logs • This can be custom or follow the OpenTracing standard • This will need to be based in and out as an HTTP header on all API calls • Other technologies may be implemented differently such as an SQS message attribute or JMS property Distributed Tracing AVIOCONSULTING.COM
- 36. • Elastic alsoprovides APM capabilities beyond just distributed tracing • Leverages Mule runtime notification events to capture tracing information for each flow and processor automatically • Agent can also collect metrics such as JVM Heap usage, CPU, etc... • An open source agent is in development for Mulesoft • Works with Mule 3 but Mule 4 isn’t supported yet, but is being worked on APM AVIOCONSULTING.COM https://www.elastic.co/blog/observability-of-mulesoft-using-elastic-apm-to-monitor-mule-flows https://github.com/michaelhyatt/elastic-apm-mule3-agent
- 37.
- 38.
- 39. All contents ©MuleSoft Inc. What’s next 39 • Share: – Tweet your pictures with the hashtag #MuleMeetup – Invite your network to join: https://meetups.mulesoft.com/dallas/ • Feedback: – Contact your organizer Ruman Khan to suggest topics – Contact MuleSoft at meetup@mulesoft.com for ways to improve the program