Dallas Mulesoft Meetup presentation covering the log aggregation options within Anypoint platform as well as integrating Mulesoft with the Elastic Stack for log aggregation and APM.

1. July 31st, 2019: Log Aggregation and ELK on Anypoint
Platform
Dallas Meetup

2. All contents © MuleSoft Inc.
Agenda
2
• Introductions
• Log Aggregation and ELK on Anypoint Platform
• What’s next
• Networking time

3. All contents © MuleSoft Inc.
Introductions
3
• About the organizer:
– Ruman Khan
– Miguel Martinez
• About the sponsor:
– AVIO Consulting
A SHOW OF HANDS:
Who is new to this MeetUp?

4. by Adam DesJardin (AVIO Consulting)
Log Aggregation and ELK
on Anypoint Platform

5. Structured Logging
• Writing log messages in a defined format so data is both human readable
and machine processable
Terminology
AVIOCONSULTING.COM
Log Aggregation
• Collecting logs from multiple sources into a single system for storage,
searching and reporting
Log Correlation
• Being able to trace all log messages from a single execution based on some
correlation value

6. Distributed Tracing
• Tracing logs and execution across multiple distributed systems, such as
microservices, to see a full end to end view of the execution
Terminology
AVIOCONSULTING.COM
APM - Application Performance Management
• Monitoring and management of application performance, providing a
detailed view into real time execution performance

7. Why use structured logging?
• Produce logs that are human and machine readable and in a standard format
• Enforce standard data to be logged in all scenarios for better operations
• Parse logs to provide a rich data structure instead of a string message
• Build reports and dashboards around log data such as errors, exception cases
and business events that occur
Structured Logging
AVIOCONSULTING.COM

8. Mulesoft JSON Logger
• Open Source by Mulesoft Professional Services
• Published into your private exchange
• Provides a base set of recommended fields as well as a message field
• Direct integration to Anypoint MQ
Structured Logging
AVIOCONSULTING.COM
https://github.com/mulesoft-consulting/json-logger

9. AVIO Custom Logger
• Open Source by AVIO Consulting
• Published into your private exchange
• Provides a base set of recommended fields as well as a message field
• Provides configurable logger categories for each message
• Provides key/value pair data element for application specific data
• Uses Log4j MapMessage for flexible formatting
• Uses Log4j JSONLayout for conversion to JSON, other layouts work as well
Structured Logging
AVIOCONSULTING.COM
https://github.com/avioconsulting/mule-custom-logger

10. Structured Logging
AVIOCONSULTING.COM
https://github.com/avioconsulting/mule-custom-logger

11. Cloudhub Logging
• OOTB Logging, 100mb or 30 days per application
• Available in Anypoint Console or via API
• Search within a single application only
Anypoint Logging
AVIOCONSULTING.COM

12. Anypoint Monitoring Logging
• Log aggregation across all Mule applications in all environments
• Provides predefined filters such as environment and application
• Search across multiple applications using simple query language
• 200gb per Production core, additional can be purchased
• Requires Titanium support subscription
Anypoint Logging
AVIOCONSULTING.COM

13. Anypoint Logging
AVIOCONSULTING.COM

14. Elasticsearch
• Distribute full-text search engine based on Lucene
• Search across all indexed data using power query language and SQL
• Scalable storage backend defined by your needs
• Can be hosted on-premise, on a cloud provider or fully managed on
Elastic.co
• Many advanced features available such as infrastructure metrics, APM and
Machine Learning
Elastic Stack
AVIOCONSULTING.COM

15. Logstash
• Server-side data processing pipeline
• Ingests data from many sources, transforms and enriches it, and stores it in
your configured data store (Elasticsearch)
• Many input sources supported such as Kafka, S3, SQS, http, beats, etc...
Elastic Stack
AVIOCONSULTING.COM
Filebeats
• Ships file based logs to Elasticsearch or Logstash
• Understands many common log formats (syslog, apache, nginx)

16. Kibana
• Web based UI on top of Elasticsearch
• Provides search, management, visualizations and dashboards
• Can visualize logs, time series data, location data, etc..
Elastic Stack
AVIOCONSULTING.COM

17. Elastic Stack
AVIOCONSULTING.COM

18. Advantages
• Run and keep your data anywhere you choose
• Collect more than just Mule logs, including microservice logs (Java, .NET,
node.js) and service metrics
• Define alerts on log events using Watchers
• Build custom dashboards and reports that can be shared with other users
• Flexible archiving options and destinations such as S3 for low cost storage
• Leverage advanced features such as Machine Learning
Elastic Stack
AVIOCONSULTING.COM

19. Elastic Stack with Mulesoft On Premise
AVIOCONSULTING.COM

20. Elastic Stack with Mulesoft On Premise
AVIOCONSULTING.COM
Filebeats
• Run on each server with a Mule runtime
• Collects Mule runtime and application specific logs
• Adds cloud provided metadata if applicable
• These are shipped directly to logstash

21. Elastic Stack with Mulesoft On Premise
AVIOCONSULTING.COM
Logstash
• Accepts logs from filebeats using the beats input
• Parses and transforms the messages as needed
• Override and set standard elastic fields such as message and timestamp

22. Elastic Stack with Mulesoft On Premise
AVIOCONSULTING.COM
Elasticsearch
• All logs are stored into a single index
• Depending on volume the index can be daily or monthly
• Single index allows searching across apps
• Indexes per environment and separate permissions for who can view them

23. Elastic Stack with Mulesoft On Premise
AVIOCONSULTING.COM
Kibana
• Users can then search logs and build dashboards/reports
• Watchers can be configured through the Kibana UI to send alerts via email,
Slack or other channels

24. Elastic Stack with Mulesoft in Cloudhub
AVIOCONSULTING.COM

25. Elastic Stack with Mulesoft in Cloudhub
AVIOCONSULTING.COM
Cloudhub Mule Apps
• A support ticket is required to enable custom logging on Cloudhub
• Standard Cloudhub logging can still be kept with the correct log4j2.xml
• Any log4j2 appender can be used to publish messages to other data stores

26. Elastic Stack with Mulesoft in Cloudhub
AVIOCONSULTING.COM
Log4j Appenders
• Appenders are written in Java and allow messages to be sent anywhere
• SQS appender supports multiple threads and batching for high throughput
https://github.com/avioconsulting/log4j2-sqs-appender

27. Elastic Stack with Mulesoft in Cloudhub
AVIOCONSULTING.COM
Amazon SQS
• Can exist in the same region as the Cloudhub applications for low latency
• Almost unlimited throughput based on number of threads and latency
• Any other queuing system can be used, but must be highly available

28. Elastic Stack with Mulesoft in Cloudhub
AVIOCONSULTING.COM
Logstash
• Accepts logs from filebeats using the SQS input
• One pipeline per SQS Queue is needed, usually one per environment
• Parses and transforms the messages as needed
• Override and set standard elastic fields such as message and timestamp

29. Elastic Stack with Mulesoft in Cloudhub
AVIOCONSULTING.COM
Elasticsearch
• All logs are stored into a single index
• Depending on volume the index can be daily or monthly
• Single index allows searching across apps
• Indexes per environment and separate permissions for who can view them

30. Elastic Stack with Mulesoft in Cloudhub
AVIOCONSULTING.COM
Kibana
• Users can then search logs and build dashboards/reports
• Watchers can be configured through the Kibana UI to send alerts via email,
Slack or other channels

31. • Cloudhub and On-premise follow very similar patterns
• Differences should only be in how the logs get from Mule applications to
logstash, and minimal difference in how they are processed in logstash
• These logs can be combined into a single Elastic index, allowing searching
across apps in a hybrid environment
• API Analytics can also be stored in Elastic for a single view into all data
Elastic Stack with Mulesoft
AVIOCONSULTING.COM

32. Elastic Stack with Mulesoft
AVIOCONSULTING.COM

33. Elastic Stack with Mulesoft
AVIOCONSULTING.COM

34. Elastic Stack with Mulesoft
AVIOCONSULTING.COM

35. • Once all logs are being aggregated into a single index distributed tracing
becomes much easier
• The remaining challenge is passing a tracing id through all API’s and flows
that can be used to correlate logs
• This can be custom or follow the OpenTracing standard
• This will need to be based in and out as an HTTP header on all API calls
• Other technologies may be implemented differently such as an SQS message
attribute or JMS property
Distributed Tracing
AVIOCONSULTING.COM

36. • Elastic also provides APM capabilities beyond just distributed tracing
• Leverages Mule runtime notification events to capture tracing information
for each flow and processor automatically
• Agent can also collect metrics such as JVM Heap usage, CPU, etc...
• An open source agent is in development for Mulesoft
• Works with Mule 3 but Mule 4 isn’t supported yet, but is being worked on
APM
AVIOCONSULTING.COM
https://www.elastic.co/blog/observability-of-mulesoft-using-elastic-apm-to-monitor-mule-flows
https://github.com/michaelhyatt/elastic-apm-mule3-agent

37. APM
AVIOCONSULTING.COM

38. Resources
AVIOCONSULTING.COM
https://www.elastic.co/blog/observability-of-mulesoft-using-elastic-apm-to-monitor-mule-flows
https://github.com/michaelhyatt/elastic-apm-mule3-agent
https://github.com/avioconsulting/log4j2-sqs-appender
https://github.com/avioconsulting/mule-custom-logger
https://github.com/mulesoft-consulting/json-logger
https://docs.mulesoft.com/runtime-manager/custom-log-appender

39. All contents © MuleSoft Inc.
What’s next
39
• Share:
– Tweet your pictures with the hashtag #MuleMeetup
– Invite your network to join: https://meetups.mulesoft.com/dallas/
• Feedback:
– Contact your organizer Ruman Khan to suggest topics
– Contact MuleSoft at meetup@mulesoft.com for ways to improve the program