Application logging issues and techniques learnt from best practice and running an on-premise Mulesoft cluster and reporting out to enterprise Splunk indexes

2. Deep Dive
Logging

3. All contents © MuleSoft Inc.
Logging – Deep Dive
3
• History of Logging
– Originally was only synchronous, but
this didn’t scale so from Mule 3.6+
libraries were changed to use
asynchronous logging
– Uses the Log4j framework which has
progressed onto log4j2
– Very limited slideshare or articles
about mule logging publically available
– Exam questions are limited to types of
logging produced by connectors and
the runtime upon startup or which logs
get called depending on flow logic

4. All contents © MuleSoft Inc.
Logging – Deep Dive
4
• Asynchronous Logging
– A thread created for each
log4j2.xml in started apps,
domains and mule servers
– Threads are flushed on server
shut down, but logging events
can be lost on system crash
– Logging threads fail over to
server log, ie mule_ee.log
Synchronous Logging
– Can be used in conjunction
with asynchronous logging
– Performance impacted slightly

5. All contents © MuleSoft Inc.
Logging – Deep Dive
5
• Logging Bad Practice
– No Logging standards
• mix of json, text or empty loggers, information not
complete
– Hard Coding of loggers
• Not able to adjust via environment variables or test
for error content
– Empty Loggers
• These can fill logs and confuse debugging
– Logging Limits not respected;
• Cloudhub is limited to 100Mb or 30 days
– Sensitive Information
• Usernames and passwords should be reported in
encrypted or obfuscated format

6. All contents © MuleSoft Inc.
Logging – Deep Dive
6
• Logging Best Practice
– Create standards
• Use json with key: value pairs
• Form common error and logging structures
• Extract Flows to external Dependencies
– Informative Messages
• Visibility into the system (where log is raised)
• Enhanced debugging available (dynamic error levels)
• Before and after system call logs (with payload)
– Request Identifier
• Use the correlationId or assigned code for
transaction (this can help with testing latency)
– Persist Logs
• Maintain historical logs
• Allow non-technical users to access and support
applications

7. All contents © MuleSoft Inc.
Logging – Deep Dive
7
• Three types of logging
– Runtime
– Connectors
– Custom (Logger element)
Hierarchy of Configuration Files
– Application Resources
– Mulesoft Server conf
– Default (see image)
Override in Deployment
- Using deployment properties (3.8+)
- Configuration Reloading
- Disabling Logging

8. All contents © MuleSoft Inc.
Logging – Deep Dive
8
• Disabling Cloudhub Logging
– Mulesoft Documentation Article
– Changing Cloudhub directly
requires raising support tickets
– This configuration allows the
local log4j configuration to be
used in place of the default
cloudhub configuration

9. All contents © MuleSoft Inc.
Logging – Deep Dive
9
• Disabling Complete Logging
– Mulesoft Knowledge Article
– Appenders element will redirect
all logging to console
– The Logging elements sets the
level to OFF, preventing any
output being written to console
– Can be used to rule out logging
issues in performance testing

10. All contents © MuleSoft Inc.
Logging – Deep Dive
10
• Live Logfile Updating
– Mulesoft Knowledge Article
– Not configured in the default or
template configurations
– Can be used to make logging
more or less verbose in live or
nearly live environments
– Would require access
permissions to the mule server
to edit the log4j2.xml file

11. All contents © MuleSoft Inc.
Logging – Deep Dive
11
• Different Log Files
– Mulesoft Knowledge Article
– Additional Thread is created
for the separate
AsyncLogger
– Can be used to separate
business logic and standard
INFO logs from error and
debug logs
– Debug logs can include
connection strings and
sensitive information

12. All contents © MuleSoft Inc.
Logging – Deep Dive
12
• Configuring Custom Log
– Mulesoft Knowledge Article
– Edit ‘mule-deploy.properties’ file
located in app folder
– Can be used to define default
logging file
– Limited by no env variable for
appname
– May come into its own with
Runtime Fabric
– Gets around needing access to
mule_home

13. All contents © MuleSoft Inc.
Logging – Deep Dive
13
• Verbose Logging
– Mulesoft Knowledge Article
– Module & Transport names
(list to the right)
– Levels are as per SLF4J
specification (see below)
Amazon S3 org.mule.module.s3
Amazon SQS org.mule.modules.sqs
Anypoint MQ
org.mule.module.messaging.mq
com.mulesoft.mq
okhttp3
okio
DB org.mule.module.db
Email org.mule.transport.email
File org.mule.transport.file
FTP
org.mule.transport.ftp
com.mulesoft.mule.transport.ftp
JMS org.mule.transport.jms
Microsoft Service Bus org.mule.modules.microsoftservicebus
Mongo org.mule.module.mongo
Salesforce
org.mule.modules.salesforce
com.sforce
org.mule.module.http.internal.HttpMessageLogger
SFTP
org.mule.transport.sftp
com.jcraft
Object store org.mule.modules.objectstore
TCP org.mule.transport.tcp
UDP org.mule.transport.udp
VM Transport org.mule.transport.vm
WMQ com.mulesoft.mule.transport.wmq

14. All contents © MuleSoft Inc.
Logging – Deep Dive
14
• Separate Logging Levels
– Mulesoft Knowledge Article
– Create sub flows with multiple
log elements
– Use elements for standard
business logic and separate
debug including payload
– Exception handling could also
include similar multiple log
elements
– Can be switched to info/debug

15. All contents © MuleSoft Inc.
Logging – Deep Dive
15
• Recommendations
– Separate custom log messages
based on env injected
properties
– Create custom classes of log
messages for reporting
business logic
– Capture a standardized
logging format for exceptions,
business logic, connector
information and validations
– Look at using RFC 7807 to
format error messages

16. All contents © MuleSoft Inc.
Logging – Deep Dive
16
• Further Research
– Mulesoft Blog
• There was a useful blog about logging
from a Mulesoft trainer in October 2019
– Meetup Example Flows
• The Denver meetup group shared a
project with useful examples of bad and
best practice
– Custom Connector Code
• BigCompass have shared a custom
connector built using the Mulesoft DevKit
to formalise logging structure

17. Q&A