There are many tools and frameworks for monitoring. Usually when you think of an Open Source solution, you don’t think to implement it in a COTS product. Nevertheless, this session will tell you how you can implement tools such as Prometheus, Grafana and ELK into such an Enterprise application platform. Monitoring performance, throughput and error rate is important to be in control of your transactions. If you use a Service Bus or SOA/BPM suite product there are a lot out of the box diagnostics waiting for you. The puzzle here is how to get it out in a useful way. Besides of the many commercial solutions also Open Source tools can help you out with it. You can export runtime diagnostics out of the Diagnostics framework, monitor your SOA Composites and trace down Service Bus statistics using Prometheus and Grafana. The session will elaborate how to set up a proper monitoring using these tools, also in a proactive way where automated monitoring is a must for every application environment.

1. A presentation by Michel Schildmeijer,
11 October 2021
Open Source Monitoring
Of an Enterprise Grade Platform

2. Me & My “careerpath” evolution
Kindergarten
1970
Junior
Highschool
1972
Highschool
1977
Study
Clinical
Pharmaco
therapy
1981
Royal
Airforce &
Clinical
Pharmacist
1987
Careerswitch
to IT
2000
• From Amsterdam, the Netherlands
• Work @ Qualogy
• Oracle ACE since 2012, ACED 2019
• Authored 2 books WebLogic in 2010
• IT Career started with UNIX in 1994
• Former BEA technology in 2000
• Focus on containerization strategies,
• DevOps etc
Born
1966

3. • Today’s Challenges
• Java Enterprise Application Servers & containerization
• Kubernetes
• Operators
• Monitoring
Topics

4. Today’s complexity of IT Platforms

5. Today’s complexity of Applications
Java
Microservices

6. Huge amounts of data
•Functional data
□ Application functional
data
□ User related data
•Non Functional Data
□ Metadata
□ Log data
□ Runtime data

7. Operational control
•Get in control of all IT systems
□ By knowing and interpret this data
•Do predictive analysis
□ Know when to expect peak loads
□ Know what trends are going on
□ Know what end users experiencing
□ Know the weaknesses and vulnerabilities of
 Systems, servers, network, databases
 Application and tiers

8. Well known Toolstacks

9. Dashboard Visualization
Gather Diagnostics
Logfiles Visualization
Collect Logentries

10. Java EE App Server
containerized

11. Oracle WebLogic Foundation
Cloud Application Foundation for Java applications
WebLogic Server
Java EE: Reliability, Availability, Scalability & Performance
Coherence EE
Data Grid Scale Out for Java, C++, .NET
Java Virtual Machines
Oracle and OpenSource JVM’s
Development
Tools
Jdeveloper
/
Eclipse
Enterprise
Manager
Admin
and
Operations
WebLogic Suite
Custom App
Packaged
App
Fusion
Middleware

12. Building Blocks Java EE WLS Containerization
• WebLogic Docker certification
□ Docker images, Dockerfiles, GitHub
• WebLogic OCI certification
□ Open Container Initiative CRI-O / KATA
• Integration
□ Management: Operator
□ Monitoring: Exporter for Prometheus
□ Migration: Deploy tooling
□ Logging: Exporter for Elastic Stack
□ Image: Tool management
Kubernetes
Operator
WebLogic WebLogic WebLogic
kubectl

13. WebLogic in Containers
Base Image Linux
Image JDK & WebLogic bins
Image WebLogic Domain
Image WebLogic Deployed app
Container Runtime
Container Software
libvirt LXC
Linux Kernel

14. Operators

15. Kubernetes Operators
• Extension on K8S api
• Configure & manage more complex instances
• Why Operators ?
□ Stateless (web ) apps are easy to manage, scale
□ But thougher when it is with stateful
□ To leverage more experience based knowledge to k8s
The operators from the field in a software driven solution!

16. WebLogic Kubernetes Operator
• Simpler WebLogic management in Kubernetes
• Kubernetes resources are allocated for WebLogic domain(s)
• Manages overall WebLogic environment through Kubernetes APIs
□ Load Balancer, Network,
□ Ingress Controllers,
□ Security,
□ HA restart, upgrade, scaling
□ Persistent storage
• Ensures WebLogic best practices are followed

17. WebLogic Kubernetes Operator
• Helm charts for operator install
• Persistent volumes for logfiles

18. WebLogic Kubernetes Operator
Kubernetes Cluster
pod
pod pod pod pod
K8S Node K8S Node
WebLogic Cluster
WLS K8S operator
Orchestrate
manage
kubectl
Manage WLS configured or dynamic
clusters
loadbalance
Expose T3 + Admin Console to Ingress
WLS Logs to Elastic Search
Operational: Start Stop, REST API, WLDF
Install Domain
inputs (yaml)
K8S Job Domain
resource
Create domain K8S resource

19. Oracle SOA Suite on
Kubernetes

20. Process Services
Orchestration
System BPM
Business Logic
Databases
Data
Services
Enterprise Service Bus (ESB)
data
Web Portals
Human Business Process Management (BPM)
Security
Registry
and
Repository
Manage
and
monitor
SOA generic landscape

21. SOA Suite Platform
WebLogic Server
Managed Server(S)
Java EE deployed Apps for
SOA
Webapps
EJB/MDB
RAR
Java/WLS/SOA Software
Linux
SOA WebLogic Cluster
Operating System
Installation & Config
Domain Configuration
JDBC
JMS
HTTP(S
)
JMX

22. fmwdb
Create DB
Deployment
Create Docker
Secret
Create PV/PVC
Create
Namespace
Create FMW
Schemas
RCU POD
Create RCU
Secret
Create WLS
Secret
Create PV/PVC
Shared NFS
Create SOA WLS
Domain
Offline
SOA OSB Template
SOA OSB
WebLogic Kubernetes Operator
Domain
Introspection
Domain
Create
Job
Domain
Introspect
Job
Domain Start
WebLogic Domain Creation
Database Creation

23. WebLogic REST Management API
• REST API for managing WebLogic
• Covers all of WebLogic management
• Configure, Start/Stop, Deploy,
Monitor...
• Kubernetes Operator makes use of
it for lifecycle
• HTTP, no WebLogic client required
• Well-defined, fully documented
• Consistent with existing management
• Generated from WebLogic MBeans
WebLogic Domain
Admin Server
REST
WebApp
Managed Server
REST
WebApp
Managed Server
REST
WebApp
REST
Start/stop
REST REST
Operator

24. Monitoring Framework

25. Diagnostic Frameworks & Diagnostic Tools in FMW
• WLDF – WebLogic Diagnostic Framework
• DFW – Diagnostic Framework
o Dynamic Monitoring Service (DMS)
o Oracle Diagnostic Logging (ODL)
o Incident Packaging System (IPS or ADRCI)
• Additional Tools
o Selective Tracing

26. WebLogic Diagnostic Framework
Watch & Notification Instrumentation
Data Collectors Image Capture
Data Providers
Archive
Accessor
Monitoring Dashboard
• Included with WebLogic
• Define Watches to monitor
runtime conditions and notify
in case of failure.
• Collect configuration and
runtime data for analysis.
• Capture images to preserve
system state in failure
situation.

27. WebLogic Diagnostic Framework
Oracle Fusion

28. Diagnostic Tools for SOA and FMW
• Predefined Watches and Diagnostic Dumps to detect,
• Diagnose & resolve problems with SOA MBeans & DMS Metrics.
• Detect critical failures and collect diagnostic dumps
o Relevant diagnostic information like logs, metrics, server images,
• Create incidents in the ADR (Automatic Diagnostic Repository)

29. WLDF, DFW and Diagnostic Tools for SOA
Monitor
System
• Predefined Watches
• Predefined Diagnostic Dumps
Critical
Failure
• Collect Diagnostic Dumps
• Create Incident
Collect
• Collect Data for processing

30. Dynamic Monitoring Service (DMS)
• DMS for runtime metrics applications and
components
• Can be monitored through WLDF Watches to trigger
data collections through DFW.

31. WebLogic Logging
Exporter

32. • Logstash & Kibana
• WebLogicLoggingExporter.yaml
• weblogic-logging-exporter-<version>.jar
• snakeyaml-1.23.jar - parser and emitter for Java
□ From Maven Central Repository
Components

33. Actions
• Install ELK in your Kubernetes Cluster – Or separate
• Copy jar files to shared location of you K8S cluster(not to pods)
• WLS Startup Class
• Set Classpath in setDomainEnv
• Setup config for export
# Export for ELK Logging
CLASSPATH="/scratch/domains/soadomain/weblogic-logging-exporter-
1.0.0.jar:/scratch/domains/soadomain//snakeyaml-1.23.jar:${CLASSPATH}"
export CLASSPATH

34. Domain Startup Class

35. ELK Config
• weblogic-logging-exporter-1.0.0.jar
• snakeyaml-1.23.jar

36. WebLogic Index Patterns

37. Open Ends
• Limited support for logs
• Only manual upload for ODL based logs

38. SOA Diagnostics logs
• Using logstash deployment in K8S
• Make use of SOA Diagnostic logs

39. Setup Prometheus &
Grafana

40. • Prometheus & Grafana Install
• wlsexporter.war & get script
□ Downloads the war & repackage with proper config
• JSON Dashboard imports
• DB Exporter deployment for querying SOA Repository
Components

41. MBean Metrics Exporter
wlsexporter.war
WLDF
Prometheus
Datasource

42. Prometheus integration
• Generate exporter wars
• Deploy warfiles to Admin and Managed Servers
• Create Rolebinding, Role and Service Account
□ wls-exporter-ServiceMonitor.yaml

43. Prometheus

44. Grafana value mapping

45. Grafana dashboard

46. Possibilities
• Query with SQL on SOA Infra Dehydration tables
• Database exporter for Prometheus / Grafana
• Status monitoring of long running instance
• Toml config with custom metrics
• No commercial Oracle Grafana plugin needed
• Still under investigation

47. Typical Flow
Db exporter to Prometheus
config.toml

48. Open Ends
• Limited support for Mbeans
• No custom or Application Defined Mbeans
• More use cases and useful dashboards in ELK

50. Thank you!
mschildmeijer@qualogy.com
https://www.qualogy.com/discover-qualogy/news-
blogs/author/michel-schildmeijer
https://twitter.com@MNEMONIC01
https://nl.linkedin.com/in/mschldmr