SlideShare a Scribd company logo
1 of 115
Download to read offline
©2018 Michel Schildmeijer©2018 Michel Schildmeijer
Oracle Fusion Middleware:
Performance & Diagnostics
solutions on WebLogic
nlOUG Tech Experience ‘The Cloud is next’
Amersfoort, 7 June 2018
By Michel Schildmeijer,
Me…..
• Michel Schildmeijer, almost 52 years
• Solutions Architect
• Author of 2 books
• Specialties:
• WebLogic 6 to 12, OSB, Tuxedo, SOA,
• Architect and design
Contents
• Troubleshoot Patterns & Methods
• A slight view under the hood
• Hints, tips & tricks
Troubleshoot patterns &
methods
• System errors, messages lost, bad
performance
• Where do I begin??
• What method should I follow?
Troubleshoot overview
• Using these troubleshooting methods helps you to:
• A structured way of detecting issues
• Get behind the cause of the problems that occur
• Methods to follow:
• Use of Operating system tools
• Use of JDK tools
• Use of WebLogic Server tools
• Use of FMW Diagnostic tools
• Use of Enterprise Manager ( Cloud Control )(?)
• Use of OMC : Application Performance Management, Log
Analytics(?)
Troubleshoot Methods
• Patterns to identify a “Hang”
• Hangs related to the WebLogic Server, Java or some
subset of resources
• Generic Server Hang Pattern
• JDBC Hang Pattern
• EJB & RMI Hang Pattern
• JMS Hang Pattern
Troubleshoot Patterns
• Server doesn’t respond to new requests
• Requests time out.
• Requests take longer and longer to
process (may be on the way to a hang).
• A server crash is not usually a symptom
of a hung server but final state
Server Hang
• DriverManager.getConnection() in code.
• SQL takes unexpectedly long time
• JDBC connection pool hangs
• Slow network causes to slow down or hang.
• A deadlock all execute threads to hang & wait
• RefreshMinutes or TestFrequencySeconds in conn
pool causes hang
• Long response JDBC conn pool shrinking and
• No Sufficient DB resources
JDBC Hang
• Issues over RMI with remote JVM
• Serialization
• Cluster issues
• JNDI lookup fails
at
weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:
341)
EJB RMI Hang
• Increasing pending messages
• Sent by a producer
• Received by a consumer
• Lagging async client
• Durable subscribers (to avoid missing placed messages in a
topic)
JMS Hang
• O/S tools:
• sar: system activity
• mpstat: per-processor statistics
• vmstat: virtual memory statistics
• netstat: network statistics
• iostat: Input/Output statistics
• top
WebLogic Troubleshoot overview
• Start with a simple check – it seems so
obvious but:
• Is there enough storage
• Is there enough RAM available
• Does the system do a lot of paging
WebLogic Troubleshoot overview
JVM Troubleshoot
The JavaVirtualMachine
Java Source code .java
Java bytecodeJVM
RuntimeCompiler
Java Runtime
JDK Compiler
• A Java application runs a piece of code
• Code enters the JVM
• It enters several stages:
• Operations
• Data structure
• Transformation ( to bytecode)
JVM from code to machine
Java code
JVMJVM
Operat
ons
Data
re
Data
Structu
re
Transformati
on
110110
100
• JVM process consists Java and JVM internal threads
• JVM threads: Garbage Collect, Optimizer and Finalizer
• Each thread has a stack to store runtime data
• Overflow will happen if this stack will store more than its size
• Each O/S has its default stacksize
• Can be extended with -Xss
Understanding Java threads
• Runtime behaviour:
o JVM runs as an OS process
o Execute work through threading model
(concurrency)
o Garbage Collector
• Each Weblogic Server (instance) runs
inside a separate JVM (instance).
JVM from code to machine
• Crash can occur during
• Compile process
• Cause: invalid native code (C, C++, Libs)
• Internal Code – Garbage Collect ( Segmentation Fault)
• Cause: GC Fault leading into bad memory access
• Long running GC can result in paging or CPU overload
• Application code
• Like 3rd Party drivers
• (JDBC, MQ libraries)
JVM from code to machine
• What to look for in java thread dumps:
• Low Memory Detector
• (Daemon thread to detect and report low
memory conditions)
• CompilerThreadn
• (Daemon thread/s to compile & optimize
byte code)
• Signal Dispatcher
• (Daemon thread to respond OS signals)
JVM troubleshoot
• Finalizer
• Daemon thread to handle objects in finalizer queue
• Reference Handler
• Daemon thread to handle objects in reference queue
• VM Thread
• Main thread from JVM
• GC task thread#n
• Thread/s to perform garbage collection
JVM troubleshoot
• Liveset and fragmentation
• Heap gets fragmented when small & large objects are
mixed store with various lifetimes. May cause longer
GC
• Garbage collect time and behavior
JVM troubleshoot
• Java HotSpot
• Java Mission Control
• JConsole
• JVisualVM
JVM tools
• Java Mission Control provides:
• Console and command line (jcmd)
• Live JVM statistics
• Graphs and metrics about local or remote systems
• Basic JMX interaction with MBeans
• Runtime data
• An in-depth look at live metrics, like memory, CPU and
stack trace on specific threads
• Profile methods or exceptions & execute commands
• Remote or local
Java Mission Control
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=3614
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
•
Java Mission Control
-XX:+UnlockCommercialFeatures-XX:+UnlockCommercialFeatures
jcmd [PID] VM.unlock_commercial_features
• Diagnostic Volume
must be set higher
than the default
(Low)
Java Mission Control
Java Troubleshoot
• Some well known errors
• Class not found – missing or incorrect Classpath set
during startup (server start params or setDomainEnv)
or adding domain libs
• NullPointers – A value is expected but it’s not there
(null):
• Poorly written
• Bad error handling
• Missing config
• Stack Overflow – Recursive procedures, a method
with no exit
• Too Many Open Files – Ulimit settings
WebLogic Server Troubleshoot
• Server applies to: Managed server
or AdminServer
• Partial or total (OutOfMemory)
• Lack of resources/deadlocks
Server Hangs
• The server does not respond to new
requests.
• Requests time out.
• Requests take longer and longer to
process (may be on the way to a hang).
• A server crash is not usually a symptom
of a hung server but may follow.
Symptoms
Deadlock
Thread 1 Thread 2
Resource A
Resource B
Lock Wait
LockWait
Deadlock
Possible Causes
• All threads waiting for RJVM, RMI responses.
• Garbage Collection taking too much time.
• Long Running JDBC calls or , connection leaks, deadlocks
• JVM hang during code optimization
• JSP compilation under heavy load.
• WebLogic Server generic:
• WebLogic Diagnostics Framework
• Logs
• WLST
• JVM specific :
• Java Mission Control,
• JStat,
• JConsole
Tools/Methods
• Force a JVM dump by:
• kill -3 <pid> … stack in STDOUT
• Usually in
<domain>/servers/logs/<serverinstance>/<serverinstance.out>
• See more in the next slides about WebLogic Threads
Tools/Methods
WebLogic Threads
• A threaddump is a snapshot of the state of a WebLogic
Server Instance
• Different ways to take a threaddump:
• kill -3 <pid> the O/S way
• WebLogic Console
• WLST(WebLogic Scripting Tool)
cd ('Servers')
ls()
cd ('AdminServer')
ls()
threadDump()
WebLogic taking threaddumps
WebLogic Thread Architecture
Default
Queue
Socket
Threads
Execute
Threads
Application
WebLogic Threads
• Active threads:
• Are either processing or waiting for requests
• Can be tagged with additional attributes such as Idle or
Hogging
• Standby threads are not used to process requests unless
throughput is needed.
• Workmanagers can control threads of specific parts
WebLogic Stuck Threads
Running for a long time
• Deadlocked
• Infinite loop
• Overloaded server
• WebLogic:
• Periodically checked how long threads are running
• Logs a warning if a thread becomes stuck
• Creates additional threads to handle new requests
• Sets its state to Failed after specified nr of threads
become stuck
• Individual work managers can detect stuck threads & reject
new requests.
Stuck Thread possible actions
• Hanging threads will be diagnosed as STUCK
• Default time to get STUCK is 600 secs
• Thread has not completed it’s work
The sense of taking threaddumps
• Thread Dumps can give you:
• View of the state of application server threads
at that instant in time
• Information about potential issues such as
• Hot spots within code which seem to be called
often
• Portions of code where the application seems to be
hung
• Locking and thread synchronization issues in an
application
The sense of taking threaddumps
• Take several dumps to investigate various states
• Threads high in stack can be treated as idle, mostly waiting
for some work to do
• Blocked or Wait on socketAccept() listener thread waiting
for socket connections
The sense of taking threaddumps
• WebLogic specific:
• Ignore threads in
ExecuteThread.waitForRequest()
• Muxer and processSocket
threads are for housekeeping
and listening
• A Muxer which is blocked should
not be waiting for a lock by a
non Muxer
The sense of taking threaddumps
• In the AdminConsole
WebLogic taking threaddumps
• Are meant to synchronize between threads
• Lock contention is a thread waiting for another to be
released
• Fat locks: Have a history of contention (several threads try
to take the lock simultaneously), or waiting for notification
• Thin locks: Without any contention.
• Recursive locks: A lock on a
thread several times without having been released.
Understanding Locks
WebLogic JDBC Troubleshoot
• Wrong or missing drivers in Classpath
• Connection errors by:
• Usernames/passwords
• Connection leaks by bad apps
• Set Inactive timeout to a value (default is 0)
• Insufficient connection parameters
• Max Capacity, Reserve Time Out(-1)
• Invalid JDBC URLS
weblogic.common.ResourceException:
Could not create pool connection.
The DBMS driver exception wa…
JDBC additonal errors
WebLogic JDBC mechanism
Application JDBC Driver
Database1. Connection
2. Statement
3. SQL
4. Results
5. Close
Pool
Conn1
Conn2
JNDI Lookup
• Using dumpPool
and WLDF capture
JDBC Profiling
• DebugJDBCConn - Trace all
connections.
• DebugJDBCSQL - Trace all JDBC API
calls
• DebugJDBCInternal - low-level
internal data source activities.
• DebugJTAJDBC - Trace transaction
management
Debug Flags
• Bad implemented applications can starve
connections and data
• Cache or “hog” connections instead of releasing
them.
• Do not explicitly release connections using
close().
• Automatically reclaim with “Inactive Connection
Timeout”
Connection Leaks
WebLogic JMS Troubleshoot
• Incorrect targeting
• Connection factories to subdeployments
instead of to entire servers
• Targeting distributed destinations to entire
servers instead to subdeployments
• Using multiple subdeployments within the
same module
• Configure JMS migration without a cluster
• Convert a single-server domain to a cluster
Common JMS issues
• OOM in case of large amount of pending messages
• JMS Quota can prevent that
• Message paging – messages in virtual memory(file or db)
• Flow control
Common JMS issues - OOM
• Expired messages
• Set a TimeToLive & Exp policy
• JMS system failure and nonpersistent messages
• Set a delivery and time to delivery mode
• Insufficient retry settings
• Message delay settings
• Nondurable topic subscriptions
Common JMS issues – Lost messages
• Applications used to automatically consume JMS
• Often used in SOA Suite applications ( SOA Suite,
OSB )
• Manages itself ( pooling )
• A few debug flags or use WLDF to diagnose
MDBs
WebLogic Log and debug
methods
• Logging services are important for diagnostics
• WebLogic produces output by using a Message Catalog
• Using the standard java.util.logging
• Can be extended using log4j modules
• All kinds of events, like startup/shutdown,
appl.deployments, subsystem failures
are written to logs
WebLogic Log and Debug options
WebLogic Log framework
• Several severity levels like:
• TRACE, DEBUG,INFO ..
• Important to set the proper level
• For FMW specific components, ODL-HANDLER can be
used
WebLogic Log and Debug options
• Subsystem logs can be from:
• HTTP
• Transaction Manager
• JDBC
• JMS server
• WebLogic Auditing provider
• FMW Logs
WebLogic Log and Debug options
• Debug
• Setting debug options in the
Admin Console
• Or use –D options in startup script
-Dweblogic.debug.DebugJDBCSQL=true
Prints information about all JDBC methods
WebLogic debug options
• Or use WLST
serverConfig() or domainConfig()
debug =
getMBean('/Servers/mserver/ServerDebug/mserver')
debug.setDebugJDBCInternal(true)
WebLogic debug options
• OpenSource Log engine from Apache
• Enable the LogMBean.isLog4jLoggingEnabled attribute
• In WLST: cmo.setLog4jLoggingEnabled(true)
• Add Log4j classes,
• WL_HOME/server/lib/wllog4j.jar & log4j.jar to server CLASSPATH
WebLogic Log4j
• Enable log4j in AdminConsole
WebLogic Log4j
• As an extension on the default logging handlers
• Comes with FMW installation, not WebLogic
default
• Logging view and config in Fusion Middleware
Control
• Can be set to several trace levels
• Output in <servername>-diagnostic.log
Oracle diagnostic Logging (ODL)
Oracle diagnostic Logging Runtime
Oracle diagnostic Logging
• Edit or create your own handler
Oracle Diagnostic Logging
• Edit or create your own handler
WebLogic Diagnostics
Framework
WebLogic Diagnostic Framework
•Included with WebLogic
•Define Watches to monitor runtime
conditions and notify in case of
failure.
•Collect configuration and runtime
data for analysis.
•Capture images to preserve system
state in failure situation.
Watch &
Notification
Instrumentation
Data Collectors Image Capture
Data Providers
Archive
Accessor
Monitoring Dashboard
• Framework for monitoring and diagnostics
• Consolidated access for MBean data,
instrumentation, logging, debugging
• Source for monitoring data
• Server & application instrumentation notifications
based on the output : log, email, SNMP, JMX, JMS
• Request dyeing and tracing
• Data archiving for later access
• WLDF Dashboard
WLDF
• Key Features
• Gathering input from JMX MBeans
• WebLogic subsystems and application
code can be instrumented to provide
events and watches
• Notifications can be defined for events
and watches
• Console extension for monitoring of
current and historical data
• Data archiving
WLDF
• Can be configured on filesystem or Database
• Configure with Data retirement
WLDF
• Create Diagnostic Module
• Collect Metrics. A collection includes:
• The MBean type to query
• The specific MBean instance names to query (all instances,
• by default)
• The MBean attributes to collect (all attributes, by default)
• How often to gather data
• Create Watches & Notifications
WLDF
Diagnostic Frameworks & Diagnostics Tools in FMW
• WLDF – WebLogic Diagnostic Framework
• DFW – Diagnostic Framework
o Dynamic Monitoring Service (DMS)
o Oracle Diagnostic Logging (ODL)
o Incident Packaging System (IPS or ADRCI)
• Additional Tools
o Selective Tracing
WebLogic Diagnostic Framework
Oracle Fusion
Middleware
Diagnostic Tools for SOA and FMW
• Predefined Watches and Diagnostic Dumps
• Detect, diagnose and resolve problems
• Use WLS and SOA MBeans and DMS Metrics.
• Detect critical failures and collect diagnostic dumps
containing relevant diagnostic information like logs,
metrics, server images,
• Create incidents in the ADR (Automatic Diagnostic
Repository)
Integration of WLDF, DFW and Diagnostic Tools for SOA and FMW
Monitor
System
• Predefined Watches
• Predefined Diagnostic Dumps
Critical
Failure
• Collect Diagnostic Dumps
• Create Incident
Collect
• Collect Data using EM or
Email
Dynamic Monitoring Service
• DMS for runtime metrics applications and components
• Can be monitored through WLDF Watches to trigger data
collections through DFW.
Dynamic Monitoring Service (DMS)
• Direct accessible through http://<wlshost>:7001/dms
• Give pretty much raw unstructured results
• Better option is to use FMW Console
DMS in FMW Console
Diagnostic Framework
(DFW)
• collect a pre-configured set of
information at exactly the right
time through a triggering
mechanism.
• Triggered from Oracle
Diagnostic Logging events or
WebLogic Diagnostic
Framework Watches.
• Information can also be
collected manually using WLST
Oracle Diagnostic Logging (ODL)
• ODL is the primary logging for FMW
• The ODL log entries are in XML format
Selective Tracing (ST)
• Selective Tracing is a
facility available through
Enterprise Manager to
limit the scope of trace
logging.
• This allows for debugging
of a production system
without overloading the
system with logging
activity.
Selective Tracing (ST)
Selective Tracing (ST)
Performance
• Performance : How system’s & applications response time
and throughput are affected by adding load.
• Capacity : Maximum threshold a system is under a given set
of conditions.
• Scalability : how does a system responds to increasing load
by adding additional resources.
• Key indicators are :
• Response time
• Throughput
Performance Terms and Definitions
• Locate & minimize bottlenecks
• Gather info about Application(s)
• Concurrent number of users and interfaces.
• Number and size of app requests.
• Amount of data and its consistency.
• Determining your target CPU utilization
• Tune your Database(s) (customers or repos)
• Tune WebLogic Server Performance Parameters
• Tune Your JVM
• Tune the Operating System
• Tuning WebLogic Persistent Store
Performance objectives
• Investigate and analyze!
• Always consider before
implementing :
• What am I changing
• Has the bottleneck gone
away?
• Is there a new bottleneck?
Tune WebLogic Server
• Enable by default
• Will use a O/S threadpool instead from “borrowing” from
WebLogic default Exec.Q
• Native muxers use platform-specific native binaries to read data
from sockets
• Non native or Java sockets :
• Uses pure Java to read data from sockets.
• Blocks until there is data to be read
from a socket(can cause issues
on server level)
Native I/O and muxers
• Represents nr of simultaneous operations performed by apps that
use the execute Q
WebLogic Thread Pool
WebLogic Socket Muxer
Application
WebLogic Kernel assigns
Execute Req to
ExecuteThread 1
ExecuteThread 1 handles
the request (for example
executes a JDBC request)
WebLogic Kernel handles self
tuning ThreadPool,
Workmanagers, allocations
The app now handles the
thread itself and gives back
result
Execute Request
• Different execute Q’s in 11g for applications and administration
work
• weblogic.kernel.default, RMI, HTTP
• 1 default Queue
• In 12c Single threadpool
• Self tuning
• Work managers
• Prioritize by rules
• Different constraints regarding min max response capacity
-Dweblogic.threadpool.MinPoolSize=5 -Dweblogic.threadpool.MaxPoolSize=5
WebLogic Thread Count
• Default All apps use all execute queues
• Configuring multiple execute queues can provide
additional control for applications.
• By using multiple execute queues, selected
applications have access to a fixed number of
execute threads, regardless of the load on WebLogic
Applications and Execute Queues
• ThreadPoolPercentSocketReaders
• Default set to 33 ( range is 1-99)
• Allocating threads as readers can increase accepting
requests
Threads as SocketReaders
• Piece of memory for read and write data to sockets (network layer)
• A pool of chunks is maintained
• For applications with large amounts of data
• Parameters are:
• weblogic.Chunksize
• Size of a chunk (in bytes). Set to network's maximum transfer unit (MTU),
after subtract it from any Ethernet or TCP header sizes
• weblogic.utils.io.chunkpoolsize
• Sets the maximum size of the chunk pool. Monitor the CPU profile or use a
memory/ heap profiler for call stacks invoking the constructor
weblogic.utils.io.Chunk.
• weblogic.PartitionSize
• Sets the number of pool partitions used (default is 4).
Partitioning the thread pool spreads the potential
for contention over more than one partition
To be set in the setDomainEnv.sh
WebLogic ChunkSize
• Specifies how many TCP connections can be buffered.
These are on the TCP stack but not yet accepted by
an app.
• Default is 50
• Maximum depends on O/S
WebLogic Backlog
• When to tune?
• When getting “connection refused” in high throughput apps
(SOA composites)
• If tuning, increase by 25% of it’s default until errors
dissapear
WebLogic Backlog
• Prepared Statement Cache
• Keeps compiled SQL statements in memory
• Set to LRU
• A data source with 10 connections deployed on 2 servers, with the Statement Cache
to 10 (default), 200 cursors can be used on the database for these cached
statements
• MaxCapacity
• Max of physical connections
• Db drivers can limit connections
• Number should equal number of client sessions
• LLR instead of XA
• Decrease 2PC overhead
• Disable wrapping datatypes (debug, track, transaction control of
blb, clob)
JDBC Connectionpool tuning
• In some cases it takes ages after startup to
continue
• Generate a stackup
Tuning Startup
• This happens when using default seed
generator:
• Use of dev/urandom during startup in stead of
random
• Specifiy in startup properties -
Djava.security.egd=file:///dev/urandom
• Or in $JAVA_HOME/jre/lib/security/java.security
securerandom.source=file:/dev/./urandom
Tuning Startup
• Keep MDS and Dehydration store clean
• MDS delete older label deployments
• Dehydration store purge strategy
Tuning Startup SOA
• Configure Transaction settings:
• BPEL SyncMaxWaitTime
• BPEL EJBS
• JTA
• DB
Tuning other SOA
:
• Discovery cache age to 28800000
• discovery wait time to 30000
• Set the cache results to true
• Dumpinterval to 10800 with a maximum of 75 MB
• Enable Large Repository
• Increase the collector configuration settings for DMS:
• Use an interval of 300 secs with a remove cycle of 3
• And use an interval of 120 secs with a remove cycle of 2, set
this to default
• Restrict the amount of displayed instance information to 12
hours
•
Tuning AdminServer EM SOA
• Most aspects of JVM tuning relate to:
• Sufficient memory heap
• An efficient garbage collection scheme
• Classloading
• Compile/interpret
Tuning the JVM
• Most important to focus on:
• Garbage collection scheme
• Compaction/Defrag
• Free up memory
• Memory heap
• Is a runtime repository for live objects, dead objects and free memory.
• No more than 75% of total RAM
• The JVM heap sets frequency & duration GC
• Large heap size :
• full garbage collection slower and less frequent
• Small heap size
• full garbage collection is faster & higher frequency
Tuning the JVM
• Goal is:
• Minimize Garbage Collect
• Set heapsize that GC does no happen all the time
Tuning the JVM
• Investigate with –verbosegc option
• How often does GC run
• How long does it take?
• Full garbage collection should not take longer than 3 to 5
seconds. Lower heap if major GC time is greater.
Tuning the JVM
• Set Xms and Xmx parameters to specify initial and Max
HeapSize
• Xmx is the Java Object Heap, not the System Heap –> which
is larger
• Values equal will cause no overhead of heap management
• PermSize and MaxPermSize
For stored classes, methods
Be aware:
• Java methods, thread stacks, native handles are in memory
separate from heap
Tuning the JVM
“The Cloud is next”
Oracle Fuson Middleware Diagnostics, Performance and Troubleshoot
Any Questions? Let’s keep in touch!!
The ACE Community
mschildmeijer@qualogy.com
https://community.oracle.com/blogs/mnemonic
https://www.qualogy.com/nl/techblog/author/michel-
schildmeijer
Any Questions?
Let’s keep in touch!!
@MNEMONIC01
@Qualogy_news
nl.linkedin.com/in/mschldmr

More Related Content

What's hot

Oracle WebLogic Diagnostics & Perfomance tuning
Oracle WebLogic Diagnostics & Perfomance tuningOracle WebLogic Diagnostics & Perfomance tuning
Oracle WebLogic Diagnostics & Perfomance tuningMichel Schildmeijer
 
Weblogic configuration & administration
Weblogic   configuration & administrationWeblogic   configuration & administration
Weblogic configuration & administrationMuhammad Mansoor
 
WebLogic JMX for DevOps
WebLogic JMX for DevOpsWebLogic JMX for DevOps
WebLogic JMX for DevOpsFrank Munz
 
Advanced WebLogic Monitoring: JMX and WLSDM Automation
Advanced WebLogic Monitoring: JMX and WLSDM AutomationAdvanced WebLogic Monitoring: JMX and WLSDM Automation
Advanced WebLogic Monitoring: JMX and WLSDM AutomationM. Fevzi Korkutata
 
Learn Oracle WebLogic Server 12c Administration
Learn Oracle WebLogic Server 12c AdministrationLearn Oracle WebLogic Server 12c Administration
Learn Oracle WebLogic Server 12c AdministrationRevelation Technologies
 
Oracle WebLogic Server Basic Concepts
Oracle WebLogic Server Basic ConceptsOracle WebLogic Server Basic Concepts
Oracle WebLogic Server Basic ConceptsJames Bayer
 
WebLogic Performance on SOLARIS SPARC Servers
WebLogic Performance on SOLARIS SPARC ServersWebLogic Performance on SOLARIS SPARC Servers
WebLogic Performance on SOLARIS SPARC ServersM. Fevzi Korkutata
 
Expect the unexpected: Anticipate and prepare for failures in microservices b...
Expect the unexpected: Anticipate and prepare for failures in microservices b...Expect the unexpected: Anticipate and prepare for failures in microservices b...
Expect the unexpected: Anticipate and prepare for failures in microservices b...Bhakti Mehta
 
Building WebLogic Domains With WLST
Building WebLogic Domains With WLSTBuilding WebLogic Domains With WLST
Building WebLogic Domains With WLSTC2B2 Consulting
 
Why jakarta ee matters (ConFoo 2021)
Why jakarta ee matters (ConFoo 2021)Why jakarta ee matters (ConFoo 2021)
Why jakarta ee matters (ConFoo 2021)Ryan Cuprak
 
Weblogic application server
Weblogic application serverWeblogic application server
Weblogic application serverAnuj Tomar
 
Oracle Weblogic Server 11g: System Administration I
Oracle Weblogic Server 11g: System Administration IOracle Weblogic Server 11g: System Administration I
Oracle Weblogic Server 11g: System Administration ISachin Kumar
 
Datasheet weblogic midvisionextensionforibmraf
Datasheet weblogic midvisionextensionforibmrafDatasheet weblogic midvisionextensionforibmraf
Datasheet weblogic midvisionextensionforibmrafMidVision
 

What's hot (20)

Oracle WebLogic Diagnostics & Perfomance tuning
Oracle WebLogic Diagnostics & Perfomance tuningOracle WebLogic Diagnostics & Perfomance tuning
Oracle WebLogic Diagnostics & Perfomance tuning
 
WebLogic and GraalVM
WebLogic and GraalVMWebLogic and GraalVM
WebLogic and GraalVM
 
Weblogic configuration & administration
Weblogic   configuration & administrationWeblogic   configuration & administration
Weblogic configuration & administration
 
WebLogic JMX for DevOps
WebLogic JMX for DevOpsWebLogic JMX for DevOps
WebLogic JMX for DevOps
 
WLS
WLSWLS
WLS
 
Advanced WebLogic Monitoring: JMX and WLSDM Automation
Advanced WebLogic Monitoring: JMX and WLSDM AutomationAdvanced WebLogic Monitoring: JMX and WLSDM Automation
Advanced WebLogic Monitoring: JMX and WLSDM Automation
 
Learn Oracle WebLogic Server 12c Administration
Learn Oracle WebLogic Server 12c AdministrationLearn Oracle WebLogic Server 12c Administration
Learn Oracle WebLogic Server 12c Administration
 
Why Play Framework is fast
Why Play Framework is fastWhy Play Framework is fast
Why Play Framework is fast
 
Jakarta EE 8 on JDK17
Jakarta EE 8 on JDK17Jakarta EE 8 on JDK17
Jakarta EE 8 on JDK17
 
Oracle WebLogic Server Basic Concepts
Oracle WebLogic Server Basic ConceptsOracle WebLogic Server Basic Concepts
Oracle WebLogic Server Basic Concepts
 
WebLogic Performance on SOLARIS SPARC Servers
WebLogic Performance on SOLARIS SPARC ServersWebLogic Performance on SOLARIS SPARC Servers
WebLogic Performance on SOLARIS SPARC Servers
 
Expect the unexpected: Anticipate and prepare for failures in microservices b...
Expect the unexpected: Anticipate and prepare for failures in microservices b...Expect the unexpected: Anticipate and prepare for failures in microservices b...
Expect the unexpected: Anticipate and prepare for failures in microservices b...
 
Weblogic - clustering failover, and load balancing
Weblogic - clustering failover, and load balancingWeblogic - clustering failover, and load balancing
Weblogic - clustering failover, and load balancing
 
Building WebLogic Domains With WLST
Building WebLogic Domains With WLSTBuilding WebLogic Domains With WLST
Building WebLogic Domains With WLST
 
Why jakarta ee matters (ConFoo 2021)
Why jakarta ee matters (ConFoo 2021)Why jakarta ee matters (ConFoo 2021)
Why jakarta ee matters (ConFoo 2021)
 
Weblogic application server
Weblogic application serverWeblogic application server
Weblogic application server
 
Oracle Weblogic Server 11g: System Administration I
Oracle Weblogic Server 11g: System Administration IOracle Weblogic Server 11g: System Administration I
Oracle Weblogic Server 11g: System Administration I
 
WildFly & WildFly Swarm
WildFly & WildFly SwarmWildFly & WildFly Swarm
WildFly & WildFly Swarm
 
Weblogic
WeblogicWeblogic
Weblogic
 
Datasheet weblogic midvisionextensionforibmraf
Datasheet weblogic midvisionextensionforibmrafDatasheet weblogic midvisionextensionforibmraf
Datasheet weblogic midvisionextensionforibmraf
 

Similar to Oracle Fuson Middleware Diagnostics, Performance and Troubleshoot

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...smallerror
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...xlight
 
Fixing twitter
Fixing twitterFixing twitter
Fixing twitterRoger Xia
 
Java troubleshooting thread dump
Java troubleshooting thread dumpJava troubleshooting thread dump
Java troubleshooting thread dumpejlp12
 
WebLogic Stability; Detect and Analyse Stuck Threads
WebLogic Stability; Detect and Analyse Stuck ThreadsWebLogic Stability; Detect and Analyse Stuck Threads
WebLogic Stability; Detect and Analyse Stuck ThreadsMaarten Smeets
 
T4T Training day - NodeJS
T4T Training day - NodeJST4T Training day - NodeJS
T4T Training day - NodeJSTim Sommer
 
Introduction to Micronaut - JBCNConf 2019
Introduction to Micronaut - JBCNConf 2019Introduction to Micronaut - JBCNConf 2019
Introduction to Micronaut - JBCNConf 2019graemerocher
 
NodeJS - Server Side JS
NodeJS - Server Side JS NodeJS - Server Side JS
NodeJS - Server Side JS Ganesh Kondal
 
Caching your rails application
Caching your rails applicationCaching your rails application
Caching your rails applicationArrrrCamp
 
John adams talk cloudy
John adams   talk cloudyJohn adams   talk cloudy
John adams talk cloudyJohn Adams
 
Chirp 2010: Scaling Twitter
Chirp 2010: Scaling TwitterChirp 2010: Scaling Twitter
Chirp 2010: Scaling TwitterJohn Adams
 
Silicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The SequelSilicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The SequelDaniel Coupal
 
"You Don't Know NODE.JS" by Hengki Mardongan Sihombing (Urbanhire)
"You Don't Know NODE.JS" by Hengki Mardongan Sihombing (Urbanhire)"You Don't Know NODE.JS" by Hengki Mardongan Sihombing (Urbanhire)
"You Don't Know NODE.JS" by Hengki Mardongan Sihombing (Urbanhire)Tech in Asia ID
 
How to JavaOne 2016 - Generate Customized Java 8 Code from Your Database [TUT...
How to JavaOne 2016 - Generate Customized Java 8 Code from Your Database [TUT...How to JavaOne 2016 - Generate Customized Java 8 Code from Your Database [TUT...
How to JavaOne 2016 - Generate Customized Java 8 Code from Your Database [TUT...Malin Weiss
 
JavaOne2016 - How to Generate Customized Java 8 Code from Your Database [TUT4...
JavaOne2016 - How to Generate Customized Java 8 Code from Your Database [TUT4...JavaOne2016 - How to Generate Customized Java 8 Code from Your Database [TUT4...
JavaOne2016 - How to Generate Customized Java 8 Code from Your Database [TUT4...Speedment, Inc.
 
Java black box profiling JUG.EKB 2016
Java black box profiling JUG.EKB 2016Java black box profiling JUG.EKB 2016
Java black box profiling JUG.EKB 2016aragozin
 
Realtime traffic analyser
Realtime traffic analyserRealtime traffic analyser
Realtime traffic analyserAlex Moskvin
 

Similar to Oracle Fuson Middleware Diagnostics, Performance and Troubleshoot (20)

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
 
Fixing twitter
Fixing twitterFixing twitter
Fixing twitter
 
Fixing_Twitter
Fixing_TwitterFixing_Twitter
Fixing_Twitter
 
Java troubleshooting thread dump
Java troubleshooting thread dumpJava troubleshooting thread dump
Java troubleshooting thread dump
 
WebLogic Stability; Detect and Analyse Stuck Threads
WebLogic Stability; Detect and Analyse Stuck ThreadsWebLogic Stability; Detect and Analyse Stuck Threads
WebLogic Stability; Detect and Analyse Stuck Threads
 
T4T Training day - NodeJS
T4T Training day - NodeJST4T Training day - NodeJS
T4T Training day - NodeJS
 
Introduction to Micronaut - JBCNConf 2019
Introduction to Micronaut - JBCNConf 2019Introduction to Micronaut - JBCNConf 2019
Introduction to Micronaut - JBCNConf 2019
 
NodeJS - Server Side JS
NodeJS - Server Side JS NodeJS - Server Side JS
NodeJS - Server Side JS
 
Caching your rails application
Caching your rails applicationCaching your rails application
Caching your rails application
 
John adams talk cloudy
John adams   talk cloudyJohn adams   talk cloudy
John adams talk cloudy
 
Chirp 2010: Scaling Twitter
Chirp 2010: Scaling TwitterChirp 2010: Scaling Twitter
Chirp 2010: Scaling Twitter
 
Silicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The SequelSilicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
 
Fastest Servlets in the West
Fastest Servlets in the WestFastest Servlets in the West
Fastest Servlets in the West
 
"You Don't Know NODE.JS" by Hengki Mardongan Sihombing (Urbanhire)
"You Don't Know NODE.JS" by Hengki Mardongan Sihombing (Urbanhire)"You Don't Know NODE.JS" by Hengki Mardongan Sihombing (Urbanhire)
"You Don't Know NODE.JS" by Hengki Mardongan Sihombing (Urbanhire)
 
How to JavaOne 2016 - Generate Customized Java 8 Code from Your Database [TUT...
How to JavaOne 2016 - Generate Customized Java 8 Code from Your Database [TUT...How to JavaOne 2016 - Generate Customized Java 8 Code from Your Database [TUT...
How to JavaOne 2016 - Generate Customized Java 8 Code from Your Database [TUT...
 
JavaOne2016 - How to Generate Customized Java 8 Code from Your Database [TUT4...
JavaOne2016 - How to Generate Customized Java 8 Code from Your Database [TUT4...JavaOne2016 - How to Generate Customized Java 8 Code from Your Database [TUT4...
JavaOne2016 - How to Generate Customized Java 8 Code from Your Database [TUT4...
 
Loom promises: be there!
Loom promises: be there!Loom promises: be there!
Loom promises: be there!
 
Java black box profiling JUG.EKB 2016
Java black box profiling JUG.EKB 2016Java black box profiling JUG.EKB 2016
Java black box profiling JUG.EKB 2016
 
Realtime traffic analyser
Realtime traffic analyserRealtime traffic analyser
Realtime traffic analyser
 

Recently uploaded

COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 

Recently uploaded (20)

COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 

Oracle Fuson Middleware Diagnostics, Performance and Troubleshoot

  • 1. ©2018 Michel Schildmeijer©2018 Michel Schildmeijer Oracle Fusion Middleware: Performance & Diagnostics solutions on WebLogic nlOUG Tech Experience ‘The Cloud is next’ Amersfoort, 7 June 2018 By Michel Schildmeijer,
  • 2. Me….. • Michel Schildmeijer, almost 52 years • Solutions Architect • Author of 2 books • Specialties: • WebLogic 6 to 12, OSB, Tuxedo, SOA, • Architect and design
  • 3. Contents • Troubleshoot Patterns & Methods • A slight view under the hood • Hints, tips & tricks
  • 5. • System errors, messages lost, bad performance • Where do I begin?? • What method should I follow? Troubleshoot overview
  • 6. • Using these troubleshooting methods helps you to: • A structured way of detecting issues • Get behind the cause of the problems that occur • Methods to follow: • Use of Operating system tools • Use of JDK tools • Use of WebLogic Server tools • Use of FMW Diagnostic tools • Use of Enterprise Manager ( Cloud Control )(?) • Use of OMC : Application Performance Management, Log Analytics(?) Troubleshoot Methods
  • 7. • Patterns to identify a “Hang” • Hangs related to the WebLogic Server, Java or some subset of resources • Generic Server Hang Pattern • JDBC Hang Pattern • EJB & RMI Hang Pattern • JMS Hang Pattern Troubleshoot Patterns
  • 8. • Server doesn’t respond to new requests • Requests time out. • Requests take longer and longer to process (may be on the way to a hang). • A server crash is not usually a symptom of a hung server but final state Server Hang
  • 9. • DriverManager.getConnection() in code. • SQL takes unexpectedly long time • JDBC connection pool hangs • Slow network causes to slow down or hang. • A deadlock all execute threads to hang & wait • RefreshMinutes or TestFrequencySeconds in conn pool causes hang • Long response JDBC conn pool shrinking and • No Sufficient DB resources JDBC Hang
  • 10. • Issues over RMI with remote JVM • Serialization • Cluster issues • JNDI lookup fails at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java: 341) EJB RMI Hang
  • 11. • Increasing pending messages • Sent by a producer • Received by a consumer • Lagging async client • Durable subscribers (to avoid missing placed messages in a topic) JMS Hang
  • 12. • O/S tools: • sar: system activity • mpstat: per-processor statistics • vmstat: virtual memory statistics • netstat: network statistics • iostat: Input/Output statistics • top WebLogic Troubleshoot overview
  • 13. • Start with a simple check – it seems so obvious but: • Is there enough storage • Is there enough RAM available • Does the system do a lot of paging WebLogic Troubleshoot overview
  • 15. The JavaVirtualMachine Java Source code .java Java bytecodeJVM RuntimeCompiler Java Runtime JDK Compiler
  • 16. • A Java application runs a piece of code • Code enters the JVM • It enters several stages: • Operations • Data structure • Transformation ( to bytecode) JVM from code to machine Java code JVMJVM Operat ons Data re Data Structu re Transformati on 110110 100
  • 17. • JVM process consists Java and JVM internal threads • JVM threads: Garbage Collect, Optimizer and Finalizer • Each thread has a stack to store runtime data • Overflow will happen if this stack will store more than its size • Each O/S has its default stacksize • Can be extended with -Xss Understanding Java threads
  • 18. • Runtime behaviour: o JVM runs as an OS process o Execute work through threading model (concurrency) o Garbage Collector • Each Weblogic Server (instance) runs inside a separate JVM (instance). JVM from code to machine
  • 19. • Crash can occur during • Compile process • Cause: invalid native code (C, C++, Libs) • Internal Code – Garbage Collect ( Segmentation Fault) • Cause: GC Fault leading into bad memory access • Long running GC can result in paging or CPU overload • Application code • Like 3rd Party drivers • (JDBC, MQ libraries) JVM from code to machine
  • 20. • What to look for in java thread dumps: • Low Memory Detector • (Daemon thread to detect and report low memory conditions) • CompilerThreadn • (Daemon thread/s to compile & optimize byte code) • Signal Dispatcher • (Daemon thread to respond OS signals) JVM troubleshoot
  • 21. • Finalizer • Daemon thread to handle objects in finalizer queue • Reference Handler • Daemon thread to handle objects in reference queue • VM Thread • Main thread from JVM • GC task thread#n • Thread/s to perform garbage collection JVM troubleshoot
  • 22. • Liveset and fragmentation • Heap gets fragmented when small & large objects are mixed store with various lifetimes. May cause longer GC • Garbage collect time and behavior JVM troubleshoot
  • 23. • Java HotSpot • Java Mission Control • JConsole • JVisualVM JVM tools
  • 24. • Java Mission Control provides: • Console and command line (jcmd) • Live JVM statistics • Graphs and metrics about local or remote systems • Basic JMX interaction with MBeans • Runtime data • An in-depth look at live metrics, like memory, CPU and stack trace on specific threads • Profile methods or exceptions & execute commands • Remote or local Java Mission Control -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=3614 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
  • 26. • Diagnostic Volume must be set higher than the default (Low) Java Mission Control
  • 27. Java Troubleshoot • Some well known errors • Class not found – missing or incorrect Classpath set during startup (server start params or setDomainEnv) or adding domain libs • NullPointers – A value is expected but it’s not there (null): • Poorly written • Bad error handling • Missing config • Stack Overflow – Recursive procedures, a method with no exit • Too Many Open Files – Ulimit settings
  • 29. • Server applies to: Managed server or AdminServer • Partial or total (OutOfMemory) • Lack of resources/deadlocks Server Hangs
  • 30. • The server does not respond to new requests. • Requests time out. • Requests take longer and longer to process (may be on the way to a hang). • A server crash is not usually a symptom of a hung server but may follow. Symptoms
  • 31. Deadlock Thread 1 Thread 2 Resource A Resource B Lock Wait LockWait Deadlock
  • 32. Possible Causes • All threads waiting for RJVM, RMI responses. • Garbage Collection taking too much time. • Long Running JDBC calls or , connection leaks, deadlocks • JVM hang during code optimization • JSP compilation under heavy load.
  • 33. • WebLogic Server generic: • WebLogic Diagnostics Framework • Logs • WLST • JVM specific : • Java Mission Control, • JStat, • JConsole Tools/Methods
  • 34. • Force a JVM dump by: • kill -3 <pid> … stack in STDOUT • Usually in <domain>/servers/logs/<serverinstance>/<serverinstance.out> • See more in the next slides about WebLogic Threads Tools/Methods
  • 36. • A threaddump is a snapshot of the state of a WebLogic Server Instance • Different ways to take a threaddump: • kill -3 <pid> the O/S way • WebLogic Console • WLST(WebLogic Scripting Tool) cd ('Servers') ls() cd ('AdminServer') ls() threadDump() WebLogic taking threaddumps
  • 38. WebLogic Threads • Active threads: • Are either processing or waiting for requests • Can be tagged with additional attributes such as Idle or Hogging • Standby threads are not used to process requests unless throughput is needed. • Workmanagers can control threads of specific parts
  • 39. WebLogic Stuck Threads Running for a long time • Deadlocked • Infinite loop • Overloaded server • WebLogic: • Periodically checked how long threads are running • Logs a warning if a thread becomes stuck • Creates additional threads to handle new requests • Sets its state to Failed after specified nr of threads become stuck • Individual work managers can detect stuck threads & reject new requests.
  • 41. • Hanging threads will be diagnosed as STUCK • Default time to get STUCK is 600 secs • Thread has not completed it’s work The sense of taking threaddumps
  • 42. • Thread Dumps can give you: • View of the state of application server threads at that instant in time • Information about potential issues such as • Hot spots within code which seem to be called often • Portions of code where the application seems to be hung • Locking and thread synchronization issues in an application The sense of taking threaddumps
  • 43. • Take several dumps to investigate various states • Threads high in stack can be treated as idle, mostly waiting for some work to do • Blocked or Wait on socketAccept() listener thread waiting for socket connections The sense of taking threaddumps
  • 44. • WebLogic specific: • Ignore threads in ExecuteThread.waitForRequest() • Muxer and processSocket threads are for housekeeping and listening • A Muxer which is blocked should not be waiting for a lock by a non Muxer The sense of taking threaddumps
  • 45. • In the AdminConsole WebLogic taking threaddumps
  • 46. • Are meant to synchronize between threads • Lock contention is a thread waiting for another to be released • Fat locks: Have a history of contention (several threads try to take the lock simultaneously), or waiting for notification • Thin locks: Without any contention. • Recursive locks: A lock on a thread several times without having been released. Understanding Locks
  • 48. • Wrong or missing drivers in Classpath • Connection errors by: • Usernames/passwords • Connection leaks by bad apps • Set Inactive timeout to a value (default is 0) • Insufficient connection parameters • Max Capacity, Reserve Time Out(-1) • Invalid JDBC URLS weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception wa… JDBC additonal errors
  • 49. WebLogic JDBC mechanism Application JDBC Driver Database1. Connection 2. Statement 3. SQL 4. Results 5. Close Pool Conn1 Conn2 JNDI Lookup
  • 50. • Using dumpPool and WLDF capture JDBC Profiling
  • 51. • DebugJDBCConn - Trace all connections. • DebugJDBCSQL - Trace all JDBC API calls • DebugJDBCInternal - low-level internal data source activities. • DebugJTAJDBC - Trace transaction management Debug Flags
  • 52. • Bad implemented applications can starve connections and data • Cache or “hog” connections instead of releasing them. • Do not explicitly release connections using close(). • Automatically reclaim with “Inactive Connection Timeout” Connection Leaks
  • 54. • Incorrect targeting • Connection factories to subdeployments instead of to entire servers • Targeting distributed destinations to entire servers instead to subdeployments • Using multiple subdeployments within the same module • Configure JMS migration without a cluster • Convert a single-server domain to a cluster Common JMS issues
  • 55. • OOM in case of large amount of pending messages • JMS Quota can prevent that • Message paging – messages in virtual memory(file or db) • Flow control Common JMS issues - OOM
  • 56. • Expired messages • Set a TimeToLive & Exp policy • JMS system failure and nonpersistent messages • Set a delivery and time to delivery mode • Insufficient retry settings • Message delay settings • Nondurable topic subscriptions Common JMS issues – Lost messages
  • 57. • Applications used to automatically consume JMS • Often used in SOA Suite applications ( SOA Suite, OSB ) • Manages itself ( pooling ) • A few debug flags or use WLDF to diagnose MDBs
  • 58. WebLogic Log and debug methods
  • 59. • Logging services are important for diagnostics • WebLogic produces output by using a Message Catalog • Using the standard java.util.logging • Can be extended using log4j modules • All kinds of events, like startup/shutdown, appl.deployments, subsystem failures are written to logs WebLogic Log and Debug options
  • 61. • Several severity levels like: • TRACE, DEBUG,INFO .. • Important to set the proper level • For FMW specific components, ODL-HANDLER can be used WebLogic Log and Debug options
  • 62. • Subsystem logs can be from: • HTTP • Transaction Manager • JDBC • JMS server • WebLogic Auditing provider • FMW Logs WebLogic Log and Debug options
  • 63. • Debug • Setting debug options in the Admin Console • Or use –D options in startup script -Dweblogic.debug.DebugJDBCSQL=true Prints information about all JDBC methods WebLogic debug options
  • 64. • Or use WLST serverConfig() or domainConfig() debug = getMBean('/Servers/mserver/ServerDebug/mserver') debug.setDebugJDBCInternal(true) WebLogic debug options
  • 65. • OpenSource Log engine from Apache • Enable the LogMBean.isLog4jLoggingEnabled attribute • In WLST: cmo.setLog4jLoggingEnabled(true) • Add Log4j classes, • WL_HOME/server/lib/wllog4j.jar & log4j.jar to server CLASSPATH WebLogic Log4j
  • 66. • Enable log4j in AdminConsole WebLogic Log4j
  • 67. • As an extension on the default logging handlers • Comes with FMW installation, not WebLogic default • Logging view and config in Fusion Middleware Control • Can be set to several trace levels • Output in <servername>-diagnostic.log Oracle diagnostic Logging (ODL)
  • 69. Oracle diagnostic Logging • Edit or create your own handler
  • 70. Oracle Diagnostic Logging • Edit or create your own handler
  • 72. WebLogic Diagnostic Framework •Included with WebLogic •Define Watches to monitor runtime conditions and notify in case of failure. •Collect configuration and runtime data for analysis. •Capture images to preserve system state in failure situation. Watch & Notification Instrumentation Data Collectors Image Capture Data Providers Archive Accessor Monitoring Dashboard
  • 73. • Framework for monitoring and diagnostics • Consolidated access for MBean data, instrumentation, logging, debugging • Source for monitoring data • Server & application instrumentation notifications based on the output : log, email, SNMP, JMX, JMS • Request dyeing and tracing • Data archiving for later access • WLDF Dashboard WLDF
  • 74. • Key Features • Gathering input from JMX MBeans • WebLogic subsystems and application code can be instrumented to provide events and watches • Notifications can be defined for events and watches • Console extension for monitoring of current and historical data • Data archiving WLDF
  • 75. • Can be configured on filesystem or Database • Configure with Data retirement WLDF
  • 76. • Create Diagnostic Module • Collect Metrics. A collection includes: • The MBean type to query • The specific MBean instance names to query (all instances, • by default) • The MBean attributes to collect (all attributes, by default) • How often to gather data • Create Watches & Notifications WLDF
  • 77. Diagnostic Frameworks & Diagnostics Tools in FMW • WLDF – WebLogic Diagnostic Framework • DFW – Diagnostic Framework o Dynamic Monitoring Service (DMS) o Oracle Diagnostic Logging (ODL) o Incident Packaging System (IPS or ADRCI) • Additional Tools o Selective Tracing
  • 79. Diagnostic Tools for SOA and FMW • Predefined Watches and Diagnostic Dumps • Detect, diagnose and resolve problems • Use WLS and SOA MBeans and DMS Metrics. • Detect critical failures and collect diagnostic dumps containing relevant diagnostic information like logs, metrics, server images, • Create incidents in the ADR (Automatic Diagnostic Repository)
  • 80. Integration of WLDF, DFW and Diagnostic Tools for SOA and FMW Monitor System • Predefined Watches • Predefined Diagnostic Dumps Critical Failure • Collect Diagnostic Dumps • Create Incident Collect • Collect Data using EM or Email
  • 81. Dynamic Monitoring Service • DMS for runtime metrics applications and components • Can be monitored through WLDF Watches to trigger data collections through DFW.
  • 82. Dynamic Monitoring Service (DMS) • Direct accessible through http://<wlshost>:7001/dms • Give pretty much raw unstructured results • Better option is to use FMW Console
  • 83. DMS in FMW Console
  • 84. Diagnostic Framework (DFW) • collect a pre-configured set of information at exactly the right time through a triggering mechanism. • Triggered from Oracle Diagnostic Logging events or WebLogic Diagnostic Framework Watches. • Information can also be collected manually using WLST
  • 85. Oracle Diagnostic Logging (ODL) • ODL is the primary logging for FMW • The ODL log entries are in XML format
  • 86. Selective Tracing (ST) • Selective Tracing is a facility available through Enterprise Manager to limit the scope of trace logging. • This allows for debugging of a production system without overloading the system with logging activity.
  • 90. • Performance : How system’s & applications response time and throughput are affected by adding load. • Capacity : Maximum threshold a system is under a given set of conditions. • Scalability : how does a system responds to increasing load by adding additional resources. • Key indicators are : • Response time • Throughput Performance Terms and Definitions
  • 91. • Locate & minimize bottlenecks • Gather info about Application(s) • Concurrent number of users and interfaces. • Number and size of app requests. • Amount of data and its consistency. • Determining your target CPU utilization • Tune your Database(s) (customers or repos) • Tune WebLogic Server Performance Parameters • Tune Your JVM • Tune the Operating System • Tuning WebLogic Persistent Store Performance objectives
  • 92. • Investigate and analyze! • Always consider before implementing : • What am I changing • Has the bottleneck gone away? • Is there a new bottleneck? Tune WebLogic Server
  • 93. • Enable by default • Will use a O/S threadpool instead from “borrowing” from WebLogic default Exec.Q • Native muxers use platform-specific native binaries to read data from sockets • Non native or Java sockets : • Uses pure Java to read data from sockets. • Blocks until there is data to be read from a socket(can cause issues on server level) Native I/O and muxers
  • 94. • Represents nr of simultaneous operations performed by apps that use the execute Q WebLogic Thread Pool WebLogic Socket Muxer Application WebLogic Kernel assigns Execute Req to ExecuteThread 1 ExecuteThread 1 handles the request (for example executes a JDBC request) WebLogic Kernel handles self tuning ThreadPool, Workmanagers, allocations The app now handles the thread itself and gives back result Execute Request
  • 95. • Different execute Q’s in 11g for applications and administration work • weblogic.kernel.default, RMI, HTTP • 1 default Queue • In 12c Single threadpool • Self tuning • Work managers • Prioritize by rules • Different constraints regarding min max response capacity -Dweblogic.threadpool.MinPoolSize=5 -Dweblogic.threadpool.MaxPoolSize=5 WebLogic Thread Count
  • 96. • Default All apps use all execute queues • Configuring multiple execute queues can provide additional control for applications. • By using multiple execute queues, selected applications have access to a fixed number of execute threads, regardless of the load on WebLogic Applications and Execute Queues
  • 97. • ThreadPoolPercentSocketReaders • Default set to 33 ( range is 1-99) • Allocating threads as readers can increase accepting requests Threads as SocketReaders
  • 98. • Piece of memory for read and write data to sockets (network layer) • A pool of chunks is maintained • For applications with large amounts of data • Parameters are: • weblogic.Chunksize • Size of a chunk (in bytes). Set to network's maximum transfer unit (MTU), after subtract it from any Ethernet or TCP header sizes • weblogic.utils.io.chunkpoolsize • Sets the maximum size of the chunk pool. Monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.Chunk. • weblogic.PartitionSize • Sets the number of pool partitions used (default is 4). Partitioning the thread pool spreads the potential for contention over more than one partition To be set in the setDomainEnv.sh WebLogic ChunkSize
  • 99. • Specifies how many TCP connections can be buffered. These are on the TCP stack but not yet accepted by an app. • Default is 50 • Maximum depends on O/S WebLogic Backlog
  • 100. • When to tune? • When getting “connection refused” in high throughput apps (SOA composites) • If tuning, increase by 25% of it’s default until errors dissapear WebLogic Backlog
  • 101. • Prepared Statement Cache • Keeps compiled SQL statements in memory • Set to LRU • A data source with 10 connections deployed on 2 servers, with the Statement Cache to 10 (default), 200 cursors can be used on the database for these cached statements • MaxCapacity • Max of physical connections • Db drivers can limit connections • Number should equal number of client sessions • LLR instead of XA • Decrease 2PC overhead • Disable wrapping datatypes (debug, track, transaction control of blb, clob) JDBC Connectionpool tuning
  • 102. • In some cases it takes ages after startup to continue • Generate a stackup Tuning Startup
  • 103. • This happens when using default seed generator: • Use of dev/urandom during startup in stead of random • Specifiy in startup properties - Djava.security.egd=file:///dev/urandom • Or in $JAVA_HOME/jre/lib/security/java.security securerandom.source=file:/dev/./urandom Tuning Startup
  • 104. • Keep MDS and Dehydration store clean • MDS delete older label deployments • Dehydration store purge strategy Tuning Startup SOA
  • 105. • Configure Transaction settings: • BPEL SyncMaxWaitTime • BPEL EJBS • JTA • DB Tuning other SOA
  • 106. : • Discovery cache age to 28800000 • discovery wait time to 30000 • Set the cache results to true • Dumpinterval to 10800 with a maximum of 75 MB • Enable Large Repository • Increase the collector configuration settings for DMS: • Use an interval of 300 secs with a remove cycle of 3 • And use an interval of 120 secs with a remove cycle of 2, set this to default • Restrict the amount of displayed instance information to 12 hours • Tuning AdminServer EM SOA
  • 107. • Most aspects of JVM tuning relate to: • Sufficient memory heap • An efficient garbage collection scheme • Classloading • Compile/interpret Tuning the JVM
  • 108. • Most important to focus on: • Garbage collection scheme • Compaction/Defrag • Free up memory • Memory heap • Is a runtime repository for live objects, dead objects and free memory. • No more than 75% of total RAM • The JVM heap sets frequency & duration GC • Large heap size : • full garbage collection slower and less frequent • Small heap size • full garbage collection is faster & higher frequency Tuning the JVM
  • 109. • Goal is: • Minimize Garbage Collect • Set heapsize that GC does no happen all the time Tuning the JVM
  • 110. • Investigate with –verbosegc option • How often does GC run • How long does it take? • Full garbage collection should not take longer than 3 to 5 seconds. Lower heap if major GC time is greater. Tuning the JVM
  • 111. • Set Xms and Xmx parameters to specify initial and Max HeapSize • Xmx is the Java Object Heap, not the System Heap –> which is larger • Values equal will cause no overhead of heap management • PermSize and MaxPermSize For stored classes, methods Be aware: • Java methods, thread stacks, native handles are in memory separate from heap Tuning the JVM
  • 112. “The Cloud is next”
  • 114. Any Questions? Let’s keep in touch!! The ACE Community mschildmeijer@qualogy.com https://community.oracle.com/blogs/mnemonic https://www.qualogy.com/nl/techblog/author/michel- schildmeijer
  • 115. Any Questions? Let’s keep in touch!! @MNEMONIC01 @Qualogy_news nl.linkedin.com/in/mschldmr