1. Linux / Java
Troubleshooting guide
Jérôme Kehrli @ NetGuardians / 2015

2. 1.The Linux System
2.The Java Virtual Machine
3.Linux in Practice
4.Java Troubleshooting

3. 310/04/2018
The Linux System

4. Linux is an operating system.
It is the software on a computer that enables applications and the computer operator to access the devices on the
computer to perform desired functions.
The operating system (OS) relays instructions from an application to, for instance, the computer's processor.
The processor performs the instructed task, then sends the results back to the application via the operating system.
The Linux Market
Linux represented a $25 billion ecosystem in 2008.
It powers everything from the New York Stock Exchange to mobile phones to supercomputers to consumer
devices.
The Linux Kernel
True Open Source : as an open operating system, Linux is developed collaboratively, meaning no one company is
solely responsible for its development or ongoing support. Licensed under GNU GPL (General Public License)
Over 1,000 developers, from at least 100 different companies (past two years alone, over 3,200 developers from
200 companies)
GNU/Linux
Hundreds of Distributions !
Worldwide market shares :
~5% PC (notebooks and desktops)
~60% Tablets (kernel only !)
~80% Smartphones (kernel only !)
>80% Public Servers on Internet
~80% Mainframes
~99% Super Computers 4
Linux Operating System

5. Unix
The Unix OS : conceived and implemented in 1969 at AT&T's Bell Laboratories by Ken
Thompson, Dennis Ritchie, Douglas McIlroy, and Joe Ossanna.
10/04/2018 5
A bit of history : Unix
First released in
1971, Unix was
written entirely in
assembly language
Later, in 1973, it was
rewritten in the C
programming
language by Dennis
Ritchie (with
exceptions to the
kernel and I/O).
The availability of a
high-level language
implementation of
Unix made its porting
to different computer
platforms easier.
Complete tree : http://www.levenez.com/unix/unix.pdf

6. Linux
In 1991, while attending the University of Helsinki, Linux Torvalds became frustrated by the licensing of MINIX,
which at the time limited it to educational use only.
He began to work on his own operating system kernel, which eventually became the Linux kernel.
Torvalds began the development of the Linux kernel on MINIX and applications written for MINIX were also used on
Linux.
Later GNU applications also replaced all MINIX components, because it was advantageous to use the freely
available code from the GNU Project with the fledgling operating system.
Developers worked to integrate GNU components with the Linux kernel, making a fully functional and free operating
system.
The GNU Project
Richard Stallman announced his intent to start coding the GNU Project - whole free-software operating system -
in a Usenet message in September 1983
By 1992, the GNU project had completed all of the major operating system components except for their kernel,
GNU Hurd.
With the release of the Linux kernel, for the first time it was possible to run an operating system composed
completely of free software.
Though the Linux kernel is not part of the GNU project, it was developed using GCC and other GNU programming
tools and was released as free software under the GNU General Public License.
GNU/Linux
GNU/Linux is a term promoted by the Free Software Foundation (FSF) and its founder Richard Stallman.
They argue that GNU was a longstanding project begun in 1984 to develop a free operating system, and that when
the Linux kernel was independently created in 1991, it merely provided a substantial missing piece
6
History of Linux and GNU

7. Linux is a Unix-like and mostly POSIX-compliant computer operating system (OS) assembled under the model of free
and open-source software development and distribution
POSIX an acronym for Portable Operating System Interface, is a family of standards specified by the IEEE Computer Society
for maintaining compatibility between operating systems.
POSIX defines the application programming interface (API), along with command line shells and utility interfaces, for software
compatibility with variants of Unix and other operating systems
A Linux-based system is a modular Unix-like operating system, deriving much of its basic design from principles
established in Unix during the 1970s and 1980s.
Such a system uses a monolithic kernel, the Linux kernel, which handles process control, networking, access to the
peripherals, and file systems.
Device drivers are either integrated directly with the kernel, or added as modules that are loaded while the system is running
Properties:
Portable - Portability means software can works on different types of hardware in same way. Linux kernel and application
programs supports their installation on any kind of hardware platform.
Open Source - Linux source code is freely available and it is community based development project. Multiple teams works in
collaboration to enhance the capability of Linux operating system and it is continuously evolving.
Multi-User - Linux is a multiuser system means multiple users can access system resources like memory/ ram/ application
programs at same time.
Multiprogramming - Linux is a multiprogramming system means multiple applications can run at same time.
Preemptive multitasking - Preemptive multitasking allows the computer system to guarantee more reliably each process a
regular "slice" of operating time (linux supports as far as real-time multitasking)
Multithreading - threads are effectively processes that run in the same memory context and share other resources with their
parent processes, such as open files
Hierarchical File System - Linux provides a standard file structure in which system files/ user files are arranged.
Shell - Linux provides a special interpreter program which can be used to execute commands of the operating system. It can
be used to do various types of operations, call application programs etc.
Security - Linux provides user security using authentication features like password protection/ controlled access to specific
files/ encryption of data.
7
GNU/Linux Specificities

8. 10/04/2018 8
Linux Kernel Architecture

9. 10/04/2018 9
Linux API
Application (JVM …)

10. 10/04/2018 10
Reminder : a process in memory
(Virtual Memory Mapping)

11. 10/04/2018 11
Process Scheduler Principles
Time Sharing
Time-sharing was developed in the late 1950s out of the realization that a single expensive computer could be
efficiently utilized if a multitasking, multiprogramming operating system allowed multiple users simultaneous
interactive access.
Typically an individual user would enter bursts of information followed by long pauses; but with a group of users
working at the same time, the pauses of one user would be filled by the activity of the others
Preemption
Preemptive multitasking involves the use of an interrupt mechanism which suspends the currently executing
process and invokes a scheduler to determine which process should execute next. Therefore, all processes will get
some amount of CPU time at any given time.
Preemptive multitasking allows the computer system to more reliably guarantee each process a regular "slice" of
operating time. It also allows the system to rapidly deal with important external events like incoming data, which
might require the immediate attention of one or another process.

12. Notion of Process and Threads
Both processes and threads are independent sequences of execution. The typical difference is that threads (of the same
process) run in a shared memory space, while processes run in separate memory spaces.
A thread is the entity within a process that can be scheduled for execution. All threads of a process share its virtual address
space and system resources. In addition, each thread maintains exception handlers, a scheduling priority, thread local
storage, a unique thread identifier, and a set of structures the system will use to save the thread context until it is scheduled.
On Linux systems, the Stack zone we have seen previously is used to implement the call stack of all the threads from the
same process
12
Multithreading
Threads instead of forking
The POSIX thread libraries are
a standards based thread API
for C/C++. It allows one to
spawn a new concurrent
process flow.
It is most effective on multi-
processor or multi-core
systems where the process
flow can be scheduled to run
on another processor thus
gaining speed through parallel
or distributed processing.
Threads require less overhead
than "forking" or spawning a
new process because the
system does not initialize a
new system virtual memory
space and environment for the
process.

13. 10/04/2018 13
Context Switching
Context Switching
A context switch is the process of storing and restoring the state (more specifically, the execution context) of a
process or thread so that execution can be resumed from the same point at a later time.
This enables multiple processes to share a single CPU and is an essential feature of a multitasking operating
system.
Most commonly, within some scheduling scheme, one process must be switched out of the CPU so another
process can run.
On a pre-emptive multitasking system,
the scheduler may also switch out
processes which are still runnable.
To prevent other processes from being
starved of CPU time, preemptive
schedulers often configure a timer
interrupt to fire when a process exceeds
its time slice.
This interrupt ensures that the scheduler
will gain control to perform a context
switch.
One can distinguish:
thread switch (switching between two
threads within a given process) and
process switch (switching between two
processes)
A thread-switch is obviously much cheaper
in terms of CPU cost than a full process
switch

14. Virtual memory is a technique that allows
the execution of processes which are not
completely available in memory.
Key component in Multi-Programming
Demand Paging
When we want to execute a process, we swap
it into memory.
Rather than swapping the entire process into
memory, however, we use a lazy swapper
called pager.
Virtual Memory and MMU

15. 10/04/2018 15
Linux Process Memory Layout

16. Functions of the call stack
Storage of functions local variables
Parameter passing
Stack and frame pointers
When stack frame sizes can differ, such as between
different functions or between invocations of a
particular function, popping a frame off the stack
does not constitute a fixed decrement of the stack
pointer.
At function return, the stack pointer is instead
restored to the frame pointer, the value of the stack
pointer just before the function was called.
Storing the address to the caller's frame
In most systems a stack frame has a field to contain
the previous value of the frame pointer register.
Example
If a subroutine DrawSquare calls a subroutine
DrawLine from four different places, DrawLine
must know where to return when its execution
completes.
To accomplish this, the address following the call
instruction, the return address, is pushed onto the
call stack with each call.
16
Call Stack Principle
A call stack – also called execution stack - is a stack data structure that stores information
about the active subroutines of a computer program

17. Disk Buffering and buffer cache
Reading from a disk is very slow compared to accessing memory. In addition, it is
common to read the same part of a disk several times during short periods of time.
By reading the information from disk only once and then keeping it in memory, one
can speed up all but the first read.
This is called disk buffering, and the memory used for the purpose is called the
buffer cache.
Disk buffering works for writes as well.
Data that is written is often soon read again, so putting data that is written in the
cache is a good idea.
By only putting the data into the cache, not writing it to disk at once, the program that
writes runs quicker. The writes can then be done in the background, without slowing
down the other programs.
Cache freeing
Since memory is a finite and scarce resource, the buffer cache cannot be big
enough.
When the cache fills up, the data that has been unused for the longest time is
discarded and the memory thus freed is used for the new data.
Syncing and issues
Because of this, you should never turn off the power without using a proper
shutdown procedure or remove a floppy from the disk drive until it has been
unmounted.
The sync command flushes the buffer, i.e., forces all unwritten data to be written to
disk, and can be used when one wants to be sure that everything is safely written.
In traditional UNIX systems, there is a program called update running in the
background which does a sync every 30 seconds, so it is usually not necessary to
use sync.
Linux has an additional daemon, bdflush, which does a more imperfect sync more
frequently to avoid the sudden freeze due to heavy disk I/O that sync sometimes
causes.
Under Linux, you do not need to do anything to make use of the cache, it happens
completely automatically. 17
Buffer Cache

18. 1810/04/2018
The Java Virtual Machine

19. Java is a general-purpose computer programming language that is concurrent, class-based and
object-oriented.
Java is specifically designed to have as few implementation dependencies as possible. It is intended
to let application developers "write once, run anywhere" (WORA), meaning that compiled Java code
can run on all platforms that support Java without the need for recompilation.
Java applications are typically compiled to bytecode that can run on any Java virtual machine (JVM)
regardless of computer architecture.
The language derives much of its syntax from C and C++, but it has fewer low-level facilities than
either of them.
As of 2015, Java is one of the most popular programming languages in use, particularly for client-
server web applications, with a reported 9 million developers.
The latest version is Java 8, which is the only version currently supported for free by Oracle, although
earlier versions are supported both by Oracle and other companies on a commercial basis.
Java software runs on everything from laptops to data centers, game consoles to scientific
supercomputers.
10/04/2018 19
Java

20. A Java virtual machine (JVM) is an abstract computing machine that enables a computer to run a Java
program.
There are three notions of the JVM: specification, implementation, and instance.
The specification is a document that formally describes what is required of a JVM implementation. Having a single
specification ensures all implementations are interoperable.
A JVM implementation is a computer program that meets the requirements of the JVM specification.
An instance of a JVM is an implementation running in a process that executes a computer program compiled into
Java bytecode.
Java Runtime Environment (JRE) is a software package that contains what is required to run a Java
program.
It includes a Java Virtual Machine implementation together with an implementation of the Java Class Library.
The Oracle Corporation, which owns the Java trademark, distributes a Java Runtime environment with their Java
Virtual Machine called HotSpot.
Java Development Kit (JDK) is a superset of a JRE and contains also tools for Java programmers, e.g.
a javac compiler.
The Oracle Corporation, which owns the Java trademark, distributes a Java Runtime environment with their Java
Virtual Machine called HotSpot.
Java Development Kit is provided free of charge either by Oracle Corporation directly, or by the OpenJDK open
source project, which is governed by Oracle.
10/04/2018 20
Java Virtual Machine

21. Java was originally developed by James Gosling at Sun Microsystems (which has since been acquired by Oracle
Corporation) and released in 1995 as a core component of Sun Microsystems' Java platform.
James Gosling, Mike Sheridan, and Patrick Naughton initiated the Java language project in June 1991
Java was originally designed for interactive television, but it was too advanced for the digital cable television industry at the
time
Gosling designed Java with a C/C++-style syntax that system and application programmers would find familiar
In 1997, Sun Microsystems approached the ISO/IEC JTC 1 standards body and later the Ecma International to
formalize Java, but it soon withdrew from the process.
Java remains a de facto standard, controlled through the Java Community Process.
On November 13, 2006, Sun released much of its Java virtual machine (JVM) as free and open-source software,
(FOSS), under the terms of the GNU General Public License (GPL).
On May 8, 2007, Sun finished the process, making all of its JVM's core code available under free software/open-source
distribution terms, aside from a small portion of code to which Sun did not hold the copyright
Following Oracle Corporation's acquisition of Sun Microsystems in 2009–10, Oracle has described itself as the "steward
of Java technology with a relentless commitment to fostering a community of participation and transparency".
This did not prevent Oracle from filing a lawsuit against Google shortly after that for using Java inside the Android SDK.
On April 2, 2010, James Gosling resigned from Oracle.
10/04/2018 21
History of Java

22. 10/04/2018 22
Java / ByteCode / MachineCode ?

23. 10/04/2018 23
Java Architecture

24. Young Generation
Young generation is the place where all the new objects are created. When young generation is filled, garbage
collection is performed.
This garbage collection is called Minor GC.
Young Generation is divided into three parts – Eden Memory and two Survivor Memory spaces.
Most of the newly created objects are located in the Eden memory space.
When Eden space is filled with objects, Minor GC is performed and all the survivor objects are moved to one of the survivor
spaces.
Objects that are survived after many cycles of GC, are moved to the Old generation memory space. Usually it’s done by
setting a threshold for the age of the young generation objects before they become eligible to promote to Old generation.
Old Generation
Old Generation memory contains the objects that are long lived and survived after many rounds of Minor GC.
Usually garbage collection is performed in Old Generation memory when it’s full.
Old Generation Garbage Collection is called Major GC and usually takes longer time.
10/04/2018 24
The JVM memory Model
Permanent Generation
Permanent Generation or “Perm Gen”
contains the application metadata required
by the JVM to describe the classes and
methods used in the application. Note that
Perm Gen is not part of Java Heap
memory.
Perm Gen is populated by JVM at runtime
based on the classes used by the
application. Perm Gen also contains Java
SE library classes and methods. Perm
Gen objects are garbage collected in a full
garbage collection.

25. 10/04/2018 25
Call stack in Java
Reminder : call stack principle
In the normal flow of a program, when the Java Virtual Machine is running your code, one method after another
will be executed, starting with the main method.
When a method has its turn at the head of the programming queue it said be on top of the stack.
After the whole of the method has been executed, it is taken off the stack to be replaced by the next method in the
queue.
Java Threads are implemented on top of system (POSIX) threads in Linux
Call stack in Java
In terms of technical behaviour and perspective, the call stack - or execution stack - in the JVM is implemented
exactly as the call stack we have seen in Linux
However, due to the interpreted nature of Java, the JVM can analyze and operate the call stack and return
information about its structure from any portion of the runtime
This can be used by a system administrator to dynamically analyze the JVM and what's its threads are doing
Also, this runtime analysis of call stacks is at the key of the reporting system behind Java's Exception System

26. 10/04/2018 26
Exception Stack Trace Example
Let's have a look at the stack trace:
This is a very simple stack trace. If we start at the beginning of the list of "at ...", we can
tell where our error happened. What we're looking for is the topmost method call that is
part of our application. In this case, it's:
To debug this, we can open up Book.java and look at line 16, which is:
This would indicate that something (probably title) is null in the above code

27. 27
Chain of Exceptions
Sometimes applications will catch an Exception and re-throw it as the cause of another
Exception. This typically looks like:
This might give you a stack trace that looks like:
What's different about this one is the "Caused by". Sometimes exceptions will have
multiple "Caused by" sections. For these, you typically want to find the "root cause",
which will be one of the lowest "Caused by" sections in the stack trace. In our case, it's:
Again, with this exception we'd want to look at line 22 of Book.java to see what might
cause the NullPointerException here.

28. 2810/04/2018
Linux In Practice

29. Any computer machine has three hard limits:
CPU : processing abilities
RAM : the amount of data it can manipulate in memory
IOs : the throughput of data to/from the harddrives or the network
Any computer can only go as fast as it is allowed to by these limits !
Understanding the system limits

30. CPU bound
If a system’s CPU enables it to execute one billion operations per seconds, it would
need 10 seconds to execute a 10 billion operations program
But that is only theoretical of course !
In practice, that program likely performs IOs, should it be only reading itself from the
filesystem, or TTY IOs, and as such it will require more than 10 secs to complete.
Also, Linux is a multitasking system and within this 10 seconds period, that program
will have to share the CPU with dozens of other programs. Even if it is the only
“active” program from a user perspective, in practice a lot of kernel processes are
awaken hundreds of times per seconds on an idle linux system.
One consequence of the following : imagine that a user launches 10 instances
(processes) of this program simultaneously, they would all answer only after 100
seconds !
And that is the best case : in practice, context switching takes time on its own.
Understanding the system limits
(cont’d)

31. Memory bound
When the memory is full, thanks to Virtual Memory, the OS will start offloading the
RAM on the swap partition.
Doing that will overwhelmingly slow things down !
In the worst case, if the amount of required memory to fit all process of memory is
several times bigger that the actual size of the RAM, the OS ends up reading the
process (or most of its pages) from the filesystem at each Context-Switch => Such a
system ends up appearing completely unresponsive to the user.
In Java, things are even worst : when the memory becomes slowly full, the Garbage
Collector ends up taking up to 100% of the actual processing time of the Java
program. “Fortunately”, when this happens, the JVM detects it and crashes
Understanding the system limits
(cont’d)

32. IO Bound
IOs on the filesystem are slower than reading from or writing to the RAM by several
orders of magnitude
IOS on the network are slower than IOs on the filesystem by several order of
magnitude
A machine running IO intensive processes will answer very slowly even though from
the CPU perspective it seems to be doing nothing !
Understanding the system limits
(cont’d)

33. Long story short :
Slowness from the user perspective = most of the time that a report, a forensic
analysis, etc. really needs to read millions of row of data and process them and, well,
that takes just time !
However, this can only be confirmed by a close monitoring of these 3 resources.
CPU : processing abilities
RAM : the amount of data it can manipulate in memory
IOs : the throughput of data to/from the harddrives or the network
We will now wee the tools we have at our disposal to understand and find out what is
going on in regards to these 3 resources
10/04/2018 33
Understanding the system limits
(Cont’d)

34. 34
Bash keyboard shortcuts
Moving the cursor:
Ctrl + a Go to the beginning of the line (Home)
Ctrl + e Go to the End of the line (End)
Ctrl + p Previous command (Up arrow)
Ctrl + n Next command (Down arrow)
Editing:
Ctrl + L Clear the Screen, similar to the clear
command
Alt + d Delete the Word after the cursor.
Ctrl + d Delete character under the cursor (Del)
Ctrl + h Delete character before the cursor
(Backspace)
Ctrl + w Cut the Word before the cursor to the
clipboard.
Ctrl + k Cut the Line after the cursor to the
clipboard.
Ctrl + u Cut/delete the Line before the cursor to
the clipboard.
Alt + t Swap current word with previous
Ctrl + t Swap the last two characters before the
cursor (typo).
Esc + t Swap the last two words before the
cursor.
TAB Tab completion for file/directory names
History:
Ctrl + r Recall the last command including the
specified character(s)
searches the command history as you
type.
Equivalent to : vim ~/.bash_history.
Ctrl + o Execute the command found via
Ctrl+r or Ctrl+s
Process control:
Ctrl + C Interrupt/Kill whatever you are running
(SIGINT)
Ctrl + s Stop output to the screen (for long
running verbose commands)
Then use PgUp/PgDn for navigation
Ctrl + q Allow output to the screen (if previously
stopped using command above)
Ctrl + D Send an EOF marker, unless disabled by
an option, this will close the current shell
(EXIT)
Ctrl + Z Send the signal SIGTSTP to the current
task, which suspends it.
To return to it later enter fg 'process
name' (foreground).

35. List directory content
ls
Change current directory to another directory
cd /some/other/path
Create a new directory
mkdir –p /some/dir/to/create
Copy a file
cp /some/source/file /some/dest/file_or_folder
Rename / Move a file
mv /some/file1 /some/file2
mv /some/file1 /some/folder
Delete a file or folder
rm /some/file
rm –R /some/folder
Get information – manual pages – on a command
man rm
10/04/2018 35
Shell basics

36. Change owner of a file
chown ngscreener.ngscreener /path/to/some/file
Add execution flag to a file
chmod +x /path/to/some/file
chmod 600 /path/to/some/file
Show content of a file
cat /some/file
View content of a file
less /some/file
Edit a file
nano /some/file
(http://www.codexpedia.com/text-editor/nano-text-editor-command-cheatsheet/)
vi /some/file
(https://www.smashingmagazine.com/2010/05/vi-editor-linux-terminal-cheat-sheet-pdf/)
Execute command as root
sudo /etc/init.d/ng-screener restart
10/04/2018 36
Files, users and mods

37. Search for a file containing “log” in its name
find /some/folder/to/search –name ‘*log*’
Search for a file containing “ng-screener” and “log” in its name (stupid way)
find /some/folder/to/search –name ‘*log*’ | grep ‘ng-screener’
Search for a text file containing the text ‘ng-screener’
grep ‘ng-screener’ /path/to/some/file
List names of all files with extension ‘.log’ in folder containing text ‘ng-screener’
for i in `find /some/folder –name ‘*.log’`; do
if [[ `grep ‘ng-screener’ $i` != "" ]]; then
echo $i;
fi;
done
10/04/2018 37
Searching files and content

38. Chain commands with pipe – list lines of file containing string “test”
cat /some/file | grep ‘test’
Redirect standard output of a command to a file, overwriting content every time
cat ng-screener 1> /some/dest/file
(or simplier)
cat ng-screener > /some/dest/file
Redirect standard output of a command to a file, appending content
cat /some/file | grep ‘test’ >> /some/dest/file
Redirect standard error to a file – here, ignore standard errors
some_buggy_command 2>/dev/null
Redirect standard error to same file than standard output
cat /some/file | grep ‘test’ >> /some/dest/file 2>>&1
10/04/2018 38
Redirects and pipes

39. Execute some script in same process (and have its changes in VARs applied to current shell)
. /etc/profile
10/04/2018 39
Shell manipulation

40. Find out about existing partitions and filesystem configured on platform
cat /etc/fstab
Result:
# /etc/fstab: static file system information.
UUID=f149a12b-23bf-48a5-b158-9b7bf8e8084e / ext4 errors=remount-ro 0
1
/dev/mmcblk0p1 /data2 ext4 noatime,nodiratime,rw,noauto 0 3
/dev/sdb1 /data3 ext4 noauto 0 4
Find out about mounted partitions and filesystem configured on platform
cat /etc/mtab
Result:
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=10240k,nr_inodes=1496593,mode=755 0 0
…
/dev/sda3 /data ext4 rw,relatime,data=ordered 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
tmpfs /run/user/115 tmpfs rw,nosuid,nodev,relatime,mode=700,uid=115,gid=122 0 0
/dev/mmcblk0p1 /data2 ext4 rw,noatime,nodiratime,data=ordered 0 0
10/04/2018 40
Platform administration
Filesystem information

41. List existing mounted partitions and available space
df
Result:
Filesystem 1K-blocks Used Available Use% Mounted on
udev 10240 0 10240 0% /dev
tmpfs 2398560 220476 2178084 10% /run
/dev/sda1 67153528 59719620 3999620 94% /
tmpfs 5996396 321680 5674716 6% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 5996396 0 5996396 0% /sys/fs/cgroup
/dev/sda3 163335628 152036484 2979044 99% /data
tmpfs 1199280 16 1199264 1% /run/user/115
tmpfs 1199280 72 1199208 1% /run/user/1000
/dev/mmcblk0p1 122775580 68945348 47570520 60% /data2
Compute size of current folder and immediate sub-folders
du --max-depth 1
Result:
52 ./postgresql
12 ./fsck
144 ./samba
18392 ./ng-screener
13936 ./installer
…
88 ./cups
503312 .
41
Platform administration
Filesystem information (cont’d)

42. Most Important signals are as follows:
Signals Sent from Command Line
A stopped process (Ctrl + Z) can be continued :
In the Background : bg
In the Foreground : fg
Send a signal to a process known by its ID :
kill -15 14234
42
Linux Signals to processes
Signal Number Default Action
SIGHUP 1 Hangup (POSIX)
SIGQUIT 2 Terminal quit (POSIX)
SIGKILL 9 Kill(can't be caught or ignored) (POSIX)
SIGTERM 15 Termination (ANSI)
SIGCONT 18 Continue executing, if stopped (POSIX)
SIGTSTOP 20 Terminal stop signal (POSIX)
Shortcut Signal
Ctrl + Z SIGSTOP
Ctrl + C SIGKILL

43. List processes on the machine
ps -efl
Result:
10/04/2018 43
Process Manipulation

44. List all Java processes
ps -efl | grep java
Result:
4 S root 2648 1 0 84 0 - 22745 wait 16:04 ? 00:00:00 runuser -s
/bin/bash admin -c java -Dspring.profiles.active=prod -jar /usr/local/ng-
screener/BrowserWeb.jar
4 S admin 2656 2648 16 85 0 - 538916 futex 16:04 ? 00:00:55 java -
Dspring.profiles.active=prod -jar /usr/local/ng-screener/BrowserWeb.jar
4 S 103 7751 7743 99 85 0 - 690141 futex 16:12 ? 00:00:30
/usr/java/latest/bin/java -Dngdaemon.marker=/usr/local/ng-screener/daemon/ -server -
Xms1005m -Xmx1005m -Djava.rmi.server.hostname=10.194.6.107 -Drmi.registryPort=63000 -
Drmi.servicePort=63000 -Djava.security.manager -Djava.security.policy=/etc/ng-
screener/daemon/java.policy -classpath /etc/ng-screener/daemon:/usr/local/ng-
screener/lib/ngDaemonRuntime-5.1.1.jar:/usr/local/ng-screener/lib/ngDaemonCore-
5.1.1.jar:/usr/local/ng-screener/lib/ngDaemonCommon-5.1.1.jar:/usr/local/ng-
screener/lib/log4j-api-2.3.jar:/usr/local/ng-screener/lib/spring-core-
4.1.7.RELEASE.jar:/usr/local/ng-screener/lib/util-logging-5.1.1.jar:/usr/local/ng-
screener/lib/log4j-core-2.3.jar:/usr/local/ng-screener/lib/log4j-jcl-
2.3.jar:/usr/local/ng-screener/lib/log4j-slf4j-impl-2.3.jar:/usr/local/ng-
screener/lib/log4j-1.2-api-2.3.jar:/usr/local/ng-screener/lib/spring-context-
4.1.7.RELEASE.jar:…:/usr/local/ng-screener/lib/json-20090211.jar:/usr/local/ng-
screener/lib/slf4j-api-1.7.6.jar:/usr/local/ng-screener/lib/htt
4 R root 8141 6686 0 78 0 - 15289 - 16:12 pts/0 00:00:00 grep java
4 R root 8145 8144 0 82 0 - 1026 - 16:12 pts/0 00:00:00 pidof
/usr/java/latest/bin/java
As one can see, result is cut and there is not much we can do about it.
We’ll see a solution for this later 44
Shell manipulation

45. Get information about memory usage and most CPU consuming programs
top
Result:
45
The Top command

46. Let’s imagine the following output of a top command:
top - 16:39:49 up 5 min, 1 user, load average: 0.32, 0.79, 0.45
Tasks: 202 total, 1 running, 201 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 4058704k total, 4042460k used, 16244k free, 9468k buffers
Swap: 8193140k total, 2476k used, 8190664k free, 2211036k cached
First glance :
Only 16Mb of free memory
So 16 Mb / 4’096 Mb * 100 =
0.4 % of free memory !
We have a big issue here
But looking closer:
Remember the buffer cache we discussed earlier ?
The free memory here is actually :
Free mem + cached = 16 Mb + 2’211 Mb = 2’226 Mb
So 2’226 Mb / 4’096 Mb * 100 =
54 % of free memory !
10/04/2018 46
The Top command
Interpreting the memory informations

47. In this regards, the result of the free command is easier to interpret:
Listing of free memory on the platform with the free command
free
total used free shared buffers cached
Mem: 4058704 4041184 17520 0 6916 150460
-/+ buffers/cache: 1883808 2174896
Swap: 8193140 25100 8168040
10/04/2018 47
The Free command

48. In order to drop the disk cache, we can use the special file /proc/sys/vm/drop_caches. By writing 3 to
it, we can clear most of the disk cache:
echo 3 | sudo tee /proc/sys/vm/drop_caches
Flush the filesystem buffer, forcing the system to write all pending changes on the disk
sync
10/04/2018 48
Manipulating the buffer cache

49. Regarding IO monitoring, the command iostat appears to be very handy.
It is used for monitoring system input/output device loading by observing the time the devices are
active in relation to their average transfer rates. The iostat command generates reports that can be
used to change system configuration to better balance the input/output load between physical disks.
The first report generated by the iostat command provides statistics concerning the time since the
system was booted.
Each subsequent report covers the time since the previous report.
Example :
iostat 10
Result:
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 20.50 562.22 145.44 820204 212176
sda1 0.03 1.14 0.04 1666 56
sda2 0.42 15.77 49.93 23000 72848
sdb 1.41 15.17 14.98 22124 21856
sdb1 1.39 14.59 14.98 21284 21856
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 13.30 0.00 176.00 0 1760
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sdb 0.90 0.00 16.00 0 160
sdb1 0.90 0.00 16.00 0 160
Monitoring IO capabilities

50. Sometimes it is important to understand what file is used by which process.
For instance to find out about the PollingStatus files used by the PollingSystem
Or to understand what file is used as logging file by what process, etc.
This can be achieved pretty easily with the command lsof
The command lsof is located in /usr/sbin which may not be in your path.
Hence the need to prefix the call with the full path of the command
The command lsof lists the files on which processes have an “open” handle at the time of the call
Example, to find out about the handles opened by process 30260 (PostgreSQL in this case)
/usr/sbin/lsof | grep 30260
Result:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postmaste 30260 postgres cwd DIR 8,17 4096 3293409 /data/pgdata
postmaste 30260 postgres rtd DIR 8,3 4096 2 /
postmaste 30260 postgres txt REG 8,3 5157208 1019260 /usr/pgsql-9.3/bin/postgres
postmaste 30260 postgres mem REG 8,3 142488 1769648 /lib64/ld-2.5.so
postmaste 30260 postgres mem REG 8,3 217592 631806 /usr/lib64/libxslt.so.1.1.17
...
postmaste 30260 postgres mem REG 8,3 7832 1769671 /lib64/libcom_err.so.2.1
postmaste 30260 postgres DEL REG 0,9 6538 /dev/zero
postmaste 30260 postgres DEL REG 0,9 2424838 /SYSV0052e2c1
postmaste 30260 postgres 0r CHR 1,3 0t0 1384 /dev/null
postmaste 30260 postgres 1w FIFO 0,6 0t0 6555 pipe
…
postmaste 30260 postgres 5r FIFO 0,6 0t0 6554 pipe
postmaste 30260 postgres 6u REG 8,17 8192 3359304 /data/pgdata/base/31194/12616
postmaste 30260 postgres 7u REG 8,17 114688 3359355 /data/pgdata/base/31194/12576
postmaste 30260 postgres 8u IPv4 6557 0t0 UDP NG-SCREENER:56798->NG-SCREENER:56798
postmaste 30260 postgres 9u IPv4 81632 0t0 TCP NG-SCREENER:postgres->NG-SCREENER:44762 (ESTABLISHED)
postmaste 30260 postgres 10u REG 8,17 57344 3359220 /data/pgdata/base/31194/12578
postmaste 30260 postgres 11u REG 8,17 98304 3359329 /data/pgdata/base/31194/12579
postmaste 30260 postgres 12u REG 8,17 8192 3359160 /data/pgdata/base/31194/12673
…
50
List filesystem handles per process

51. Map the processes (1/2)
Use the following command :
ps -eaxo pid,ppid,euid,class,rtprio,pri,etime,cputime,
pcpu,rss:8,size:8,pmem,nlwp,state,sgi_p,args
| tee /tmp/ps_list | less
This commands returns the most usefull troubleshooting informations for all processes of the system :
pid : a number representing the process ID
ppid : the process ID of the parent process
euid : the user ID of the process owner
class : scheduling class of the process (see "man ps“
rtprio : realtime priority
pri : priority of the process. Higher number means lower priority.
etime : elapsed time since the process was started, in the form [[DD-]hh:]mm:ss.
cputime : cumulative CPU time, "[DD-]hh:mm:ss" format.
pcpu : cpu utilization of the process in "##.#" format. Currently, it is the CPU time used divided by the time the process has
been running (cputime/realtime ratio), expressed as a percentage. It will not add up to 100% unless you are lucky.
rss:8 : resident set size, the non-swapped physical memory that a task has used (in kiloBytes).
size:8 : approximate amount of swap space that would be required if the process were to dirty all writable pages and then
be swapped out. This number is very rough!
pmem : ratio of the process's resident set size to the physical memory on the machine, expressed as a percentage.
nlwp : number of threads (lwps) in the process (thread count)
state : minimal state display (one character). (see "man ps")
sgi_p : Shows the CPU in which the process is currently being executed. Displays "*" if the process is not currently running
or runnable.
args : command with all its arguments as a string. (See "man ps")
51
Understanding the processes
layout (1/2)

52. Map the processes (2/2)
How to use this command ?
In case of process overusing CPUs (infinite loop, unexpected heavy processing, etc.) sgi_p : tells if the process is
currently being executed on a CPU. Seeing consistently a number there indicates the process is using 100% of a
CPU. This can be confirmed by pcpu which gives the estimated percentage of CPU used by a process. The
number of threads given by nlwp is also interesting.
Finally, even maybe more important than all the above is cputime which gives the cumulated CPU time since the
process has been started. In case this value is close to etime then we may have a problem with this process.
In regards to memory consumption analysis, rss:8 gives the size of the RAM (physical memory) actually used by
a process and size:8 gives the total size of memory = RAM + SWAP used by the process. Finally, pmem gives
the percentage of the RAM (physical memory) taken by the process (confirms previous informations)
args gives the complete command line used to start the process. Appending the result to a file and seeing it with
less is a good idea to be able to read it completely.
On the other hand, avoiding less enables to have the columns well aligned in the terminal.
52
Understanding the processes
layout (2/2)

53. 5310/04/2018
Java troubleshooting

54. As discussed previously, long commands are cut by the command ps
We can use the command jps provided by the JDK to discover Java processes and their arguments:
jps –mlv
Result:
14176 sun.tools.jps.Jps -mlv -Dapplication.home=/usr/java/jdk1.8.0_66 -Xms8m
2956 org.elasticsearch.bootstrap.Elasticsearch start -p /usr/local/ng-
screener/NgStorage//elasticsearch.pid -Des.path.home=/usr/local/ng-screener/NgStorage/
-Des.path.logs=/var/log/ng-screener/ -Des.path.data=/storage/ngstorage -
Des.path.work=/tmp/elasticsearch -Xms256m -Xmx1g -Djava.awt.headless=true -
XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -
XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -
XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -
Des.path.home=/usr/local/ng-screener/NgStorage
3037 /usr/local/ng-screener/BrowserWeb.jar -Dspring.profiles.active=prod
3844 com.netguardians.ngscreener.daemon.NgDaemon -Dngdaemon.marker=/usr/local/ng-
screener/daemon/ -Xms1005m -Xmx1005m -Djava.rmi.server.hostname=10.194.6.107 -
Drmi.registryPort=63000 -Drmi.servicePort=63000 -Djava.security.manager -
Djava.security.policy=/etc/ng-screener/daemon/java.policy
10/04/2018 54
Java process discovery

55. Once we know about a process PID, we can dump its threads using the command jstack
This command will dump each and every thread of the JVM at the precise state in which it is at the
moment of the call of the command.
Note 1 : this command has a very long output !
Note 2 : it most often require developer-level understanding to interpret it
However : using this command to know which thread is active gives invaluable indications on what is
going on in the JVM !
One would typically run it several times after short pauses to find our which/if a thread is really busy.
Example:
jstack 3844
Result:
55
Java thread discovery

56. 56
Java thread discovery
(cont’d)

57. 57
Java thread discovery
(cont’d)

58. All the former examples are threads doing nothing, they’re waiting !
This is a portion of a jstack result (old format) we got in production when a thread was stuck on an
Oracle Database which was not answering :
Thread 5912: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0
(Compiled frame; information may be imprecise)
- java.net.SocketInputStream.socketRead(java.io.FileDescriptor, byte[], int, int, int) @bci=8, line=116 (Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=170 (Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=141 (Compiled frame)
- oracle.net.ns.Packet.receive() @bci=31, line=240 (Compiled frame)
- oracle.net.ns.DataPacket.receive() @bci=1, line=92 (Compiled frame)
- oracle.net.ns.NetInputStream.getNextPacket() @bci=48, line=172 (Compiled frame)
- oracle.net.ns.NetInputStream.read(byte[], int, int) @bci=33, line=117 (Compiled frame)
- oracle.net.ns.NetInputStream.read(byte[]) @bci=5, line=92 (Compiled frame)
- oracle.net.ns.NetInputStream.read() @bci=5, line=77 (Compiled frame)
- oracle.jdbc.driver.T4CMAREngine.unmarshalUB1() @bci=11, line=1034 (Compiled frame)
- oracle.jdbc.driver.T4CMAREngine.unmarshalSB1() @bci=1, line=1010 (Compiled frame)
- oracle.jdbc.driver.T4CTTIrxd.setNumberOfColumns(int) @bci=11, line=95 (Compiled frame)
- oracle.jdbc.driver.T4C8Oall.receive() @bci=54, line=580 (Compiled frame)
- oracle.jdbc.driver.T4CPreparedStatement.doOall8(boolean, boolean, boolean, boolean) @bci=655, line=194
(Compiled frame)
- oracle.jdbc.driver.T4CPreparedStatement.fetch() @bci=63, line=1017 (Compiled frame)
- oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(boolean) @bci=149, line=314 (Compiled frame)
- oracle.jdbc.driver.OracleResultSetImpl.next() @bci=205, line=228 (Compiled frame)
- com.netguardians.ngscreener.daemon.modules.polling.connectors.t24.T24Connector.pollLogs() @bci=159, line=262
(Compiled frame)
- com.netguardians.ngscreener.daemon.modules.polling.connectors.t24.T24Connector.pollLogs() @bci=1, line=24
(Interpreted frame)
- com.netguardians.ngscreener.daemon.modules.polling.spi.AbstractPollingConnector.run() @bci=27, line=209
(Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=221, line=555 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)
58
Java thread discovery
(cont’d)

59. This is an example of a thread that’s running and going just fine (old format)
Thread 5962: (state = IN_JAVA)
- org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken() @bci=435, line=1305
(Compiled frame; information may be imprecise)
- org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken() @bci=13, line=178 (Compiled frame)
- org.apache.lucene.analysis.standard.StandardFilter.incrementToken() @bci=17, line=49 (Compiled frame)
- org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken() @bci=4, line=54 (Compiled frame)
- org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken() @bci=16, line=82 (Compiled frame)
- com.netguardians.ngscreener.daemon.tools…..SyslogStorageSP.isSyslogNgLineMatchLoadingConditions
(java.lang.String, com.netguardians.ngscreener.daemon.common.dto.LoadingConditionsDTO, boolean)
@bci=496, line=403 (Compiled frame)
- com.netguardians.ngscreener.daemon.tools.syslogstorage.LogInputStream.readSequential() @bci=52, line=117
(Compiled frame)
- com.netguardians.ngscreener.daemon.tools.syslogstorage.LogInputStream.readLog() @bci=57, line=86 (Compiled frame)
- com.netguardians.ngscreener.daemon.modules.forensic.job.model.LogFileReader.get() @bci=56, line=66 (Compiled frame)
- com.netguardians.ngscreener.daemon.modules.forensic.job.Job.get() @bci=4, line=94 (Compiled frame)
- com.netguardians.ngscreener.daemon.modules.forensic.job.framework.JobThreadPool$Worker.run() @bci=80, line=110
(Compiled frame)
10/04/2018 59
Java thread discovery
(cont’d)

60. The command jmap is used to export all the objects from the JVM (a memory dump) in a file on the
filesystem.
Example:
jmap -dump:file=/tmp/dump 3844
Result:
Dumping heap to /tmp/dump ...
Heap dump file created
That file can later be sent to the NG R&D Team for proper investigation in case of an unsolveable
production issue
10/04/2018 60
Dumping memory of the JVM