Brad Hubbard - Red Hat Ceph Day Melbourne - November 5, 2015

1. TROUBLESHOOTING CEPH
Brad Hubbard
Senior Software Maintenance Engineer
05-11-2015

2. What sort of trouble?
Identify your problem domain
Ceph is a mature, resilient and robust piece of software but, when things do go
wrong, empower yourself to identify these specific areas and analyse them using
common Linux, and Ceph-specific, tooling.
●
Performance
●
“Hang”
●
Crash
●
Unexpected or undesirable behaviour

3. Performance
Establish a baseline and re-test regularly
●
rados bench
●
ceph tell osd.N bench
●
fio – rbd ioengine
●
fio – libaio ioengine
●
pblio - https://github.com/pblcache/pblcache/wiki/Pblio
●
netperf – test all network segments
●
dd
●
pcp, sysstat, collectl, insert favourite tool here...
●
The Ceph Benchmarking Tool - https://github.com/ceph/cbt
●
Be mindful of the cache and its effects

4. Performance
Specifically, poor performance
Zero in on the problem area by identifying if it is specific to a particular host, or hosts,
or if a particular sub-domain is implicated.
●
HEALTH_OK ?
●
Re-use the tools mentioned in the previous slide as well as host specific tools
●
ss, netstat and friends
●
tcpdump
●
iostat
●
top
●
pcp, sar, collectl
●
free, vmstat
●
Increase ceph logging verbosity
●
$ gawk '/ERR/||/WRN/' /var/log/ceph/*log

5. Performance
Slow requests
When Ceph detects a request that is too slow (tunable) it will issue a warning.
{date} {osd.num} [WRN] slow request 30.005692 seconds old, received at {date-
time}: osd_op(client.4240.0:8 benchmark_data_ceph-1_39426_object7 [write
0~4194304] 0.69848840) v4 currently waiting for subops from [610]
●
HEALTH_OK ?
●
Check performance statistics on implicated hosts
●
Turn up debugging on the implicated OSDs
•
# ceph tell osd.N injectargs '--debug_osd 20 --debug_ms 1'
●
Gather information about slow ops
•
# ceph --admin-daemon /var/run/ceph/ceph-osd.N.asok dump_historic_ops
•
# ceph --admin-daemon /var/run/ceph/ceph-osd.N.asok perf dump

6. Hang
Is it really a hang?
Sometimes situations described as a “hang” turn out to be something different such
as code stuck in a tight loop, a dead-lock, firewall problems, etc.
● Use strace to check if the process is still making progress (may miss hung threads)
●
Check for a high load average and/or high %iowait on the CPUs
●
Use ps to check for ceph processes in d-state (Uninterruptible sleep)
● Use ps to find the ceph threads that are sleeping and what function they are
sleeping in
•
# ps axHo stat,tid,ppid,comm,wchan
●
Check syslog and dmesg for “hung_task_timeout” warnings
●
Use gstack or gcore to figure out where we are in the ceph code and what
subsystems in the kernel we are exercising

7. Hang
Is it really a hang?
Note that if everything points to uninterruptible threads in kernel space this is a kernel
problem but it obviously still has the potential to severely degrade ceph performance
and needs to be identified and fixed.
●
Sysrq to dump kernel thread stacks. Dumps to syslog, search for “ D “
•
# echo 1 > /proc/sys/kernel/sysrq
•
# echo 't' > /proc/sysrq-trigger
•
# sleep 20
•
# echo 't' > /proc/sysrq-trigger
•
# echo 0 > /proc/sys/kernel/sysrq
xfssyncd/dm-2 D 0000000000000011 0 3207 2 0x00000080
●
sysrq data may implicate a certain subsystem or help to identify a known issue or
confirm suspicions
●
May require a vmcore be collected and analysed

8. Hang
Is it really a hang?
What at first appears to be a hang may in fact be a thread, or threads, caught in a
tight loop due to some logic condition and failing to make progress. To the user that
process seems “hung” but it is actually running. We need to identify where the
process is spending the bulk of it's time.
●
Look for high CPU usage of Ceph processes
●
Check strace and/or ltrace output for hints at what the process may be doing
●
Employ the “Poor Man's Profiler” technique - http://poormansprofiler.org/
•
# for x in `seq 1 5`; do for pid in `pidof ceph-mon ceph-osd`; do gstack $pid;
echo; done; done > /tmp/ceph-stacks
•
This can potentially generate a lot of data so you may want to only target a single
process, the one(s) with high CPU utilisation
●
Visually inspect the relevant source code to work out why it might not make
progress
●
More advanced techniques such as scripting gdb, systemtap probes

9. Hang
Is it really a hang?
Dead-lock or live-lock.
●
Gcore and/or gstack
●
Visually inspect relevant source code
●
Might need some help with this one

10. Crash
Where did ceph go?
If ceph crashes it will attempt to log details of the crash. Code in handle_fatal_signal()
and __ceph_assert_fail() will try to dump the stack as well as relevant information and
a debug log of recent events. Search the logs for “objdump”.
common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")
ceph version 0.80.8-84-gb5a67f0 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)
1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)
+0x2a9) [0x9acc49]
2: (ceph::HeartbeatMap::is_healthy()+0xb6) [0x9ad4b6]
3: (OSD::_is_healthy()+0x21) [0x5fde61]
4: (OSD::tick()+0x498) [0x64d978]
...
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.

11. Crash
asserts
The ceph code base includes thousands of asserts. Assert is a system call that aborts
the program if if the assertion evaluates to false.
●
Conditions that are considered fatal
●
Memory corruption
●
Result of on-disk corruption
●
Intentional aborts
75 was = h->suicide_timeout.read();
76 if (was && was < now) {
77 ldout(m_cct, 1) << who << " '" << h->name << "'"
78 << " had suicide timed out after " << h->suicide_grace << dendl;
79 assert(0 == "hit suicide timeout");
80 }

12. Crash
Fatal signals
Indicate a fatal error such as a segmentation fault, bus error or abort. Search for
“objdump” or “*** Caught signal”
●
Indicative of a programming error
●
Usually a memory accounting/access error
●
Check for existing bugs with the same signature or open a new tracker or Bugzilla

13. Crash
Example
0> 2015-09-24 04:14:49.345105 7fea04f79700 -1 *** Caught signal (Aborted) **
in thread 7fea04f79700
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: /usr/bin/ceph-osd() [0x9f63f2]
2: (()+0xf130) [0x7fea14462130]
3: (gsignal()+0x37) [0x7fea12e7c5d7]
4: (abort()+0x148) [0x7fea12e7dcc8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fea137809b5]
6: (()+0x5e926) [0x7fea1377e926]
7: (()+0x5e953) [0x7fea1377e953]
8: (()+0x5eb73) [0x7fea1377eb73]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xb697b7]
10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xab1a35]
...
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

14. Crash
Extracting more information
Crashes can be tricky to diagnose completely but, if you are up for the challenge or
would just like to gather more information, eu-addr2line, gdb or objdump may provide
further insight.
# eu-addr2line -e /usr/bin/ceph-osd 0xb697b7
include/buffer.h:224
# objdump -rdS /usr/bin/ceph-osd
# gdb `which ceph-osd`
(gdb) disass /m 0xb697b7
Dump of assembler code for function ceph::buffer::list::iterator::copy(unsigned int,
char*):
...

15. Unexpected or undesirable behaviour
That doesn't seem right?
Sometimes ceph may not do what you expect or want
●
Identify the expected or desirable behaviour
●
Figure out if this is by design or the result of a corner case or error
●
What behaviour do you see (when I do X, I see Y)
●
Timestamp an instance of the error/behaviour
●
Increase debugging and trace the transaction through the logs
●
If this is Openstack behaviour trace it via Nova, Glance, Cinder, rbd logs
●
If this is Rados gateway behaviour trace the httpd logs and match these with the
rgw and Ceph logs
●
Start at the user end and work back towards ceph
●
Timestamps help a lot!

16. Debug logging
OSD
●
debug ms = 1
●
debug osd = 20
●
debug objecter = 20
●
debug monc = 20
●
debug journal = 20
●
debug filestore = 20
●
debug newstore = 30
●
debug objclass = 20

17. Debug logging
MON
●
debug mon = 20
●
debug ms = 1
●
debug paxos = 20
●
debug auth = 20

18. Debug logging
RADOS Gateway
●
debug rgw = 20
●
debug ms = 1

19. Debug logging
MDS
●
debug ms = 1
●
debug mds = 20
●
debug auth = 20
●
debug monc = 20
●
mds debug scatterstat = true
●
mds verify scatter = true
●
mds log max segments = 2

20. Debug logging
Client
[client] # Section, can also be global since it is inherited
debug ms = 1
debug rbd = 20
debug objectcacher = 20
debug objecter = 20
log file = /var/log/ceph/rbd.log
# touch /var/log/ceph/rbd.log
# chmod 777 /var/log/ceph/rbd.log

21. Debug logging
Openstack
Turn up logging verbosity for whichever is relevant Nova, Glance, Cinder, rbd or all of
the above
●
Trace error/behaviour down through the logs from high level (Nova) to low level
(rbd and the ceph cluster)
●
Try running relevant commands from a lower level
●
Make sure it isn't an Openstack problem

22. Debug logging
Linux kernel (krbd) client
The kernel RBD client logs to syslog and/or dmesg

23. Debug logging
Without restart
Turn debug logging on
●
ceph tell osd.* injectargs '--debug_osd 20 --debug_ms 1'
Turn debug logging off
●
ceph tell osd.* injectargs '--debug_osd 0/5 --debug_ms 0/5'

24. Source code
Use the source Luke!
Upstream source
# ceph -v
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
●
git clone https://github.com/ceph/ceph.git
●
git checkout e4bfad3a3c51054df7e537a724c8d0bf9be972ff
●
git checkout v0.94.1

25. Source code
Use the source Luke!
Downstream source
●
yumdownloader –archlist=src --enablerepo=rhel-7-server-rhceph-1.3-*source-rpms ceph
●
rpm -ivh ceph-0.94.1-19.el7cp.src.rpm
●
rpmbuild -bp --nodeps rpmbuild/SPECS/ceph.spec
●
cd rpmbuild/BUILD/ceph-0.94.1/
●
Ubuntu equivalent commands
●
Use your favourite editor (yes, of course it's “vi”) to browse the source files

26. Resources
Additional sources of help
If all else fails (or even as a first resort) seek help.
●
Email:
•
ceph-users@ceph.com
●
IRC:
•
irc.oftc.net #ceph
●
Known issues
•
http://tracker.ceph.com/
●
https://bugzilla.redhat.com/
●
Documentation
•
http://docs.ceph.com/docs/master/
●
Red Hat support
•
https://access.redhat.com
●
https://access.redhat.com/support

27. THANK YOU
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews