Crash_Report_Mechanism_In_Tizen

Crash Report Mechanism in Tizen
Yu Yucai, 2013/05/31

2
Agenda
• Crash Client (Tizen 2.1.x)
• Samsung’s crash-work-sdk
• Intel’s corewatcher
• Crash Server
• CrashDB Web
• Crash reason location algrithm

3
Agenda
• Crash Client
• Intel’s corewatcher(3rd Gen)
• Crash Server
• CrashDB Web

4
Crash Client - Samsung’s crash-work-sdk
• Mechanism
• Crash process flow
• Crash report file: *.cs file

5
• Mechanism: force preload libsys-assert.so to overload crash signal
handlers.
sh-4.1# cat /etc/ld.so.preload
/usr/lib/libsys-assert.so
sh-4.1# ldd `which top`
linux-gate.so.1 => (0xb77fd000)
/usr/lib/libsys-assert.so (0xb77f5000)
libproc-3.2.8.so => /lib/libproc-3.2.8.so (0xb77c1000)
libncurses.so.5 => /usr/lib/libncurses.so.5 (0xb7781000)
libc.so.6 => /lib/libc.so.6 (0xb7625000)
libdl.so.2 => /lib/libdl.so.2 (0xb7621000)
/lib/ld-linux.so.2 (0xb77fe000)

6
Stage 1: in sys-assert.c (libsys-assert.so)
-> int sig_to_handle[] = { SIGILL, SIGABRT, SIGBUS, SIGFPE, SIGSEGV, };
-> sighandler notify system_server via /opt/share/crash/curbs.log pipeline.
Stage 2: in ss_bs.c (system-server)
-> ecore_file_monitor_add(CRASH_NOTI_PATH,(void *) __crash_file_cb, NULL);
-> __crash_file_cb:
-> launch_crash_worker()
......
launch /usr/bin/crash-worker to generate cs file
launch /usr/apps/org.tizen.crash-popup/bin/crash-popup to popup crash (only 1st)

7
Breakpoint 1, launch_app_with_nice (file=0xb46017b0 "/usr/bin/crash-worker", argv=0xbfe52624, pid=0x0, _nice=0)
at /usr/src/debug/system-server-0.1.65/ss_launch.c:140
140 {
(gdb) bt
#0 launch_app_with_nice (file=0xb46017b0 "/usr/bin/crash-worker",
argv=0xbfe52624, pid=0x0, _nice=0)
#1 0x0804d3c9 in launch_app_cmd_with_nice (
cmdline=0xb4601758 "/usr/bin/crash-worker S top 391655492 913 top",
_nice=0) at /usr/src/debug/system-server-0.1.65/ss_launch.c:196
#2 0x0804d744 in ss_launch_evenif_exist (
execpath=0x8060e71 "/usr/bin/crash-worker",
arg=0xbfe53d16 "S top 391655492 913 top")
#3 0x08058b73 in launch_crash_worker (
filename=0xbfe5823c "/opt/share/crash/curbs.log", popup_on=1)
at /usr/src/debug/system-server-0.1.65/ss_bs.c:327
#4 0x08058d47 in __crash_file_cb (data=0x0, em=0x8dabb10,
event=ECORE_FILE_EVENT_MODIFIED,
path=0xbfe5823c "/opt/share/crash/curbs.log")
at /usr/src/debug/system-server-0.1.65/ss_bs.c:374
#5 0xb782f345 in _ecore_file_monitor_inotify_handler ()
from /usr/lib/libecore_file.so.1
#6 0xb7840e5c in _ecore_main_loop_iterate_internal ()
from /usr/lib/libecore.so.1
#7 0xb784141f in ecore_main_loop_begin () from /usr/lib/libecore.so.1
#8 0x0804bd95 in system_main (argc=1, argv=0xbfe59404)
---Type <return> to continue, or q <return> to quit---
at /usr/src/debug/system-server-0.1.65/ss_main.c:102
#9 0x0804bdf1 in elm_main (argc=1, argv=0xbfe59404)
#10 0x0804be4e in main (argc=1, argv=0xbfe59404)

8
Stage 3: crash-worker to generate cs file
sh-4.1# pwd
/opt/usr/share/crash/report
sh-4.1# ls -tlr
total 72
-rwxrw-rw- 1 root app 22248 1982-05-02 18:28 top_19820502182857.cs

9
• What’s *.cs file.
sh-4.1# cat top_19820502184511.cs
S/W Version Information
Model: handset-blackbay
Tizen-Version: 2.1.0
Build-Number: Tizen_handset-blackbay-tizen-2.1-
mobile_20130430.1
Build-Date: 2013.05.01 03:24:28
Crash Information
PID: 1194
Date: 1982-05-02 18:45:11(GMT+0900)
Executable File Path: top
This process is multi-thread process
pid=1194 tid=1194
Signal: 11
(SIGSEGV)
si_code: 0
signal sent by kill (sent by pid 1195, uid 0)
Register Information
gs = 0x00000033, fs = 0x00000000
es = 0x0000007b, ds = 0x0000007b
edi = 0x08057040, esi = 0x00000000
ebp = 0xbfe13b68, esp = 0xbfe13040
eax = 0xfffffffc, ebx = 0x00000001
ecx = 0xbfe1332c, edx = 0x00000000
eip = 0xb7896424
Memory Information
MemTotal: 995664 KB
MemFree: 457540 KB
Buffers: 14768 KB
Cached: 233204 KB
VmPeak: 2748 KB
VmSize: 2748 KB
VmLck: 0 KB
VmHWM: 1336 KB
VmRSS: 1336 KB
VmData: 492 KB
VmStk: 136 KB

10
• What’s *.cs file.
Maps Information
08048000 08056000 r-xp /usr/bin/top
b7683000 b768d000 r-xp /lib/libnss_files-2.13.so
b768f000 b7698000 r-xp /lib/libnss_nis-2.13.so
b769a000 b76ad000 r-xp /lib/libnsl-2.13.so
b76b1000 b76b7000 r-xp /lib/libnss_compat-2.13.so
b76ba000 b76bc000 r-xp /lib/libdl-2.13.so
b76be000 b7814000 r-xp /lib/libc-2.13.so
b781a000 b7857000 r-xp /usr/lib/libncurses.so.5.7
b785a000 b7868000 r-xp /lib/libproc-3.2.8.so
b788e000 b7892000 r-xp /usr/lib/libsys-assert.so
b7896000 b7897000 r-xp [vdso]
b7897000 b78b3000 r-xp /lib/ld-2.13.so
End of Maps Information
Callstack Information (PID:1194)
Call Stack Count: 1
can't open 0: (0x8049a81) [top] + 0x1a81
End of Call Stack
Latest Debug Message Information
--------- beginning of /dev/log_main
05-02 16:31:42.769 E/Tizen::App( 616): void
Tizen::App::_ConditionManagerStub::OnInstallComplete(const
Tizen::App::AppId&)(133) > failed to
GetAppLaunchConditionListN(org.tizen.taskmgr)
…
omPackageManager(const Tizen::Base::String&, const
Tizen::Base::String&, const Tizen::Base::String&)(493) > Cannot
acquire feature list.
05-02 16:31:43.819 E/Tizen::App( 616): void Tizen::App::_
End of latest debug message

11
Agenda
• Crash Client
• Crash Server
• CrashDB Web

12
Crash Client - Intel’s corewatcher
• Mechanism.
• How to upload crashes to server.
• Crash report file: /var/lib/corewatcher/processed/*.txt.

13
• Mechanism.
• Corewatcher as daemon
• Listen to /var/lib/corewatcher/
• When crash comes, invoke gdb to analysis
• Upload crashes to CrashDB server
• Environment about corewatcher
• /proc/sys/kernel/core_pattern=/var/lib/corewatcher/core_%e_%t
• core_uses_pid=1
bt full
info shared

14
• CrashDB server: https://tz.otcshare.org/crashdb/
• How to upload crash to server
• WWLAN(3G/2G)
• WiFi/SED
• crash_submit: http://otcqa.sh.intel.com/wiki/Crash_Submit
Even though tz.otcshare.org has security restriction(403 forbidden outside of
Intel), crash submit is allowed.

15
• Crash report path: /var/lib/corewatcher/processed/*.txt
• Crash report content(without debug info):
cmdline: /usr/bin/mate-calc
version: 2.1.0
backtrace: |
#0 0x00007fd494c2db41 in g_logv () from /usr/lib64/libglib-2.0.so.0
#1 0x00007fd494c2dcfd in g_log () from /usr/lib64/libglib-2.0.so.0
#2 0x00007fd4959a10ee in g_settings_set_property () from /usr/lib64/libgio-2.0.so.0
#3 0x00007fd4956ae098 in g_object_constructor () from /usr/lib64/libgobject-2.0.so.0
#4 0x00007fd4956af562 in g_object_newv () from /usr/lib64/libgobject-2.0.so.0

16
maps: |
0x00007fd4971f5f00 0x00007fd49745ecf0 Yes (*) /usr/lib64/libgtk-x11-2.0.so.0
0x00007fd496ef7420 0x00007fd496f4ccb8 Yes (*) /usr/lib64/libgdk-x11-2.0.so.0
0x00007fd496cc0030 0x00007fd496ccc290 Yes (*) /usr/lib64/libatk-1.0.so.0
0x00007fd496aa8490 0x00007fd496ab05bc Yes (*) /usr/lib64/libpangoft2-1.0.so.0
0x00007fd496898f20 0x00007fd49689dd14 Yes (*) /usr/lib64/libpangocairo-1.0.so.0
0x00007fd49667aa80 0x00007fd49668c018 Yes (*) /usr/lib64/libgdk_pixbuf-2.0.so.0
0x00007fd496377cc0 0x00007fd49643941c Yes (*) /usr/lib64/libcairo.so.2
0x00007fd496127680 0x00007fd496146fe0 Yes (*) /usr/lib64/libpango-1.0.so.0
0x00007fd495e95a90 0x00007fd495ef1fa4 Yes (*) /usr/lib64/libfreetype.so.6
(*): Shared library is missing debugging information.

17
• Crash report content(with debug info):
backtrace: |
#0 g_logv (log_domain=0x7f3d296fbfe1 "GLib-GIO", log_level=G_LOG_LEVEL_ERROR,
format=<optimized out>, args=args@entry=0x7fff7ea01e58) at gmessages.h:101
#1 0x00007f3d28941cfd in g_log (log_domain=log_domain@entry=0x7f3d296fbfe1 "GLib-GIO",
log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=format@entry=0x7f3d29717dc0 "Settings schema '%s'
is not installedn") at gmessages.c:1003
#2 0x00007f3d296b50ee in g_settings_set_property (object=<optimized out>, prop_id=<optimized
out>, value=<optimized out>, pspec=<optimized out>) at gsettings.c:492
#3 0x00007f3d293c2098 in object_set_property (nqueue=0xf90e60, value=0xf96408,
pspec=0xf95ca0, object=0xf95540) at gobject.c:1357

18
maps: |
From To Syms Read Shared Object Library
No linux-vdso.so.1
0x00007f3d2af09f00 0x00007f3d2b172cf0 Yes /usr/lib64/libgtk-x11-2.0.so.0
0x00007f3d2ac0b420 0x00007f3d2ac60cb8 Yes /usr/lib64/libgdk-x11-2.0.so.0
0x00007f3d2a9d4030 0x00007f3d2a9e0290 Yes /usr/lib64/libatk-1.0.so.0
0x00007f3d2a7bc490 0x00007f3d2a7c45bc Yes /usr/lib64/libpangoft2-1.0.so.0
0x00007f3d2a5acf20 0x00007f3d2a5b1d14 Yes /usr/lib64/libpangocairo-1.0.so.0
0x00007f3d2a38ea80 0x00007f3d2a3a0018 Yes /usr/lib64/libgdk_pixbuf-2.0.so.0
(*): Shared library is missing debugging information.

19
• Crash report path: /var/lib/corewatcher/processed/*.txt
cmdline: /usr/bin/mate-calc
version: 2.1.0
backtrace: |
#1 0x00007fd494c2dcfd in g_log () from /usr/lib64/libglib-2.0.so.0
#2 0x00007fd4959a10ee in g_settings_set_property () from /usr/lib64/libgio-2.0.so.0
#3 0x00007fd4956ae098 in g_object_constructor () from /usr/lib64/libgobject-2.0.so.0
#4 0x00007fd4956af562 in g_object_newv () from /usr/lib64/libgobject-2.0.so.0
Quality: 66

20
backtrace: |
#1 0x00007f3d28941cfd in g_log (log_domain=log_domain@entry=0x7f3d296fbfe1 "GLib-GIO",
log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=format@entry=0x7f3d29717dc0 "Settings schema '%s'
is not installedn") at gmessages.c:1003
#2 0x00007f3d296b50ee in g_settings_set_property (object=<optimized out>, prop_id=<optimized
out>, value=<optimized out>, pspec=<optimized out>) at gsettings.c:492
#3 0x00007f3d293c2098 in object_set_property (nqueue=0xf90e60, value=0xf96408,
pspec=0xf95ca0, object=0xf95540) at gobject.c:1357
Quality 100

21
• Example (before submit):
root@xiuhua:~# sdb root on
Switched to 'root' account mode
root@xiuhua:~# sdb shell
sh-4.1# find /var/lib/corewatcher/
/var/lib/corewatcher/
/var/lib/corewatcher/processed
/var/lib/corewatcher/processed/core_top_1398992161.1307.processed
/var/lib/corewatcher/processed/core_top_1398992226.1333.processed
/var/lib/corewatcher/processed/top_1398992226.txt

22
• Example (after submitted):
root@xiuhua:~# sdb root on
Switched to 'root' account mode
root@xiuhua:~# sdb shell
sh-4.1# find /var/lib/corewatcher/
/var/lib/corewatcher/
/var/lib/corewatcher/processed
/var/lib/corewatcher/processed/core_top_1398992161.1307.submitted
/var/lib/corewatcher/processed/core_top_1398992226.1333.submitted

23
Agenda
• Crash Client
• Crash Server
• CrashDB Web

24
Crash Server – CrashDB Web
• Production Env: https://tz.otcshare.org/crashdb/
• Demo: http://crashdb-y.sh.intel.com/

25
Agenda
• Crash Client
• Crash Server
• CrashDB Web

26
Crash Server – Guilty Function Location
https://bugs.tizen.org/jira/browse/TIVI-649
'Security-server has closed unexpectedly' popped up when playing videos or launching clock
(gdb) bt
#0 0xb4e9c999 in vfprintf () from /lib/libc.so.6
#1 0xb4f3e7b4 in __vsnprintf_chk () from /lib/libc.so.6
#2 0xb5560c00 in __dlog_print () from /usr/lib/libdlog.so.0
#3 0x081019fd in process_cookie_request (sockfd=27) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:367
#4 0x08103b7e in security_server_thread (param=0xb451519c) at /usr/src/debug/security-server-
#5 0xb554be19 in start_thread () from /lib/libpthread.so.0 #6 0xb4f2affe in clone () from
/lib/libc.so.6

27
(gdb) bt
/lib/libc.so.6

28
(gdb) bt
/lib/libc.so.6
Guilty Function

29
(gdb) f 3
367 SEC_SVR_DBG("Server: Cookie created for client PID %d LABEL >%s<",
(gdb) p created_cookie->smack_label
$2 = 0x1777 <Address 0x1777 out of bounds> (gdb) p *created_cookie $4 = {cookie =
"270217{257354063221 e筗Y370230~021024004244", path_len = 49, permission_len = 90, pid =
1562, path = 0x85e0ba8 "/usr/apps/org.tizen.video-player/bin/video-player", permissions = 0x85b6168,
smack_label = 0x1777 <Address 0x1777 out of bounds>, prev = 0x8589190, next = 0x0}

30
(gdb) bt
/lib/libc.so.6
White list
• /lib/libc.so.6
• /usr/lib/libdlog.so.0

31
(gdb) bt
/lib/libc.so.6
White list
• /lib/libc.so.6
• /usr/lib/libdlog.so.0
Guilty Function

Crash_Report_Mechanism_In_Tizen

More Related Content

What's hot

Viewers also liked

Similar to Crash_Report_Mechanism_In_Tizen

Crash_Report_Mechanism_In_Tizen