25 September 2014
Docker, 그 기발한 활용법
삼성SDS Perfect ICT 연구회
정영훈 수석, 박재화 수석
dumpdocker - 자동 덤프 분석 툴

Perfect ICT 연구회
삼성SDS의 기술 연구 모임
-세계 최고 수준의 장애 분석 전문가 양성을 목표로 함
-오픈 소스의 데이터센터 내 활용에 대한 연구 수행
-ICT 인프라 문제 해결 방법론 연구 수행
Perfect RCA + OPEN ICT = Perfect ICT

목차
•dumpdocker 탄생 비화: 속 터지는 덤프 분석
•덤프 분석에 docker 를 활용해 보자
•First Pass 덤프 자동 분석
•dumpdocker 프로젝트 로드맵

dumpdocker 탄생 비화:
속 터지는 덤프 분석

리눅스 서버와 오픈 소스 사용의 급격한 증가

오픈 소스 운영 환경의 특징
다양한 하드웨어, 리눅스 배포판
유사한 기능의 많은 소프트웨어

KVM Crash 덤프 분석 사례
KVM crash 발생, 덤프 분석 요청
CRASH
조금만 기다려 주세요.
D+05
헉 덤프 사이즈가 60GB 가 넘는 군요. 분석하는데 시간이 많이 들 듯.
D+04
좀 더 기다려 주세요.
D+06
조금만 더 기다려 주세요.
D+07
NOT YET!!!
D+10
NOT YET!!! size 가 너무 커서 분석에 시간이 걸려요.
D+14
NOT YET. SORRY....
D+20
일단 완전치는 않으나 분석 결과 임.
(gdb) info stack #0 0x00007fd427c03445 in ?? () from gdbenv/lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fd42ebd3970 in ?? () #2 0x202730203d3d2074 in ?? () #3 0x0a2e64656c696166 in ?? () #4 0x0000000000000000 in ?? ()
D+25
로그 좀 수집해 주세요...?
D+30
Workaround 제공
D+32
Thank you ?

무엇이 문제인가?
분석 환경 구축이 어렵다.
덤프 분석에 대해 운영자도 (기본 정도는) 알면 좋다.
 First Pass 분석 수준
(gdb) bt
#0 0x00007fd427c03445 in ?? ()
#1 0x00007fd427c06bab in ?? ()
#2 0x00007fd42bbc48c3 in ?? ()
#3 0x00007fd427d4717d in ?? ()
#4 0x00007fd42bb0d070 in ?? () at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/memory.c:316
...
(gdb) info shared
From To Syms Read Shared Object Library
0x00007fd42b4ea1e0 0x00007fd42b4ed6b8 Yes (*) ./libs/lib/x86_64-linux-gnu/librt.so.1
0x00007fd42b20c990 0x00007fd42b279728 Yes (*) ./libs/lib/x86_64-linux-gnu/libglib-2.0.so.0
...
0x00007fd428e9d640 0x00007fd428f1c3f8 Yes (*) ./libs/usr/lib/x86_64-linux-gnu/libgnutls.so.26
0x00007fd428bf7f40 0x00007fd428c46e98 Yes (*) ./libs/usr/lib/x86_64-linux-gnu/libSDL-1.2.so.0
No /usr/lib/x86_64-linux-gnu/libX11.so.6
0x00007fd4286b85a0 0x00007fd4286b8754 Yes (*) ./libs/lib/x86_64-linux-gnu/libaio.so.1
...
좀 SMART 하게 이 문제들을 해결 할 수 없을까?

libraries
/usr/bin/kvm
gdb
dumpdb
덤프 분석에
를 활용해보자!

dumpdocker 아키텍처
GitHub
docker Repository
docker-1
분석 환경 구축
자동 덤프 분석
docker-2
덤프 KDB
Search engine
Dump Knowledge DB
First Pass
분석보고서
알려진 문제 검색
dumpdocker

docker is an open-source engine that automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere.

Server
Host OS
Hypervisor
Guest OS
Guest OS
Bins/Libs
Bins/Libs
App A
App B
Virtual Machine
가상화된 어플리케이션에는
어플리케이션 자체와 필요한
바이너리/라이브러리 뿐만 아니라
운영체제 전체가 포함됨

Server
Host OS
Docker Engine
Bins/Libs
Bins/Libs
App A
App B
Docker
도커 엔진 컨테이너는
어플리케이션과 꼭 필요한
바이너리/라이브러리만 갖춤
운영체제 상의 사용자 공간에
격리된 프로세스로 구동되기
때문에 가상 머신의 이점과 함께
이동성이 훨씬 뛰어남

docker 퀵인트로: 이미지 다운로드
[root@rhel7sds ~/docker] docker pull rhel7
Pulling repository rhel7
e1f5733f050b: Pulling image (latest) from rhel7, endpoint: http://registry.access.redhat.com/v1/
e1f5733f050b: Download complete

docker 퀵인트로: 이미지 리스트
[root@rhel7sds ~/docker] docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
fedora rawhide 58faa899733f 6 days ago 371.7 MB
fedora 20 ea7d6801c538 6 days ago 373.7 MB
fedora heisenbug ea7d6801c538 6 days ago 373.7 MB
fedora latest ea7d6801c538 6 days ago 373.7 MB
fedora 21 58faa899733f 6 days ago 371.7 MB
redhat/rhel latest 463ff6be4238 3 weeks ago 140.2 MB
rhel7 0 463ff6be4238 3 weeks ago 140.2 MB
rhel7 0-21 463ff6be4238 3 weeks ago 140.2 MB
rhel7 latest 463ff6be4238 3 weeks ago 140.2 MB
redhat/rhel7 0 463ff6be4238 3 weeks ago 140.2 MB
redhat/rhel7 0-21 463ff6be4238 3 weeks ago 140.2 MB
redhat/rhel7 latest 463ff6be4238 3 weeks ago 140.2 MB

docker 퀵인트로: 실행
[root@rhel7sds ~/docker] cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)
[root@rhel7sds ~/docker] docker run -i -t fedora /bin/bash
bash-4.2# cat /etc/redhat-release
Fedora release 20 (Heisenbug)
bash-4.2# exit
[root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash
bash-4.3# cat /etc/redhat-release
Fedora release 21 (Twenty One)
bash-4.3# exit

docker 퀵인트로: 실행
[root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash
bash-4.3# cat /etc/redhat-release
Fedora release 21 (Twenty One)
bash-4.3#
[root@rhel7sds ~] docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4043a778e43b fedora:21 /bin/bash 30 seconds ago Up 29 seconds evil_davinci4
[root@rhel7sds ~] ps aux
root 3642 0.0 0.3 229340 5044 pts/13 Sl+ 13:27 0:00 docker run -i -t fedora:21 /bin/bash
root 3650 0.0 0.1 18056 1832 pts/1 Ss+ 13:27 0:00 /bin/bash

docker 퀵인트로: 실행
[root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash
bash-4.3# cat /etc/redhat-release
Fedora release 21 (Twenty One)
bash-4.3# ls
bin dev home lib64 mnt proc run selinux sys usr
boot etc lib media opt root sbin srv tmp var
bash-4.3# gdb
bash: gdb: command not found
bash-4.3# yum install gdb
...
bash-4.3# gdb
GNU gdb (GDB) Fedora 7.8-20.fc21
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
For help, type “help”.
Type “apropos word” to search for commands related to “word”.
(gdb) q
bash-4.3#
[root@rhel7sds ~] docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4043a778e43b fedora:21 /bin/bash 10 minutes ago Up 10 minutes evil_davinci4

docker 퀵인트로: 실행
bash-4.3# exit
exit
[root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash
bash-4.3# gdb
bash: gdb: command not found
[root@rhel7sds ~] docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4d9c95ab95c6 fedora:21 /bin/bash 6 seconds ago Up 5 seconds desperate_pare2

docker 퀵인트로: 활용 사례
참고: http://blog.flux7.com/blogs/docker/8-ways-to-use-docker-in-the-real-world

docker 기반 덤프 분석 환경?

/
var
usr
tmp
sbin
root
proc
lib
mnt
home
etc
dev
boot
bin
share
sbin
local
lib
include
bin
sbin
src
lib
bin
log
tmp
spool

Server
Host OS
Docker Engine
Bins/Libs
App B
덤프가 발생한 서버의
어플리케이션과
바이너리 및 라이브러리로
docker 이미지 생성

dumpdocker

root@dumpenv:/dump# ls
core.2073 libs libs.tar
root@dumpenv:/dump# gdb ./libs/usr/sbin/libvirtd ./core.2073
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
...
Reading symbols from ./libs/usr/sbin/libvirtd...(no debugging symbols found)...done.
warning: exec file is newer than core file.
[New LWP 2074]
[New LWP 2075]
[New LWP 2076]
[New LWP 2077]
[New LWP 2078]
[New LWP 2079]
[New LWP 2080]
[New LWP 2081]
[New LWP 2082]
[New LWP 2083]
[New LWP 2073]
warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the expected address (wrong library or version mismatch?)
warning: Could not load shared library symbols for 65 libraries, e.g. /usr/lib64/libvirt-qemu.so.0.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `libvirtd --daemon'.
#0 0x00007f79d41105bc in ?? ()
(gdb)
덤프 분석: w/o dumpdocker

덤프 분석: w/o dumpdocker
(gdb) info shared
From To Syms Read Shared Object Library
No /usr/lib64/libvirt-qemu.so.0
No /usr/lib64/libvirt.so.0
No /lib64/libblkid.so.1
No /usr/lib64/libnetcf.so.1
No /lib64/libudev.so.0
No /usr/lib64/libpciaccess.so.0
No /usr/lib64/libpcap.so.1
No /lib64/libcap-ng.so.0
No /usr/lib64/libyajl.so.1
No /lib64/libnl.so.1
No /lib64/libaudit.so.1
No /lib64/libdevmapper.so.1.02
No /usr/lib64/libxml2.so.2
No /usr/lib64/libnuma.so.1
No /usr/lib64/libavahi-common.so.3
No /usr/lib64/libavahi-client.so.3
...
...
(gdb)
(gdb) bt
#0 0x00007f79d41105bc in ?? ()
#1 0x0000000000000000 in ?? ()

덤프 분석: w/o dumpdocker
(gdb) set sysroot /dump/libs
Reading symbols from /dump/libs/usr/lib64/libvirt-qemu.so.0...(no debugging symbols found)...done.
Loaded symbols for /dump/libs/usr/lib64/libvirt-qemu.so.0
...
(gdb) info shared
From To Syms Read Shared Object Library
0x00007f79d73c1a30 0x00007f79d73c20a8 Yes (*) /dump/libs/usr/lib64/libvirt-qemu.so.0
0x00007f79d6fc7b20 0x00007f79d7129148 Yes (*) /dump/libs/usr/lib64/libvirt.so.0
...
No /lib64/ld-linux-x86-64.so.2
0x00007f79d27f0e80 0x00007f79d281b618 Yes (*) /dump/libs/lib64/libsepol.so.1
...
No /usr/lib64/sasl2/libanonymous.so
No /usr/lib64/sasl2/libsasldb.so
No /lib64/libdb-4.7.so
No /usr/lib64/sasl2/libcrammd5.so
No /usr/lib64/sasl2/libdigestmd5.so
No /lib64/libnss_files.so.2
(*): Shared library is missing debugging information.
(gdb) bt
#0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /dump/libs/lib64/libpthread.so.0
#1 0x00007f79d6fe58c6 in virCondWait () from /dump/libs/usr/lib64/libvirt.so.0
#2 0x00007f79d6fe5e93 in ?? ()
#3 0x00007f79d6fe56e9 in ?? ()
#4 0x00007f79d410c9d1 in ?? ()
#5 0x00007f79d3a52b5d in ?? ()

덤프 분석: w/ dumpdocker
[root@ubuntu1204 ~/dumpdocker]# dumpdocker /usr/sbin/libvirtd ./core.2073
tar: Removing leading `/' from member names
/sbin/
/sbin/consoletype
/sbin/lsinitrd
/sbin/blkdeactivate
/sbin/partx
/sbin/ldconfig
/sbin/pvck
...
[root@ubuntu1204 ~/dumpdocker]#
root@dumpenv:~# scp root@ubuntu1204:/ubuntu1204.libvirtd.201409121615.tar .
root@ubuntu1204's password:
ubuntu1204.libvirtd.201409121615.tar 0% 0 0.0KB/s --:-- ETA
ubuntu1204.libvirtd.201409121615.tar 0% 16MB 15.8MB/s 01:29 ETA
ubuntu1204.libvirtd.201409121615.tar 100% 97MB 10.1MB/s 02:21 root@dumpenv:~# cat ubuntu1204.libvirtd.201409121615.tar | docker import - ubuntu1204
4537fca38755100fbc0e2a96168f397df1d6f76f3829506a5af0da8a61e6a3fa
덤프 발생 서버:
덤프 분석 서버:

덤프 분석: w/ dumpdocker
root@dumpenv:~# docker run -i -v /dump:/var/crash -t ubuntu1204 /bin/bash
bash-4.1# mkdir /tmp
bash-4.1# cd /var/crash
bash-4.1# ls
core.2073 libs libs.tar
bash-4.1# gdb /usr/sbin/libvirtd ./core.2073
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Reading symbols from /usr/sbin/libvirtd... (no debugging symbols found)...done.
[New Thread 2074]
...
[New Thread 2073]
Loaded symbols for /lib64/libnss_files.so.2Core was generated by `libvirtd --daemon'.
#0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
(gdb) info shared
From To Syms Read Shared Object Library
0x00007f79d73c1a30 0x00007f79d73c20a8 Yes (*) /usr/lib64/libvirt-qemu.so.0
0x00007f79d6fc7b20 0x00007f79d7129148 Yes (*) /usr/lib64/libvirt.so.0
...
0x00007f79c200ea00 0x00007f79c2016698 Yes (*) /usr/lib64/sasl2/libdigestmd5.so
0x00007f79c14001f0 0x00007f79c1408648 Yes (*) /lib64/libnss_files.so.2
(*): Shared library is missing debugging information.
(gdb) bt
#0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00007f79d6fe58c6 in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007f79d6fe5e93 in ?? () from /usr/lib64/libvirt.so.0
#3 0x00007f79d6fe56e9 in ?? () from /usr/lib64/libvirt.so.0
#4 0x00007f79d410c9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f79d3a52b5d in clone () from /lib64/libc.so.6

TODO
[dumpdocker]
- docker 이미지 생성 자동화 스크립트
- First Pass 덤프 분석 리포트 자동 생성
- 덤프 DB docker 생성
- 덤프 분석 사례 수집

운영 서버와 동일한 환경으로 분석 가능
고객 시스템 구성 정보 제외로 보안 이슈 제거
배포본 독립적인 분석/테스트 환경 구축
신속한 장애 분석 환경 구축
오픈소스 기반의 경제적 분석 환경
dumpdocker 덤프 분석 환경

First Pass 덤프 자동 분석
살살해~

First Pass 자동 덤프 분석보고서
12가지 항목에 대해 자동으로 분석하여 결과 출력
===============================
First Pass Dump Analysis Report
===============================
version 1.0
Fri Sep 12 08:30:18 UTC 2014
Exec file name : /usr/bin/kvm
Core file name : CoreDump
Table of contents
=================
1. General information
2. Environment varialbes
3. Stacktrace
4. The failed frame
5. The information of the failed frame
6. Source code information
7. Assembly code
8. Register information
9. Virtual address space
10. Thread information
11. Shared library information
12. Full stacktrace

First Pass 자동 덤프 분석보고서
일반 정보, Signal, 환경 변수
1. General information
======================
Core was generated by `/usr/bin/kvm -name instance-00018bf4 -S -M pc-1.2 -cpu SandyBridge,+pdpe1gb,+os'.
Program terminated with signal 6, Aborted.
2. Environment variables
========================
/usr/bin/kvm
-name
instance-00018bf4
-S
-M
pc-1.2
-cpu
SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-enable-kvm
-m
64225
-smp
16,sockets=16,cores=1,threads=1
...

First Pass 자동 덤프 분석보고서
완벽한 stacktrace!! dumpdocker 환경 복제 덕분
NOTE를 참조해서 고객께 친절한 설명 가능
3. Stacktrace
=============
#0 0x00007fd427c03445 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fd427c06bab in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fd427bfc10e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fd427bfc1b2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932
#5 kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm- 1.2.0+noroms/kvm-all.c:925
#6 0x00007fd42bb0b465 in kvm_irqchip_send_msi (s=0x7fd42e6dfa90, msg=...) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:1113
#7 0x00007fd42baed55b in kvm_apic_mem_write (opaque=<optimized out>, addr=24576, data=<optimized out>, size=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm- 1.2.0+noroms/hw/i386/../kvm/apic.c:161
...
NOTE:
##################################################################
The process called abort().
You should focus on the frame before calling abort().
And it is good to check the application's log file.
##################################################################

First Pass 자동 덤프 분석보고서
문제 발생한 frame 에서 좀 더 자세한 정보 모으기
4. The failed frame #4
======================
#4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932
932 }
5. The information of the failed frame #4
=========================================
Stack frame at 0x7fff368b0780:
rip = 0x7fd42bb0ad88 in kvm_irqchip_commit_routes (/build/qemu-kvm-swljsd/qemu-kvm- 1.2.0+noroms/kvm-all.c:932); saved rip 0x7fd42bb0b465
inlined into frame 5, caller of frame at 0x7fff368b0770
source language c.
Arglist at unknown address.
Locals at unknown address, Previous frame's sp is 0x7fff368b0770
Saved registers:
rbx at 0x7fff368b0748, rbp at 0x7fff368b0750, r12 at 0x7fff368b0758, r13 at 0x7fff368b0760, rip at 0x7fff368b0768

First Pass 자동 덤프 분석보고서
소스 코드 경로 지정 시, 친절한 listing 까지
6. Source code information
==========================
Current source file is /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c
Compilation directory is /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/build/x86_64-softmmu
Located in /dump/src/qemu-kvm-1.2.0+noroms/kvm-all.c
Contains 2045 lines.
Source language is c.
Compiled with DWARF 2 debugging format.
Does not include preprocessor macro info.
Line 932 of "/build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c" starts at address 0x7fd42bb0ad63 <kvm_irqchip_commit_routes+67> and ends at 0x7fd42bb0ad67 <kvm_irqchip_commit_routes+71>.
927 int ret;
928
929 s->irq_routes->flags = 0;
930 ret = kvm_vm_ioctl(s, KVM_SET_GSI_ROUTING, s->irq_routes);
931 assert(ret == 0);
932 }
933
934 static void kvm_add_routing_entry(KVMState *s,
935 struct kvm_irq_routing_entry *entry)
936 {

First Pass 자동 덤프 분석보고서
어셈블리 코드 확인, 해설
7. Assembly code
================
=> 0x4005e5 <convertint+28>: mov (%rax),%eax
0x4005e7 <convertint+30>: mov %eax,-0x4(%rbp)
0x4005ea <convertint+33>: mov -0x4(%rbp),%eax
0x4005ed <convertint+36>: pop %rbp
0x4005ee <convertint+37>: retq
0x4005ef: nop
0x4005f0 <__libc_csu_init>: mov %rbp,-0x28(%rsp)
0x4005f5 <__libc_csu_init+5>: mov %r12,-0x20(%rsp)
0x4005fa <__libc_csu_init+10>: lea 0x200823(%rip),%rbp # 0x600e24
0x400601 <__libc_csu_init+17>: lea 0x20081c(%rip),%r12 # 0x600e24
0x400608 <__libc_csu_init+24>: mov %r13,-0x18(%rsp)
0x40060d <__libc_csu_init+29>: mov %r14,-0x10(%rsp)
0x400612 <__libc_csu_init+34>: mov %r15,-0x8(%rsp)
Failed Instruction=> 0x4005e5 <convertint+28>: mov (%rax),%eax
NOTE:
##################################################################
It tried to load from the memory, the address saved in rax
and copy to the register, eax. It failed because the content
in rax register is not a valid address.
If you need to check assembly code, trace the rax not eax.
##################################################################

First Pass 자동 덤프 분석보고서
레지스터 정보
8. Register information
=======================
rax 0x0 0
rbx 0xffffffea 4294967274
rcx 0xffffffffffffffff -1
rdx 0x6 6
rsi 0x6064 24676
rdi 0x6064 24676
rbp 0x7fd42ebd3770 0x7fd42ebd3770
rsp 0x7fff368b0770 0x7fff368b0770
r8 0xfefefefefefefeff -72340172838076673
r9 0x0 0
r10 0x8 8
r11 0x202 514
r12 0x7fd42e6e0c28 140549288758312
r13 0x3f0 1008
r14 0x7fd42bb0d070 140549242802288
r15 0x7fff368b0888 140734108469384
rip 0x7fd42bb0ad88 0x7fd42bb0ad88
eflags 0x202 [ IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0

First Pass 자동 덤프 분석보고서
Virtual address space 사용 현황
9. Virtual address space
========================
0x00007fc308000000 - 0x00007fc308021000 is load1 : 135168 bytes
0x00007fc308021000 - 0x00007fc308021000 is load2 : 0 bytes
0x00007fc30c000000 - 0x00007fc30c021000 is load3 : 135168 bytes
0x00007fc30c021000 - 0x00007fc30c021000 is load4 : 0 bytes
0x00007fc310000000 - 0x00007fc310021000 is load5 : 135168 bytes
0x00007fc310021000 - 0x00007fc310021000 is load6 : 0 bytes
0x00007fc314000000 - 0x00007fc314021000 is load7 : 135168 bytes
....
0x00007fd42be47000 - 0x00007fd42bf16000 is load691 : 847872 bytes
0x00007fd42bf16000 - 0x00007fd42bf46000 is load692 : 196608 bytes
0x00007fd42bf46000 - 0x00007fd42c7d2000 is load693 : 8962048 bytes
0x00007fd42e6bd000 - 0x00007fd42e853000 is load694 : 1662976 bytes
0x00007fd42e853000 - 0x00007fd42e863000 is load695 : 65536 bytes
0x00007fd42e863000 - 0x00007fd42e8a1000 is load696 : 253952 bytes
0x00007fd42e8a1000 - 0x00007fd42e8b1000 is load697 : 65536 bytes
0x00007fd42e8b1000 - 0x00007fd436958000 is load698 : 134901760 bytes
0x00007fff36890000 - 0x00007fff368b2000 is load699 : 139264 bytes
0x00007fff369f3000 - 0x00007fff369f4000 is load700 : 4096 bytes
0xffffffffff600000 - 0xffffffffff600000 is load701 : 0 bytes
==========================================================================
TOTAL : 68041953280 bytes
64889 MB

First Pass 자동 덤프 분석보고서
Thread 정보
10. Thread information
======================
Id Target Id Frame
56 Thread 0x7fc35cff9700 (LWP 27483) 0x00007fd427cb8384 in pwritev64 () from /lib/x86_64-linux- gnu/libc.so.6
55 Thread 0x7fc35e7fc700 (LWP 57393) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
54 Thread 0x7fc37e7fc700 (LWP 57404) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
53 Thread 0x7fc412bf2700 (LWP 57395) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
52 Thread 0x7fc410be7700 (LWP 28584) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
51 Thread 0x7fd3d6cdc700 (LWP 28585) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
...
3 Thread 0x7fc4133f3700 (LWP 65017) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7fd3dce4d700 (LWP 24695) 0x00007fd427f95d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
* 1 Thread 0x7fd42b8e67c0 (LWP 24676) 0x00007fd427c03445 in raise () from /lib/x86_64-linux- gnu/libc.so.6
Number of threads: 56

First Pass 자동 덤프 분석보고서
Shared object 현황
11. Shared library information
==============================
From To Syms Read Shared Object Library
0x00007fd42b4ea1e0 0x00007fd42b4ed6b8 Yes (*) /lib/x86_64-linux-gnu/librt.so.1
0x00007fd42b20c990 0x00007fd42b279728 Yes (*) /lib/x86_64-linux-gnu/libglib-2.0.so.0
0x00007fd42aff0e20 0x00007fd42aff16a8 Yes (*) /lib/x86_64-linux-gnu/libutil.so.1
0x00007fd42addbe90 0x00007fd42ade9de8 Yes (*) /usr/lib/librbd.so.1
0x00007fd42aa394a0 0x00007fd42ab6f188 Yes (*) /usr/lib/librados.so.2
...
0x00007fd3de494740 0x00007fd3de57d008 Yes (*) /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
0x00007fd3fc0e5f00 0x00007fd3fc0e7dc8 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libplain.so
0x00007fd3de22ffc0 0x00007fd3de232458 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so
0x00007fd3dc4407c0 0x00007fd3dc449058 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so
0x00007fd3dc23a200 0x00007fd3dc23cf78 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so
0x00007fd3d6cfbca0 0x00007fd3d6e162d8 Yes (*) /usr/lib/x86_64-linux-gnu/libdb-5.1.so
0x00007fd3dc034ed0 0x00007fd3dc036cc8 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so
(*): Shared library is missing debugging information.

First Pass 자동 덤프 분석보고서
Full stacktrace
12. Full stacktrace
===================
#0 0x00007fd427c03445 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007fd427c06bab in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2 0x00007fd427bfc10e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#3 0x00007fd427bfc1b2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932
No locals.
#5 kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm- all.c:925
ret = -22
#6 0x00007fd42bb0b465 in kvm_irqchip_send_msi (s=0x7fd42e6dfa90, msg=...) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:1113
virq = 1008
msi = {address_lo = 0, address_hi = 0, data = 0, flags = 0, pad = '000' <repeats 15 times>}
route = 0x7fd42ebd3770
__PRETTY_FUNCTION__ = "kvm_irqchip_send_msi"
#7 0x00007fd42baed55b in kvm_apic_mem_write (opaque=<optimized out>, addr=24576, data=<optimized out>, size=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/hw/i386/../kvm/apic.c:161
msg = {address = 24576, data = 16594}
ret = <optimized out>

libraries
/usr/bin/kvm
gdb
dumpdb
dumpdocker 프로젝트 로드맵

dumpdocker 아키텍처
GitHub
docker Repository
docker-1
분석 환경 구축
자동 덤프 분석
docker-2
덤프 KDB
Search engine
Dump Knowledge DB
First Pass
분석보고서
알려진 문제 검색
dumpdocker

dumpdocker 로드맵
Dump KDB docker
docker 이미지 자동 생성
많은 dump 분석으로 풍부한 KDB 콘텐츠 확보
First pass 자동 덤프 분석
(完)
(完)
Search engine

오픈 소스 덤프 분석 함께 합시다.
정영훈
Firsts Pass Dump Analysis
younghun.chung@gmail.com
박재화
dumpdocker core
jaehwa@gmail.com
이종화
dumpdocker (python)
earthsea@gmail.com
-dumpdocker image + core dump
-E-mail 연락
-Free 분석, 정보 공유
-https://github.com/pjhwa/dumpdocker
with
YOU

THANK YOU