Docker 활용법: dumpdocker

7,435 views

Published on

Presentation @ RED HAT FORUM SEOUL 2014

Published in: Technology
0 Comments
122 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,435
On SlideShare
0
From Embeds
0
Number of Embeds
618
Actions
Shares
0
Downloads
318
Comments
0
Likes
122
Embeds 0
No embeds

No notes for slide

Docker 활용법: dumpdocker

  1. 1. 25 September 2014 Docker, 그 기발한 활용법 삼성SDS Perfect ICT 연구회 정영훈 수석, 박재화 수석 dumpdocker - 자동 덤프 분석 툴
  2. 2. Perfect ICT 연구회 삼성SDS의 기술 연구 모임 -세계 최고 수준의 장애 분석 전문가 양성을 목표로 함 -오픈 소스의 데이터센터 내 활용에 대한 연구 수행 -ICT 인프라 문제 해결 방법론 연구 수행 Perfect RCA + OPEN ICT = Perfect ICT
  3. 3. 목차 •dumpdocker 탄생 비화: 속 터지는 덤프 분석 •덤프 분석에 docker 를 활용해 보자 •First Pass 덤프 자동 분석 •dumpdocker 프로젝트 로드맵
  4. 4. dumpdocker 탄생 비화: 속 터지는 덤프 분석
  5. 5. 리눅스 서버와 오픈 소스 사용의 급격한 증가
  6. 6. 오픈 소스 운영 환경의 특징 다양한 하드웨어, 리눅스 배포판 유사한 기능의 많은 소프트웨어
  7. 7. KVM Crash 덤프 분석 사례 KVM crash 발생, 덤프 분석 요청 CRASH 조금만 기다려 주세요. D+05 헉 덤프 사이즈가 60GB 가 넘는 군요. 분석하는데 시간이 많이 들 듯. D+04 좀 더 기다려 주세요. D+06 조금만 더 기다려 주세요. D+07 NOT YET!!! D+10 NOT YET!!! size 가 너무 커서 분석에 시간이 걸려요. D+14 NOT YET. SORRY.... D+20 일단 완전치는 않으나 분석 결과 임. (gdb) info stack #0 0x00007fd427c03445 in ?? () from gdbenv/lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fd42ebd3970 in ?? () #2 0x202730203d3d2074 in ?? () #3 0x0a2e64656c696166 in ?? () #4 0x0000000000000000 in ?? () D+25 로그 좀 수집해 주세요...? D+30 Workaround 제공 D+32 Thank you ?
  8. 8. 무엇이 문제인가? 분석 환경 구축이 어렵다. 덤프 분석에 대해 운영자도 (기본 정도는) 알면 좋다.  First Pass 분석 수준 (gdb) bt #0 0x00007fd427c03445 in ?? () #1 0x00007fd427c06bab in ?? () #2 0x00007fd42bbc48c3 in ?? () #3 0x00007fd427d4717d in ?? () #4 0x00007fd42bb0d070 in ?? () at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/memory.c:316 ... (gdb) info shared From To Syms Read Shared Object Library 0x00007fd42b4ea1e0 0x00007fd42b4ed6b8 Yes (*) ./libs/lib/x86_64-linux-gnu/librt.so.1 0x00007fd42b20c990 0x00007fd42b279728 Yes (*) ./libs/lib/x86_64-linux-gnu/libglib-2.0.so.0 ... 0x00007fd428e9d640 0x00007fd428f1c3f8 Yes (*) ./libs/usr/lib/x86_64-linux-gnu/libgnutls.so.26 0x00007fd428bf7f40 0x00007fd428c46e98 Yes (*) ./libs/usr/lib/x86_64-linux-gnu/libSDL-1.2.so.0 No /usr/lib/x86_64-linux-gnu/libX11.so.6 0x00007fd4286b85a0 0x00007fd4286b8754 Yes (*) ./libs/lib/x86_64-linux-gnu/libaio.so.1 ... 좀 SMART 하게 이 문제들을 해결 할 수 없을까?
  9. 9. libraries /usr/bin/kvm gdb dumpdb 덤프 분석에 를 활용해보자!
  10. 10. dumpdocker 아키텍처 GitHub docker Repository docker-1 분석 환경 구축 자동 덤프 분석 docker-2 덤프 KDB Search engine Dump Knowledge DB First Pass 분석보고서 알려진 문제 검색 dumpdocker
  11. 11. docker is an open-source engine that automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere.
  12. 12. Server Host OS Hypervisor Guest OS Guest OS Bins/Libs Bins/Libs App A App B Virtual Machine 가상화된 어플리케이션에는 어플리케이션 자체와 필요한 바이너리/라이브러리 뿐만 아니라 운영체제 전체가 포함됨
  13. 13. Server Host OS Docker Engine Bins/Libs Bins/Libs App A App B Docker 도커 엔진 컨테이너는 어플리케이션과 꼭 필요한 바이너리/라이브러리만 갖춤 운영체제 상의 사용자 공간에 격리된 프로세스로 구동되기 때문에 가상 머신의 이점과 함께 이동성이 훨씬 뛰어남
  14. 14. docker 퀵인트로: 이미지 다운로드 [root@rhel7sds ~/docker] docker pull rhel7 Pulling repository rhel7 e1f5733f050b: Pulling image (latest) from rhel7, endpoint: http://registry.access.redhat.com/v1/ e1f5733f050b: Download complete
  15. 15. docker 퀵인트로: 이미지 리스트 [root@rhel7sds ~/docker] docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE fedora rawhide 58faa899733f 6 days ago 371.7 MB fedora 20 ea7d6801c538 6 days ago 373.7 MB fedora heisenbug ea7d6801c538 6 days ago 373.7 MB fedora latest ea7d6801c538 6 days ago 373.7 MB fedora 21 58faa899733f 6 days ago 371.7 MB redhat/rhel latest 463ff6be4238 3 weeks ago 140.2 MB rhel7 0 463ff6be4238 3 weeks ago 140.2 MB rhel7 0-21 463ff6be4238 3 weeks ago 140.2 MB rhel7 latest 463ff6be4238 3 weeks ago 140.2 MB redhat/rhel7 0 463ff6be4238 3 weeks ago 140.2 MB redhat/rhel7 0-21 463ff6be4238 3 weeks ago 140.2 MB redhat/rhel7 latest 463ff6be4238 3 weeks ago 140.2 MB
  16. 16. docker 퀵인트로: 실행 [root@rhel7sds ~/docker] cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@rhel7sds ~/docker] docker run -i -t fedora /bin/bash bash-4.2# cat /etc/redhat-release Fedora release 20 (Heisenbug) bash-4.2# exit [root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash bash-4.3# cat /etc/redhat-release Fedora release 21 (Twenty One) bash-4.3# exit
  17. 17. docker 퀵인트로: 실행 [root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash bash-4.3# cat /etc/redhat-release Fedora release 21 (Twenty One) bash-4.3# [root@rhel7sds ~] docker ps -l CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4043a778e43b fedora:21 /bin/bash 30 seconds ago Up 29 seconds evil_davinci4 [root@rhel7sds ~] ps aux root 3642 0.0 0.3 229340 5044 pts/13 Sl+ 13:27 0:00 docker run -i -t fedora:21 /bin/bash root 3650 0.0 0.1 18056 1832 pts/1 Ss+ 13:27 0:00 /bin/bash
  18. 18. docker 퀵인트로: 실행 [root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash bash-4.3# cat /etc/redhat-release Fedora release 21 (Twenty One) bash-4.3# ls bin dev home lib64 mnt proc run selinux sys usr boot etc lib media opt root sbin srv tmp var bash-4.3# gdb bash: gdb: command not found bash-4.3# yum install gdb ... bash-4.3# gdb GNU gdb (GDB) Fedora 7.8-20.fc21 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. For help, type “help”. Type “apropos word” to search for commands related to “word”. (gdb) q bash-4.3# [root@rhel7sds ~] docker ps -l CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4043a778e43b fedora:21 /bin/bash 10 minutes ago Up 10 minutes evil_davinci4
  19. 19. docker 퀵인트로: 실행 bash-4.3# exit exit [root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash bash-4.3# gdb bash: gdb: command not found [root@rhel7sds ~] docker ps -l CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4d9c95ab95c6 fedora:21 /bin/bash 6 seconds ago Up 5 seconds desperate_pare2
  20. 20. docker 퀵인트로: 활용 사례 참고: http://blog.flux7.com/blogs/docker/8-ways-to-use-docker-in-the-real-world
  21. 21. docker 기반 덤프 분석 환경?
  22. 22. / var usr tmp sbin root proc lib mnt home etc dev boot bin share sbin local lib include bin sbin src lib bin log tmp spool
  23. 23. Server Host OS Docker Engine Bins/Libs App B 덤프가 발생한 서버의 어플리케이션과 바이너리 및 라이브러리로 docker 이미지 생성
  24. 24. dumpdocker
  25. 25. root@dumpenv:/dump# ls core.2073 libs libs.tar root@dumpenv:/dump# gdb ./libs/usr/sbin/libvirtd ./core.2073 GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7 ... Reading symbols from ./libs/usr/sbin/libvirtd...(no debugging symbols found)...done. warning: exec file is newer than core file. [New LWP 2074] [New LWP 2075] [New LWP 2076] [New LWP 2077] [New LWP 2078] [New LWP 2079] [New LWP 2080] [New LWP 2081] [New LWP 2082] [New LWP 2083] [New LWP 2073] warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the expected address (wrong library or version mismatch?) warning: Could not load shared library symbols for 65 libraries, e.g. /usr/lib64/libvirt-qemu.so.0. Use the "info sharedlibrary" command to see the complete listing. Do you need "set solib-search-path" or "set sysroot"? Core was generated by `libvirtd --daemon'. #0 0x00007f79d41105bc in ?? () (gdb) 덤프 분석: w/o dumpdocker
  26. 26. 덤프 분석: w/o dumpdocker (gdb) info shared From To Syms Read Shared Object Library No /usr/lib64/libvirt-qemu.so.0 No /usr/lib64/libvirt.so.0 No /lib64/libblkid.so.1 No /usr/lib64/libnetcf.so.1 No /lib64/libudev.so.0 No /usr/lib64/libpciaccess.so.0 No /usr/lib64/libpcap.so.1 No /lib64/libcap-ng.so.0 No /usr/lib64/libyajl.so.1 No /lib64/libnl.so.1 No /lib64/libaudit.so.1 No /lib64/libdevmapper.so.1.02 No /usr/lib64/libxml2.so.2 No /usr/lib64/libnuma.so.1 No /usr/lib64/libavahi-common.so.3 No /usr/lib64/libavahi-client.so.3 ... ... (gdb) (gdb) bt #0 0x00007f79d41105bc in ?? () #1 0x0000000000000000 in ?? ()
  27. 27. 덤프 분석: w/o dumpdocker (gdb) set sysroot /dump/libs Reading symbols from /dump/libs/usr/lib64/libvirt-qemu.so.0...(no debugging symbols found)...done. Loaded symbols for /dump/libs/usr/lib64/libvirt-qemu.so.0 ... (gdb) info shared From To Syms Read Shared Object Library 0x00007f79d73c1a30 0x00007f79d73c20a8 Yes (*) /dump/libs/usr/lib64/libvirt-qemu.so.0 0x00007f79d6fc7b20 0x00007f79d7129148 Yes (*) /dump/libs/usr/lib64/libvirt.so.0 ... No /lib64/ld-linux-x86-64.so.2 0x00007f79d27f0e80 0x00007f79d281b618 Yes (*) /dump/libs/lib64/libsepol.so.1 ... No /usr/lib64/sasl2/libanonymous.so No /usr/lib64/sasl2/libsasldb.so No /lib64/libdb-4.7.so No /usr/lib64/sasl2/libcrammd5.so No /usr/lib64/sasl2/libdigestmd5.so No /lib64/libnss_files.so.2 (*): Shared library is missing debugging information. (gdb) bt #0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 () from /dump/libs/lib64/libpthread.so.0 #1 0x00007f79d6fe58c6 in virCondWait () from /dump/libs/usr/lib64/libvirt.so.0 #2 0x00007f79d6fe5e93 in ?? () #3 0x00007f79d6fe56e9 in ?? () #4 0x00007f79d410c9d1 in ?? () #5 0x00007f79d3a52b5d in ?? ()
  28. 28. 덤프 분석: w/ dumpdocker [root@ubuntu1204 ~/dumpdocker]# dumpdocker /usr/sbin/libvirtd ./core.2073 tar: Removing leading `/' from member names /sbin/ /sbin/consoletype /sbin/lsinitrd /sbin/blkdeactivate /sbin/partx /sbin/ldconfig /sbin/pvck ... [root@ubuntu1204 ~/dumpdocker]# root@dumpenv:~# scp root@ubuntu1204:/ubuntu1204.libvirtd.201409121615.tar . root@ubuntu1204's password: ubuntu1204.libvirtd.201409121615.tar 0% 0 0.0KB/s --:-- ETA ubuntu1204.libvirtd.201409121615.tar 0% 16MB 15.8MB/s 01:29 ETA ubuntu1204.libvirtd.201409121615.tar 100% 97MB 10.1MB/s 02:21 root@dumpenv:~# cat ubuntu1204.libvirtd.201409121615.tar | docker import - ubuntu1204 4537fca38755100fbc0e2a96168f397df1d6f76f3829506a5af0da8a61e6a3fa 덤프 발생 서버: 덤프 분석 서버:
  29. 29. 덤프 분석: w/ dumpdocker root@dumpenv:~# docker run -i -v /dump:/var/crash -t ubuntu1204 /bin/bash bash-4.1# mkdir /tmp bash-4.1# cd /var/crash bash-4.1# ls core.2073 libs libs.tar bash-4.1# gdb /usr/sbin/libvirtd ./core.2073 GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7 Reading symbols from /usr/sbin/libvirtd... (no debugging symbols found)...done. [New Thread 2074] ... [New Thread 2073] Loaded symbols for /lib64/libnss_files.so.2Core was generated by `libvirtd --daemon'. #0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) info shared From To Syms Read Shared Object Library 0x00007f79d73c1a30 0x00007f79d73c20a8 Yes (*) /usr/lib64/libvirt-qemu.so.0 0x00007f79d6fc7b20 0x00007f79d7129148 Yes (*) /usr/lib64/libvirt.so.0 ... 0x00007f79c200ea00 0x00007f79c2016698 Yes (*) /usr/lib64/sasl2/libdigestmd5.so 0x00007f79c14001f0 0x00007f79c1408648 Yes (*) /lib64/libnss_files.so.2 (*): Shared library is missing debugging information. (gdb) bt #0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f79d6fe58c6 in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007f79d6fe5e93 in ?? () from /usr/lib64/libvirt.so.0 #3 0x00007f79d6fe56e9 in ?? () from /usr/lib64/libvirt.so.0 #4 0x00007f79d410c9d1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f79d3a52b5d in clone () from /lib64/libc.so.6
  30. 30. TODO [dumpdocker] - docker 이미지 생성 자동화 스크립트 - First Pass 덤프 분석 리포트 자동 생성 - 덤프 DB docker 생성 - 덤프 분석 사례 수집
  31. 31. 운영 서버와 동일한 환경으로 분석 가능 고객 시스템 구성 정보 제외로 보안 이슈 제거 배포본 독립적인 분석/테스트 환경 구축 신속한 장애 분석 환경 구축 오픈소스 기반의 경제적 분석 환경 dumpdocker 덤프 분석 환경
  32. 32. First Pass 덤프 자동 분석 살살해~
  33. 33. First Pass 자동 덤프 분석보고서 12가지 항목에 대해 자동으로 분석하여 결과 출력 =============================== First Pass Dump Analysis Report =============================== version 1.0 Fri Sep 12 08:30:18 UTC 2014 Exec file name : /usr/bin/kvm Core file name : CoreDump Table of contents ================= 1. General information 2. Environment varialbes 3. Stacktrace 4. The failed frame 5. The information of the failed frame 6. Source code information 7. Assembly code 8. Register information 9. Virtual address space 10. Thread information 11. Shared library information 12. Full stacktrace
  34. 34. First Pass 자동 덤프 분석보고서 일반 정보, Signal, 환경 변수 1. General information ====================== Core was generated by `/usr/bin/kvm -name instance-00018bf4 -S -M pc-1.2 -cpu SandyBridge,+pdpe1gb,+os'. Program terminated with signal 6, Aborted. 2. Environment variables ======================== /usr/bin/kvm -name instance-00018bf4 -S -M pc-1.2 -cpu SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 64225 -smp 16,sockets=16,cores=1,threads=1 ...
  35. 35. First Pass 자동 덤프 분석보고서 완벽한 stacktrace!! dumpdocker 환경 복제 덕분 NOTE를 참조해서 고객께 친절한 설명 가능 3. Stacktrace ============= #0 0x00007fd427c03445 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fd427c06bab in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007fd427bfc10e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007fd427bfc1b2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932 #5 kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm- 1.2.0+noroms/kvm-all.c:925 #6 0x00007fd42bb0b465 in kvm_irqchip_send_msi (s=0x7fd42e6dfa90, msg=...) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:1113 #7 0x00007fd42baed55b in kvm_apic_mem_write (opaque=<optimized out>, addr=24576, data=<optimized out>, size=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm- 1.2.0+noroms/hw/i386/../kvm/apic.c:161 ... NOTE: ################################################################## The process called abort(). You should focus on the frame before calling abort(). And it is good to check the application's log file. ##################################################################
  36. 36. First Pass 자동 덤프 분석보고서 문제 발생한 frame 에서 좀 더 자세한 정보 모으기 4. The failed frame #4 ====================== #4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932 932 } 5. The information of the failed frame #4 ========================================= Stack frame at 0x7fff368b0780: rip = 0x7fd42bb0ad88 in kvm_irqchip_commit_routes (/build/qemu-kvm-swljsd/qemu-kvm- 1.2.0+noroms/kvm-all.c:932); saved rip 0x7fd42bb0b465 inlined into frame 5, caller of frame at 0x7fff368b0770 source language c. Arglist at unknown address. Locals at unknown address, Previous frame's sp is 0x7fff368b0770 Saved registers: rbx at 0x7fff368b0748, rbp at 0x7fff368b0750, r12 at 0x7fff368b0758, r13 at 0x7fff368b0760, rip at 0x7fff368b0768
  37. 37. First Pass 자동 덤프 분석보고서 소스 코드 경로 지정 시, 친절한 listing 까지 6. Source code information ========================== Current source file is /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c Compilation directory is /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/build/x86_64-softmmu Located in /dump/src/qemu-kvm-1.2.0+noroms/kvm-all.c Contains 2045 lines. Source language is c. Compiled with DWARF 2 debugging format. Does not include preprocessor macro info. Line 932 of "/build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c" starts at address 0x7fd42bb0ad63 <kvm_irqchip_commit_routes+67> and ends at 0x7fd42bb0ad67 <kvm_irqchip_commit_routes+71>. 927 int ret; 928 929 s->irq_routes->flags = 0; 930 ret = kvm_vm_ioctl(s, KVM_SET_GSI_ROUTING, s->irq_routes); 931 assert(ret == 0); 932 } 933 934 static void kvm_add_routing_entry(KVMState *s, 935 struct kvm_irq_routing_entry *entry) 936 {
  38. 38. First Pass 자동 덤프 분석보고서 어셈블리 코드 확인, 해설 7. Assembly code ================ => 0x4005e5 <convertint+28>: mov (%rax),%eax 0x4005e7 <convertint+30>: mov %eax,-0x4(%rbp) 0x4005ea <convertint+33>: mov -0x4(%rbp),%eax 0x4005ed <convertint+36>: pop %rbp 0x4005ee <convertint+37>: retq 0x4005ef: nop 0x4005f0 <__libc_csu_init>: mov %rbp,-0x28(%rsp) 0x4005f5 <__libc_csu_init+5>: mov %r12,-0x20(%rsp) 0x4005fa <__libc_csu_init+10>: lea 0x200823(%rip),%rbp # 0x600e24 0x400601 <__libc_csu_init+17>: lea 0x20081c(%rip),%r12 # 0x600e24 0x400608 <__libc_csu_init+24>: mov %r13,-0x18(%rsp) 0x40060d <__libc_csu_init+29>: mov %r14,-0x10(%rsp) 0x400612 <__libc_csu_init+34>: mov %r15,-0x8(%rsp) Failed Instruction=> 0x4005e5 <convertint+28>: mov (%rax),%eax NOTE: ################################################################## It tried to load from the memory, the address saved in rax and copy to the register, eax. It failed because the content in rax register is not a valid address. If you need to check assembly code, trace the rax not eax. ##################################################################
  39. 39. First Pass 자동 덤프 분석보고서 레지스터 정보 8. Register information ======================= rax 0x0 0 rbx 0xffffffea 4294967274 rcx 0xffffffffffffffff -1 rdx 0x6 6 rsi 0x6064 24676 rdi 0x6064 24676 rbp 0x7fd42ebd3770 0x7fd42ebd3770 rsp 0x7fff368b0770 0x7fff368b0770 r8 0xfefefefefefefeff -72340172838076673 r9 0x0 0 r10 0x8 8 r11 0x202 514 r12 0x7fd42e6e0c28 140549288758312 r13 0x3f0 1008 r14 0x7fd42bb0d070 140549242802288 r15 0x7fff368b0888 140734108469384 rip 0x7fd42bb0ad88 0x7fd42bb0ad88 eflags 0x202 [ IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0
  40. 40. First Pass 자동 덤프 분석보고서 Virtual address space 사용 현황 9. Virtual address space ======================== 0x00007fc308000000 - 0x00007fc308021000 is load1 : 135168 bytes 0x00007fc308021000 - 0x00007fc308021000 is load2 : 0 bytes 0x00007fc30c000000 - 0x00007fc30c021000 is load3 : 135168 bytes 0x00007fc30c021000 - 0x00007fc30c021000 is load4 : 0 bytes 0x00007fc310000000 - 0x00007fc310021000 is load5 : 135168 bytes 0x00007fc310021000 - 0x00007fc310021000 is load6 : 0 bytes 0x00007fc314000000 - 0x00007fc314021000 is load7 : 135168 bytes .... 0x00007fd42be47000 - 0x00007fd42bf16000 is load691 : 847872 bytes 0x00007fd42bf16000 - 0x00007fd42bf46000 is load692 : 196608 bytes 0x00007fd42bf46000 - 0x00007fd42c7d2000 is load693 : 8962048 bytes 0x00007fd42e6bd000 - 0x00007fd42e853000 is load694 : 1662976 bytes 0x00007fd42e853000 - 0x00007fd42e863000 is load695 : 65536 bytes 0x00007fd42e863000 - 0x00007fd42e8a1000 is load696 : 253952 bytes 0x00007fd42e8a1000 - 0x00007fd42e8b1000 is load697 : 65536 bytes 0x00007fd42e8b1000 - 0x00007fd436958000 is load698 : 134901760 bytes 0x00007fff36890000 - 0x00007fff368b2000 is load699 : 139264 bytes 0x00007fff369f3000 - 0x00007fff369f4000 is load700 : 4096 bytes 0xffffffffff600000 - 0xffffffffff600000 is load701 : 0 bytes ========================================================================== TOTAL : 68041953280 bytes 64889 MB
  41. 41. First Pass 자동 덤프 분석보고서 Thread 정보 10. Thread information ====================== Id Target Id Frame 56 Thread 0x7fc35cff9700 (LWP 27483) 0x00007fd427cb8384 in pwritev64 () from /lib/x86_64-linux- gnu/libc.so.6 55 Thread 0x7fc35e7fc700 (LWP 57393) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 54 Thread 0x7fc37e7fc700 (LWP 57404) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 53 Thread 0x7fc412bf2700 (LWP 57395) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 52 Thread 0x7fc410be7700 (LWP 28584) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 51 Thread 0x7fd3d6cdc700 (LWP 28585) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 ... 3 Thread 0x7fc4133f3700 (LWP 65017) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x7fd3dce4d700 (LWP 24695) 0x00007fd427f95d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 * 1 Thread 0x7fd42b8e67c0 (LWP 24676) 0x00007fd427c03445 in raise () from /lib/x86_64-linux- gnu/libc.so.6 Number of threads: 56
  42. 42. First Pass 자동 덤프 분석보고서 Shared object 현황 11. Shared library information ============================== From To Syms Read Shared Object Library 0x00007fd42b4ea1e0 0x00007fd42b4ed6b8 Yes (*) /lib/x86_64-linux-gnu/librt.so.1 0x00007fd42b20c990 0x00007fd42b279728 Yes (*) /lib/x86_64-linux-gnu/libglib-2.0.so.0 0x00007fd42aff0e20 0x00007fd42aff16a8 Yes (*) /lib/x86_64-linux-gnu/libutil.so.1 0x00007fd42addbe90 0x00007fd42ade9de8 Yes (*) /usr/lib/librbd.so.1 0x00007fd42aa394a0 0x00007fd42ab6f188 Yes (*) /usr/lib/librados.so.2 ... 0x00007fd3de494740 0x00007fd3de57d008 Yes (*) /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 0x00007fd3fc0e5f00 0x00007fd3fc0e7dc8 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libplain.so 0x00007fd3de22ffc0 0x00007fd3de232458 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so 0x00007fd3dc4407c0 0x00007fd3dc449058 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so 0x00007fd3dc23a200 0x00007fd3dc23cf78 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so 0x00007fd3d6cfbca0 0x00007fd3d6e162d8 Yes (*) /usr/lib/x86_64-linux-gnu/libdb-5.1.so 0x00007fd3dc034ed0 0x00007fd3dc036cc8 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so (*): Shared library is missing debugging information.
  43. 43. First Pass 자동 덤프 분석보고서 Full stacktrace 12. Full stacktrace =================== #0 0x00007fd427c03445 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007fd427c06bab in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00007fd427bfc10e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #3 0x00007fd427bfc1b2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932 No locals. #5 kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm- all.c:925 ret = -22 #6 0x00007fd42bb0b465 in kvm_irqchip_send_msi (s=0x7fd42e6dfa90, msg=...) at /build/qemu-kvm- swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:1113 virq = 1008 msi = {address_lo = 0, address_hi = 0, data = 0, flags = 0, pad = '000' <repeats 15 times>} route = 0x7fd42ebd3770 __PRETTY_FUNCTION__ = "kvm_irqchip_send_msi" #7 0x00007fd42baed55b in kvm_apic_mem_write (opaque=<optimized out>, addr=24576, data=<optimized out>, size=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/hw/i386/../kvm/apic.c:161 msg = {address = 24576, data = 16594} ret = <optimized out>
  44. 44. libraries /usr/bin/kvm gdb dumpdb dumpdocker 프로젝트 로드맵
  45. 45. dumpdocker 아키텍처 GitHub docker Repository docker-1 분석 환경 구축 자동 덤프 분석 docker-2 덤프 KDB Search engine Dump Knowledge DB First Pass 분석보고서 알려진 문제 검색 dumpdocker
  46. 46. dumpdocker 로드맵 Dump KDB docker docker 이미지 자동 생성 많은 dump 분석으로 풍부한 KDB 콘텐츠 확보 First pass 자동 덤프 분석 (完) (完) Search engine
  47. 47. 오픈 소스 덤프 분석 함께 합시다. 정영훈 Firsts Pass Dump Analysis younghun.chung@gmail.com 박재화 dumpdocker core jaehwa@gmail.com 이종화 dumpdocker (python) earthsea@gmail.com -dumpdocker image + core dump -E-mail 연락 -Free 분석, 정보 공유 -https://github.com/pjhwa/dumpdocker with YOU
  48. 48. THANK YOU

×