Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kernelvm 201312-dlmopen

1,992 views

Published on

Published in: Technology
  • Be the first to comment

Kernelvm 201312-dlmopen

  1. 1. dlmopen(3C)をつかった VM たざき はじめ (@thehajime) カーネル/VM探検隊 2013/12/8
  2. 2. デモ 2
  3. 3. dlmopen とは? • dlmopen(3) dlopenの亜種 • • link map ID (lmid)で名前空間分離 • 2006年以降誰もいじってない 3
  4. 4. なぜに dlmopen ? • これ全部VMであげたい! • fs.inotify.max_user_instances = 128 (LXCの場合) • loadavgが30とか • 必要なのはネットワーク部分だ けなのに。。 4
  5. 5. なぜに dlmopen ? • ノードまたがるデバッ グ。。 • 1、2台ならまだ余裕 • 30台で動くアプリを gdb と か無理。。。 5
  6. 6. なぜに dlmopen ? • dlmopenで同一 ELF バイナリを複数回load • lmid (Link-map ID)を別にする • global変数はぶつかるよ? • 退避 • IPアドレスとか、ノード間で違う情報は? • ネットワークスタックも別にする 6
  7. 7. Direct Code Execution でぃれくと こーど えぐじぇきゅーじょん • ns-3 (ネットワークシミュレータ)の拡張 • カーネルコードはネットワーク部のみ利用 • ファイルシステム無: chrootなディレクトリを割当 • /proc もなし(sysctl ライクな口と一部familyのsocketのみ) • LKMも今は使えず • struct net_device <=> ns3::NetDevice • jiffies <=> ns3::Time (仮想クロック) 7
  8. 8. どうやって動く ? #!/usr/bin/python from ns.dce import * from ns.core import * • main (ns-3 シナリオ) • PIEバイナリを dlmopen • 必要なsyscall/libcallを上書き • nodes = NodeContainer() nodes.Create (100) dce = DceManagerHelper() dce.SetNetworkStack ("liblinux.so"); dce.Install (nodes); (weak_alias) カーネル突入部は liblinux.so へ リダイレクト app = DceApplicationHelper() app.SetBinary ("ospfd") app.Install (nodes) Simulator.Stop (Seconds(1000.0)) Simulator.Run () 8
  9. 9. liblinux.so Application (ip, iptables, quagga) POSIX layer • カーネルソースを sharedでビルド DCE TCP UDP Heap Stack ICMP • ARP Netfilter memory Netlink arch/simにglueコード DCCP Virtualization Core layer IPv6 Qdisc IPSec struct net_device SCTP IPv4 Bridging struct net_device jiffies/ gettimeofday() Tunneling bottom halves/rcu/ timer/interrupt Synchronize Kernel layer ns3::NetDevice network simulation core 9 Simulated Clock
  10. 10. rump (netbsd) BSD Stack glue apps (gdb) bt #0 rumpcomp_sockin_sendmsg (s=7, msg=0x703010, flags=0, snd=0x7ffffffed178) at buildrump.sh/src/sys/rump/net/lib/ rumpcomp_user.c:426 #1 0x00007ffff7df8526 in sockin_usrreq (so=so@entry=0x6fedb0, req=req@entry=9, m=0x6cce00, nam=nam@entry= control=control@entry=0x0, l=<optimized out>) at buildrump.sh/src/sys/rump/net/lib/libsockin/sockin.c:510 #2 0x00007ffff7be4e79 in sosend (so=0x6fedb0, addr=0x0, uio=0x7ffffffed500, top=0x6cce00, control=0x0, flags=0, l=0x at /home/tazaki/gitworks/buildrump.sh/src/lib/librumpnet/../../sys/rump/../kern/uipc_socket.c:1048 #3 0x00007ffff7be7b4c in soo_write (fp=<optimized out>, offset=<optimized out>, uio=0x7ffffffed500, cred=<optimize flags=<optimized out>) at /home/tazaki/gitworks/buildrump.sh/src/lib/librumpnet/../../sys/rump/../kern/sys_socket.c:11 #4 0x00007ffff788f620 in dofilewrite (fd=fd@entry=3, fp=0x6f8e80, buf=0x400e88, nbyte=37, offset=0x6f8e80, flags=fla retval=retval@entry=0x7ffffffed5e0) at /home/tazaki/gitworks/buildrump.sh/src/lib/librump/../../sys/rump/../kern/sys_g #5 0x00007ffff788f72f in sys_write (l=<optimized out>, uap=0x7ffffffed5f0, retval=0x7ffffffed5e0) at /home/tazaki/gitwo buildrump.sh/src/lib/librump/../../sys/rump/../kern/sys_generic.c:323 #6 0x00007ffff78de3cd in sy_call (rval=0x7ffffffed5e0, uap=0x7ffffffed5f0, l=0x700800, sy=<optimized out>) at /home/ta buildrump.sh/src/lib/librump/../../sys/rump/../sys/syscallvar.h:61 #7 rump_syscall (num=num@entry=4, data=data@entry=0x7ffffffed5f0, dlen=dlen@entry=24, retval=retval@entry=0x at /home/tazaki/gitworks/buildrump.sh/src/lib/librump/../../sys/rump/librump/rumpkern/rump.c:1024 #8 0x00007ffff78d573b in rump___sysimpl_write (fd=<optimized out>, buf=<optimized out>, nbyte=<optimized out> tazaki/gitworks/buildrump.sh/src/lib/librump/../../sys/rump/librump/rumpkern/rump_syscalls.c:121 #9 0x0000000000400d08 in main () at webbrowser.c:86 (gdb) 10
  11. 11. OSv BSD Stack glue apps (java) (gdb) bt #0 if_transmit (ifp=0xffffc0003fdfa800, m=0xffffc00005bfe100) at ../../bsd/sys/net/if.c:3082 #1 0x0000000000252a57 in ether_output_frame (ifp=0xffffc0003fdfa800, m=0xffffc00005bfe100) at ../../bsd/sys/net/if_e #2 0x0000000000252a0a in ether_output (ifp=0xffffc0003fdfa800, m=0xffffc00005bfe100, dst=0xffffc0003e9e8db0, ro=0x2000059102a0) at ../../bsd/sys/net/if_ethersubr.c:356 #3 0x0000000000277982 in ip_output (m=0xffffc00005bfe100, opt=0x0, ro=0x2000059102a0, flags=0, imo=0x0, inp=0xffffc00009ea6400) at ../../bsd/sys/netinet/ip_output.c:612 #4 0x000000000028cb49 in tcp_output (tp=0xffffc00009eafc00) at ../../bsd/sys/netinet/tcp_output.c:1219 #5 0x0000000000296276 in tcp_output_connect (so=0xffffc0000a5a0800, nam=0xffffc00005a8e140) at ../../bsd/sys/netin tcp_offload.h:270 #6 0x0000000000296b25 in tcp_usr_connect (so=0xffffc0000a5a0800, nam=0xffffc00005a8e140, td=0x0) at ../../bsd/sys tcp_usrreq.c:453 #7 0x000000000023503e in soconnect (so=0xffffc0000a5a0800, nam=0xffffc00005a8e140, td=0x0) at ../../bsd/sys/kern/u 744 #8 0x000000000023ad0e in kern_connect (fd=46, sa=0xffffc00005a8e140) at ../../bsd/sys/kern/uipc_syscalls.c:364 #9 0x00000000002511fa in linux_connect (s=46, name=0x200005910660, namelen=16) at ../../bsd/sys/compat/linux/linu #10 0x000000000023c088 in connect (fd=46, addr=0x200005910660, len=16) at ../../bsd/sys/kern/uipc_syscalls_wrap.c:1 #11 0x000010000220c65a in NET_Connect () #12 0x000010000220d0fa in Java_java_net_PlainSocketImpl_socketConnect () #13 0x000020000021cd8e in ?? () #14 0x00002000059106d8 in ?? () (snip) (gdb) 11
  12. 12. DCE (dce:node0) bt #0 sim_dev_xmit (dev=0x7ffff5587020, data=0x7ffff3e0688a "", len=105) at arch/sim/sim.c:349 #1 kernel_dev_xmit (skb=0x7ffff5ccaa68, dev=0x7ffff5587020) at arch/sim/sim-device.c:20 #2 dev_hard_start_xmit (skb=0x7ffff5ccaa68, dev=0x7ffff5587020, txq=0x7ffff5571a90) at net/core/dev.c:2580 #3 dev_queue_xmit (skb=0x7ffff5ccaa68) at net/core/dev.c:2830 #4 neigh_hh_output (skb=0x7ffff5ccaa68, hh=0x7ffff5ce8850) at include/net/neighbour.h:357 #5 dst_neigh_output (skb=0x7ffff5ccaa68, n=0x7ffff5ce8790, dst=0x7ffff3e045d0) at include/net/dst.h:409 #6 ip_finish_output2 (skb=0x7ffff5ccaa68) at net/ipv4/ip_output.c:201 #7 ip_finish_output (skb=0x7ffff5ccaa68) at net/ipv4/ip_output.c:234 #8 ip_output (skb=0x7ffff5ccaa68) at net/ipv4/ip_output.c:307 #9 dst_output (skb=0x7ffff5ccaa68) at include/net/dst.h:448 #10 ip_local_out (skb=0x7ffff5ccaa68) at net/ipv4/ip_output.c:110 #11 ip_queue_xmit (skb=0x7ffff5ccaa68, fl=0x7ffff3e04e78) at net/ipv4/ip_output.c:403 #12 tcp_transmit_skb (sk=0x7ffff3e04bd0, skb=0x7ffff5ccaa68, clone_it=1, gfp_mask=32) at net/ipv4/tcp_output.c:1021 #13 mptcp_write_xmit (meta_sk=0x7ffff3e053d0, mss_now=1428, nonagle=0, push_one=0, gfp=32) at net/mptcp/mptcp_output.c:1 #14 tcp_write_xmit (sk=0x7ffff3e053d0, mss_now=516, nonagle=0, push_one=0, gfp=32) at net/ipv4/tcp_output.c:1930 #15 __tcp_push_pending_frames (sk=0x7ffff3e053d0, cur_mss=516, nonagle=0) at net/ipv4/tcp_output.c:2154 #16 tcp_push_pending_frames (sk=0x7ffff3e053d0) at include/net/tcp.h:1610 #17 do_tcp_setsockopt (sk=0x7ffff3e053d0, level=6, optname=3, optval=0x7ffff439cc78 "", optlen=4) at net/ipv4/tcp.c:2625 #18 tcp_setsockopt (sk=0x7ffff3e053d0, level=6, optname=3, optval=0x7ffff439cc78 "", optlen=4) at net/ipv4/tcp.c:2762 #19 sock_common_setsockopt (sock=0x7ffff3e03850, level=6, optname=3, optval=0x7ffff439cc78 "", optlen=4) at net/core/sock.c:24 #20 sim_sock_setsockopt (socket=0x7ffff3e03850, level=6, optname=3, optval=0x7ffff439cc78, optlen=4) at arch/sim/sim-socket.c:16 glue(linux) #21 sim_sock_setsockopt_forwarder (v0=0x7ffff3e03850, v1=6, v2=3, v3=0x7ffff439cc78, v4=4) at arch/sim/sim.c:97 #22 ns3::LinuxSocketFdFactory::Setsockopt (this=0x64f000, socket=0x7ffff3e03850, level=6, optname=3, optval=0x7ffff439cc78, optl linux-socket-fd-factory.cc:947 glue(POSIX)#23 ns3::LinuxSocketFd::Setsockopt (this=0x815f20, level=6, optname=3, optval=0x7ffff439cc78, optlen=4) at ../model/linux-socket-f #24 dce_setsockopt (fd=11, level=6, optname=3, optval=0x7ffff439cc78, optlen=4) at ../model/dce-fd.cc:529 #25 setsockopt () at ../model/libc-ns3.h:179 #26 sockopt_cork (sock=11, onoff=0) at sockunion.c:534 #27 bgp_write (thread=0x7ffff439ce10) at bgp_packet.c:691 #28 thread_call (thread=0x7ffff439ce10) at thread.c:1177 #29 main (argc=5, argv=0x658100) at bgp_main.c:455 #30 ns3::DceManager::DoStartProcess (context=0x6fa970) at ../model/dce-manager.cc:281 #31 ns3::TaskManager::Trampoline (context=0x6fab50) at ../model/task-manager.cc:274 12 #32 ns3::UcontextFiberManager::Trampoline (a0=32767, a1=-139668064, a2=0, a3=7318352) at ../model/ucontext-fiber-manager.cc:1 glue Linux Stack apps
  13. 13. valgrind ==5864== Memcheck, a memory error detector ==5864== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al. ==5864== Using Valgrind-3.6.0.SVN and LibVEX; rerun with -h for copyright info ==5864== Command: ../build/bin/ns3test-dce-vdl --verbose ==5864== ==5864== Conditional jump or move depends on uninitialised value(s) ==5864== at 0x7D5AE32: tcp_parse_options (tcp_input.c:3782) ==5864== by 0x7D65DCB: tcp_check_req (tcp_minisocks.c:532) ==5864== by 0x7D63B09: tcp_v4_hnd_req (tcp_ipv4.c:1496) ==5864== by 0x7D63CB4: tcp_v4_do_rcv (tcp_ipv4.c:1576) ==5864== by 0x7D6439C: tcp_v4_rcv (tcp_ipv4.c:1696) ==5864== by 0x7D447CC: ip_local_deliver_finish (ip_input.c:226) ==5864== by 0x7D442E4: ip_rcv_finish (dst.h:318) ==5864== by 0x7D2313F: process_backlog (dev.c:3368) ==5864== by 0x7D23455: net_rx_action (dev.c:3526) ==5864== by 0x7CF2477: do_softirq (softirq.c:65) ==5864== by 0x7CF2544: softirq_task_function (softirq.c:21) ==5864== by 0x4FA2BE1: ns3::TaskManager::Trampoline(void*) (task-manager.cc:261) ==5864== Uninitialised value was created by a stack allocation ==5864== at 0x7D65B30: tcp_check_req (tcp_minisocks.c:522) ==5864== 13
  14. 14. gdb Home Agent AP1 correspondent node ping6 AP2 Wi-Fi Wi-Fi handoff mobile node (gdb) b mip6_mh_filter if dce_debug_nodeid()==0 Breakpoint 1 at 0x7ffff287c569: file net/ipv6/mip6.c, line 88. <continue> (gdb) bt 4 #0  mip6_mh_filter (sk=0x7ffff7f69e10, skb=0x7ffff7cde8b0) at net/ipv6/mip6.c:109 #1  0x00007ffff2831418 in ipv6_raw_deliver (skb=0x7ffff7cde8b0, nexthdr=135) at net/ipv6/raw.c:199 #2  0x00007ffff2831697 in raw6_local_deliver (skb=0x7ffff7cde8b0, nexthdr=135) at net/ipv6/raw.c:232 #3  0x00007ffff27e6068 in ip6_input_finish (skb=0x7ffff7cde8b0) at net/ipv6/ip6_input.c:197 14
  15. 15. CI • Linuxカーネルテスト • gcov • nightly build/tests 15
  16. 16. ありがとうございました Direct Code Execution http://bit.ly/ns-3-dce https://github.com/direct-code-execution 16

×