• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
110864103 adventures-in-bug-hunting
 

110864103 adventures-in-bug-hunting

on

  • 361 views

 

Statistics

Views

Total Views
361
Views on SlideShare
324
Embed Views
37

Actions

Likes
1
Downloads
0
Comments
0

1 Embed 37

https://twitter.com 37

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    110864103 adventures-in-bug-hunting 110864103 adventures-in-bug-hunting Presentation Transcript

    • adventuresinbug hunting@joedamatohttp://timetobleed.com
    • whoami
    • http://timetobleed.com
    • @joedamato
    • http://boundary.com(make use of it)
    • first, a confession.
    • debugging > programming
    • before we get thishorror show rolling• kernels, drivers, glibc, and everything elsechanges.• code snips will differ from what you arerunning on your machines.• some things are simplified in the interest oftime.
    • bprobe• boundary IPFIX flow meter• collects flow data by sniffing packets with libpcap• also collects low level NIC data from the driver• packets tx/rx• bytes tx/rx• ethernet collisions• ethernet errors
    • ethernet bonding (aka teaming)• combine a group of physical NICs (eth0, eth1, ...)into a single virtual device (bond0, bond1, ...).• different modes• active-passive• round robin• link aggregation
    • ethernet bonding (aka teaming)
    • how does bonding work (on linux) ?• at a high level...• the bonding driver creates a “virtual device”• when a packet is sent, bonding driver figuresout which physical NIC to transmit the packeton.• when a packet comes in, the NICs pass theincoming packet up for the higher layers of thenetwork stack to figure out.
    • bprobe and bonding• bprobe discovers bonded networkinterfaces.• uses libpcap to monitor the underlyingphysical NICs instead of bond devices.• detecting link failures, etc
    • everything was lookinggood until....
    • Bug was filed...• Debian Lenny, 64bit.• Bonded ethernet interfaces.•No incoming packets are showing up.
    • Step 0•Take a step back.•Breathe.•Do not break the computer.
    • Step 1• Examine our assumptions:• The packets are making it to the kernel.• The packets are being handed up from thekernel to libpcap.• libpcap doesn’t lose any packets beforebprobe examines them.• bprobe has some weird bug in it.
    • packets are making it to the kernel?
    • watch -n 1 cat /proc/net/dev
    • packets are making it to the kernel
    • packets are being handed up from the kernel?
    • Peel some layers away• bprobe is really libpcap + packet analysis +output.• if this is a bug in the kernel or libpcap thenother programs that use libpcap (liketcpdump) will also fail the same way.• so, do they?
    • tcpdump• bonded ethernet interfaces (on linux) are virtualdevices created by combining other devices.• for example:• bond0• eth0• eth2• eth4• ...
    • First, sniff bond0...% sudo tcpdump -i bond0 dst 172.16.209.136 and proto 112:57:26.275660 IP 172.16.209.1 > 172.16.209.136: ICMPecho request, id 62831, seq 54, length 6412:57:27.275731 IP 172.16.209.1 > 172.16.209.136: ICMPecho request, id 62831, seq 55, length 64^C2 packets captured2 packets received by filter0 packets dropped by kernel
    • Everything is cool.
    • Now eth0 (the active NIC inbond0)% sudo tcpdump -i eth0 dst172.16.209.136 and proto 1^C0 packets captured2 packets received by filter0 packets dropped by kernel
    • Everything is not cool.
    • incoming packets appearto be missing when sniffingthe physical device.
    • (only on debian lenny)
    • outgoing packets show upregardless.
    • tcpdump mailing list
    • only way to figure out where theyare getting lost is to follow themthrough the kernel.
    • Step 2Let’s start digging.
    • Steps 3-5• Dig until you see something you haven’tseen before.• Read all of the code and understand it.• Go to step 2.
    • how are packets received?• packets come in from the wire.• a couple different ways for the kernel to“know” about new packets.• let’s just look at the simple case.• an interrupt is raised when a packet arrives.• both paths hand data up to the higherlayers in similar ways.
    • e1000
    • e1000
    • netif_rx• queues packets up.• another thread pulls packets off and processes them.
    • OK, but how does pcap findout about these packets?
    • a more fundamental question...how does pcap actuallywork?
    • packet protocol family(in the kernel)libpcap(in userland)bprobe/tcpdump/etc(in userland)network device agnostic layer(in the kernel)
    • packet protocol family(in the kernel)libpcap(in userland)bprobe/tcpdump/etc(in userland)network device agnostic layer(in the kernel)
    • bprobe/tcpdump/etc(userland)• call pcap_open_live or pcap_create/pcap_activateto initialize libpcap.• call pcap_next_ex to get packets from libpcap.• examine the packets and do stuff.
    • packet protocol family(in the kernel)libpcap(in userland)bprobe/tcpdump/etc(in userland)network device agnostic layer(in the kernel)
    • libpcap (userland)• creates a socket of type PF_PACKET• two ways to get get packets from the kernel:• one by one (slow)• via shared memory (fast)• libpcap tries to use the fast method• if it fails, it falls back to slow.
    • libpcap creating PF_PACKET socket
    • new “fast” way is being setup.
    • the new way of pullingpackets out.
    • the old way is getting setup whenthe new way failed to initialize.
    • pull packets out fromthe kernel the old way.
    • packet protocol family(in the kernel)libpcap(in userland)bprobe/tcpdump/etc(in userland)network device agnostic layer(in the kernel)
    • PF_PACKET (kernel)• libpcap creates the PF_PACKET socket• the PF_PACKET code in the kernel(eventually) executes.• this code does some initialization andinserts a protocol hook...
    • packet protocol family(in the kernel)libpcap(in userland)bprobe/tcpdump/etc(in userland)network device agnostic layer(in the kernel)
    • network device agnostic layer• pulls packets off the backlog queue.• calls netif_receive_skb()• has some logic to determine who the realsender is when bonding is enabled.• passes the packet through the protocolhooks.
    • (run through all protocol blockshanding the packet over)
    • we now know the path packets takeso they can be examined by pcap apps.
    • packet protocol family(in the kernel)libpcap(in userland)bprobe/tcpdump/etc(in userland)network device agnostic layer(in the kernel)
    • back to the bug• so, the bug was that packeting sniffingphysical NICs on bonded hosts was notrevealing incoming packets.• what do we now know about ourenvironment?• what would be the best place to look totrack down this bug?
    • we know
    • assume the followingsetup• bond0• eth0• eth1• eth2• packet came in on eth0• thus:• skb->dev = eth0• skb->dev->master = bond0
    • we know
    • before• skb->dev = eth0after• skb->dev = bond0• code returns eth0 as orig_dev
    • we know
    • LOOK
    • we know
    • Did you see it?
    • Bug• We overwrite the packet’s device with the bonddevice.• The protocol hook check, checks to see if the hook isfor the device on the packet.• It isn’t• we are sniffing eth0• skb->dev was overwritten to bond0.• That’s why if you sniff “bond0” you see packets but ifyou sniff “eth0” you see nothing.
    • packets are being handed up from the kernel
    • YYYYYyyyyYYyYYyyYYyYYYYYYYYYYYYYYYyYeeeEEeeEEeEEEeEEEEeeeeeEEeEEEeeEEeEeEEaAAaaaAaaAaAAaAaAAaaAaAAAAAaaaAAa!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    • EASY FIX
    • YYYYYyyyyYYyYYyyYYyYYYYYYYYYYYYYYYyYeeeEEeeEEeEEEeEEEEeeeeeEEeEEEeeEEeEeEEaAAaaaAaaAaAAaAaAAaaAaAAAAAaaaAAa!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    • reboot and try the newkernel...
    • First, sniff bond0...% sudo tcpdump -i bond0 dst 172.16.209.136 and proto 112:57:26.275660 IP 172.16.209.1 > 172.16.209.136: ICMPecho request, id 62831, seq 54, length 6412:57:27.275731 IP 172.16.209.1 > 172.16.209.136: ICMPecho request, id 62831, seq 55, length 64^C2 packets captured2 packets received by filter0 packets dropped by kernel
    • Everything is cool.
    • Now eth0 (the active NIC inbond0)% sudo tcpdump -i eth0 dst172.16.209.136 and proto 1^C0 packets captured2 packets received by filter0 packets dropped by kernel
    • Everything is not cool.
    • NO
    • !"
    • нет
    • NEIN!
    • tcpdump/bprobe/otherpcap apps STILL FAIL.
    • ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
    • In real life I spent thenext 4 days looking overthe same kernel code,hundreds of times.
    • Every single day from themoment I woke up (9am) until Isearched all day until I collapsedwith exhaustion (3am).
    • I got so wound up in tryingto get my fix working, I losttrack of the process.
    • It was a miserable 4 days.
    • Until I realized...
    • Step 0•Take a step back.•Breathe.•Do not break the computer.
    • Step 1• Examine our assumptions:• The kernel code is still broken.• The incoming packets are being queued up forlibpcap to pull out of PF_PACKET properly.• There probably isn’t bug in bProbe andtcpdump.
    • Step 2Let’s start digging.
    • Steps 3-5• Dig until you see something you haven’tseen before.• Read all of the code and understand it.• Go to step 2.
    • verify my assumptionmodify libpcap to verify that thekernel really is still broken
    • i used apt-get source toretrieve the official source fordebian lenny’s libpcap and Ifound somethingsurprising.
    • old way of doing pcap• debian lenny’s kernel supports the new wayof getting packets out of the kernel viammap.•but, debian lenny’s libpcap is not newenough and therefore uses the old wayto examine packets.•this also means that unless i statically linkthe libpcap version i want, my app will justperform worse on lenny.
    • reading a packet the oldway
    • that if statement fails.• we are sniffing packets on a physical device• BUT in the kernel we are changing thedevice a packet comes in on to the bonddevice (remember in netif_receive_skb?)
    • that if statement fails.• the index of the bond device is different fromthe index of the physical device we are sniffing• so this if statement evaluates to TRUE• libpcap returns without processingthe packet.
    • why?this code exists to prevent a race conditionwhen sniffing packets the old way in somekernels.
    • solution• boot into our fixed debian lenny kernel.• download a version of libpcap that is newer andsupports the mmap method for packet sniffing.• new method doesn’t have this race conditionand has better performance.• link bprobe/tcpdump/other pcap apps against it.
    • First, sniff bond0...% sudo tcpdump -i bond0 dst 172.16.209.136 and proto 112:57:26.275660 IP 172.16.209.1 > 172.16.209.136: ICMPecho request, id 62831, seq 54, length 6412:57:27.275731 IP 172.16.209.1 > 172.16.209.136: ICMPecho request, id 62831, seq 55, length 64^C2 packets captured2 packets received by filter0 packets dropped by kernel
    • Next, sniff eth0...% sudo tcpdump -i eth0 dst 172.16.209.136 and proto 112:57:26.275660 IP 172.16.209.1 > 172.16.209.136: ICMPecho request, id 62831, seq 54, length 6412:57:27.275731 IP 172.16.209.1 > 172.16.209.136: ICMPecho request, id 62831, seq 55, length 64^C2 packets captured2 packets received by filter0 packets dropped by kernel
    • YYYYYyyyyYYyYYyyYYyYYYYYYYYYYYYYYYyYeeeEEeeEEeEEEeEEEEeeeeeEEeEEEeeEEeEeEEaAAaaaAaaAaAAaAaAAaaAaAAAAAaaaAAa!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    • summarize• kernel bug when overwriting the device thepacket arrived on.• fixed this bug, but bprobe/tcpdump stillfailed.• libpcap bug when pulling packets out thekernel the old way• can avoid this bug and get betterperformance with a newer libpcap
    • Step 0•Take a step back.•Breathe.•Do not break the computer.
    • Step 1-5• Examine your assumptions.• Start digging.• Keep going until you see something youhaven’t seen before.• Read all of the code and understand it.• Go to step 2.
    • Спасибо!Happy debugging!
    • questions?twitter: @joedamatoblog: http://timetobleed.com
    • if there is extra time...
    • an warmup bug
    • cool operating system.
    • no, not really.
    • but, people use it.
    • ipfix_reader•a test program•links against yajl because itgenerates JSON output•works on ubuntu, but noton centos5
    • TOO EASY, JOE.
    • but, wait.
    • here’s another program thatlinks fine to a lib in /usr/local/libON THE SAME SYSTEM.
    • W A T• We have 2 programs:• Both link against libraries in /usr/local/lib/• Only one works.• The broken program’s library is in /usr/local/lib/
    • Step 0•Take a step back.•Breathe.•Do not break the computer.
    • Step 1• Examine our assumptions:• The programs and libraries are both 64bit.• /usr/local/lib/ is in the library search path
    • both programs and their libraries are 64bit.?
    • program 1: ipfix_reader
    • program 2: bprobe
    • both programs and their libraries are 64bit.
    • /usr/local/lib/ is in the library search path?
    • Let’s check...ldconfig -p
    • /usr/local/lib/ is in the library search path
    • So...ipfix_reader doesn’t workbecause /usr/local/lib isnot in the search path.
    • but...how can bprobe beworking fine?
    • Strange• This is confusing.• bprobe should fail.• But, the shared libraries a particular binarydynamically links to at runtime are builtinto the binary itself.• So....
    • Step 2Let’s start digging.
    • Steps 3-5• Dig until you see something you haven’tseen before.• Read all of the code and understand it.• Go to step 2.
    • Let’s take a look withreadelf
    • (let’s resize it)
    • rpath
    • ah ha!• bprobe works and can link because thebinary is storing the library path inside ofitself.• but, now there are 2 more questions:• how did the rpath tag get there?• why doesn’t ipfix_reader have one?
    • how did the rpath tag get there?
    • why doesn’t ipfix_reader have rpath?
    • almost forgot...
    • an warmup bug feature