This document describes the process of debugging a packet sniffing issue when using bonded ethernet interfaces on Linux. The bug was initially found when incoming packets were not being detected when sniffing individual physical interfaces, even though outgoing packets were working. After carefully examining assumptions and digging through the kernel and libpcap code, the bug was found to be: 1) the kernel was overwriting the packet's device field, and 2) an older version of libpcap was incorrectly handling packets when using its legacy retrieval method. Updating the kernel fix and libpcap resolved the issue.
44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kett...44CON
Just like drinking is not a game in Finland; neither is browser bug hunting - it’s serious business! Browser bugs have been supporting Atte Kettunen (@attekett) traditional Finnish way of living since late 2011 and he’s going to tell you all about how he has been living the dream browser bug hunting - focusing on one of the most secure browser around, Google Chrome!
He’ll tell you a tale of his experiences with bounty programs and how those have evolved since he started way back (vendors can show the love too!) and how he’s managed to survive in the harsh environment of browser bug hunting. He’ll impart some important bug hunting social skills by showing you how and how NOT to step on the others guys toes - very competitive cottage industry is browser bug hunting. ;)
Atte is also going to share with you how and why he selected his current target feature *(still full of bugs!), how he built his fuzzer-module(s) and the results produced. We’ll all walk a mile in a bug hunters shoes together and take a peek at the tool sets, as well as the infrastructures that are used to find browser bugs by individuals and vendors!
44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kett...44CON
Just like drinking is not a game in Finland; neither is browser bug hunting - it’s serious business! Browser bugs have been supporting Atte Kettunen (@attekett) traditional Finnish way of living since late 2011 and he’s going to tell you all about how he has been living the dream browser bug hunting - focusing on one of the most secure browser around, Google Chrome!
He’ll tell you a tale of his experiences with bounty programs and how those have evolved since he started way back (vendors can show the love too!) and how he’s managed to survive in the harsh environment of browser bug hunting. He’ll impart some important bug hunting social skills by showing you how and how NOT to step on the others guys toes - very competitive cottage industry is browser bug hunting. ;)
Atte is also going to share with you how and why he selected his current target feature *(still full of bugs!), how he built his fuzzer-module(s) and the results produced. We’ll all walk a mile in a bug hunters shoes together and take a peek at the tool sets, as well as the infrastructures that are used to find browser bugs by individuals and vendors!
[Ruxcon Monthly Sydney 2011] Proprietary Protocols Reverse Engineering : Rese...Moabi.com
This presentation given in 2011 during the first Ruxcon Monthly (Ruxmon) Sydney focuses on proprietary protocols reverse engineering and vulnerability audits.
Kernel Recipes 2019 - Faster IO through io_uringAnne Nicolas
Since the dawn of time, Linux has had to make do with inferior IO interfaces. Native Linux AIO supports only a niche application class (O_DIRECT), and even for that use case, it’s far too slow for modern storage. This talk will detail io_uring, a modern IO interface for Linux, that’s both fully featured and performant.
Jens Axboe
Kernel Recipes 2019 - Formal modeling made easyAnne Nicolas
Modeling parts of Linux has become a recurring topic. For instance, the memory model, the model for PREEMPT_RT synchronization, and so on. But the term "formal model" causes panic for most of the developers. Mainly because of the complex notations and reasoning that involves formal languages. It seems to be a very theoretical thing, far from our day-by-day reality.
Believe me. Modeling can be more practical than you might guess!
This talk will discuss the challenges and benefits of modeling, based on the experience of developing the PREEMPT_RT model. It will present a methodology for modeling the Linux behavior as Finite-State Machines (automata), using terms that are very known by kernel developers: tracing events! With the particular focus on how to use models for the formal verification of Linux kernel, at runtime, with low overhead, and in many cases, without even modifying Linux kernel!
Daniel Bristot de Oliveira
University of Virginia
cs4414: Operating Systems
Rust Expressions and Higher-Order Procedures
How to Share a Processor
Non-Preemptive and Preemptive Multitasking
Kernel Timer Interrupt
Если нашлась одна ошибка — есть и другие. Один способ выявить «наследуемые» у...Positive Hack Days
Ведущий: Асука Накадзима (Asuka Nakajima)
Практика повторного использования исходного кода позволяет сократить расходы на разработку программного обеспечения. Тем не менее, если в оригинальном исходном коде кроется уязвимость, она будет перенесена и в новое приложение. Докладчик расскажет о необычном способе обнаружения «наследуемых» уязвимостей в бинарных файлах без необходимости обращаться к исходному коду или символьным файлам.
This presentation gives overview or our Ganeti deployment
There is related YouTube playlist (in Croatian) with presentations https://www.youtube.com/playlist?list=PLDMnMa3XBHD_K6Rl2FBe2CC-MS6mdTOrJ
In current era of exploitation it is coming more complex to develop even PoC for vulnerability, especially when it comes to more complicated one, like race conditions, sandbox escapes ...
And it seems that nowdays is still quite common write concept of exploitability for vendors, or even final code, in prehistoric way, and even using shellcoding.
We will show how vulnerability "design patterns" transform writing code, from current widespread form of magic black box, to developing software which breaks another one. We believe that developing is the way to go for boosting vulnerability research, for sake of security and your own time.
This presentation is based upon the tv show, Ramsey's Kitchen Nightmares. It was presented at the very first BarCamp Canberra on the 19th of April 2008.
More information (not quite a transcription) is available at http://www.ruthellison.com/2008/04/20/gordon-ramsay-a-guerrilla-ux-consultant/
Euro 2016 : Im Viertelfinale spielt Deutschland gegen Italien. Deutschland ist die Nummer 2 im europäischen E-Commerce, Italien hingegen auf Platz 6. Steht die Entscheidung schon fest? Wer macht das Rennen in unserer E-Commerce EM 2016?
[Ruxcon Monthly Sydney 2011] Proprietary Protocols Reverse Engineering : Rese...Moabi.com
This presentation given in 2011 during the first Ruxcon Monthly (Ruxmon) Sydney focuses on proprietary protocols reverse engineering and vulnerability audits.
Kernel Recipes 2019 - Faster IO through io_uringAnne Nicolas
Since the dawn of time, Linux has had to make do with inferior IO interfaces. Native Linux AIO supports only a niche application class (O_DIRECT), and even for that use case, it’s far too slow for modern storage. This talk will detail io_uring, a modern IO interface for Linux, that’s both fully featured and performant.
Jens Axboe
Kernel Recipes 2019 - Formal modeling made easyAnne Nicolas
Modeling parts of Linux has become a recurring topic. For instance, the memory model, the model for PREEMPT_RT synchronization, and so on. But the term "formal model" causes panic for most of the developers. Mainly because of the complex notations and reasoning that involves formal languages. It seems to be a very theoretical thing, far from our day-by-day reality.
Believe me. Modeling can be more practical than you might guess!
This talk will discuss the challenges and benefits of modeling, based on the experience of developing the PREEMPT_RT model. It will present a methodology for modeling the Linux behavior as Finite-State Machines (automata), using terms that are very known by kernel developers: tracing events! With the particular focus on how to use models for the formal verification of Linux kernel, at runtime, with low overhead, and in many cases, without even modifying Linux kernel!
Daniel Bristot de Oliveira
University of Virginia
cs4414: Operating Systems
Rust Expressions and Higher-Order Procedures
How to Share a Processor
Non-Preemptive and Preemptive Multitasking
Kernel Timer Interrupt
Если нашлась одна ошибка — есть и другие. Один способ выявить «наследуемые» у...Positive Hack Days
Ведущий: Асука Накадзима (Asuka Nakajima)
Практика повторного использования исходного кода позволяет сократить расходы на разработку программного обеспечения. Тем не менее, если в оригинальном исходном коде кроется уязвимость, она будет перенесена и в новое приложение. Докладчик расскажет о необычном способе обнаружения «наследуемых» уязвимостей в бинарных файлах без необходимости обращаться к исходному коду или символьным файлам.
This presentation gives overview or our Ganeti deployment
There is related YouTube playlist (in Croatian) with presentations https://www.youtube.com/playlist?list=PLDMnMa3XBHD_K6Rl2FBe2CC-MS6mdTOrJ
In current era of exploitation it is coming more complex to develop even PoC for vulnerability, especially when it comes to more complicated one, like race conditions, sandbox escapes ...
And it seems that nowdays is still quite common write concept of exploitability for vendors, or even final code, in prehistoric way, and even using shellcoding.
We will show how vulnerability "design patterns" transform writing code, from current widespread form of magic black box, to developing software which breaks another one. We believe that developing is the way to go for boosting vulnerability research, for sake of security and your own time.
This presentation is based upon the tv show, Ramsey's Kitchen Nightmares. It was presented at the very first BarCamp Canberra on the 19th of April 2008.
More information (not quite a transcription) is available at http://www.ruthellison.com/2008/04/20/gordon-ramsay-a-guerrilla-ux-consultant/
Euro 2016 : Im Viertelfinale spielt Deutschland gegen Italien. Deutschland ist die Nummer 2 im europäischen E-Commerce, Italien hingegen auf Platz 6. Steht die Entscheidung schon fest? Wer macht das Rennen in unserer E-Commerce EM 2016?
NetWorks is an unusual cooperation and collaboration of artists, art space, gallery and museum to not only present works of accomplished Rhode Island artists, but also to provide a window into their motivations.
Steelcon 2015 - 0wning the internet of trashinfodox
My presentation slides from Steelcon 2015 on "Owning the Internet of Trash", a presentation on exploitation of endemic vulnerabilities in the so called "internet of things", with a focus on finding vulnerabilities in, exploiting, and gaining persistent access to, routers and other such embedded devices.
This talk was recorded, a video will be linked soonish, and went over some basics of analysing firmware, hardware, and suchlike to find bugs in things and hack the planet!
Steelcon 2014 - Process Injection with Pythoninfodox
This is the slides to accompany the talk given by Darren Martyn at the Steelcon security conference in July 2014 about process injection using python.
Covers using Python to manipulate processes by injecting code on x86, x86_64, and ARMv7l platforms, and writing a stager that automatically detects what platform it is running on and intelligently decides which shellcode to inject, and via which method.
The Proof of Concept code is available at https://github.com/infodox/steelcon-python-injection
A story of how we went about packaging perl and all of the dependencies that our project has.
Where we were before, the chosen path, and the end result.
The pitfalls and a view on the pros and cons of the previous state of affairs versus the pros/cons of the end result.
All of Your Network Monitoring is (probably) Wrongice799
Monitorama 2016 talk about network monitoring covering topics like network device drivers, ethtool, and some interesting bugs/features.
For more information about monitoring and tuning the entire Linux network stack, see: blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/
Reid Wightman's presentation at AppSec DC 2012. Reid provides background and the lates on Digital Bond's Project Basecamp. New PLC exploit modules include a Stuxnet-type attack on the Modicon Quantum.
Presentation on how GRNET uses Ceph as a storage backend on its Cloud Computing services. Technical specs, lessons learned, future plans.
Presentation held at the 1st GEANT SIG-CISS Meeting in Amsterdam, 2017-09-25.
GRNET - Greek Research and Technology network is the state-owned Greek NREN.
Pcapy and dpkt - tcpdump on steroids - Ran Leibman - DevOpsDays Tel Aviv 2018DevOpsDays Tel Aviv
Tcpdump is awesome for debugging issues on the network layer. But sometime you want to do a bit more, like look into the application layers or do some aggregation. In this talk I’m going to show you how to use python together with the pcapy and dpkt modules to take tcpdump to the next level.
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
What’s Slowing Down Your Kafka Pipeline? With Ruizhe Cheng and Pete Stevenson...HostedbyConfluent
What’s Slowing Down Your Kafka Pipeline? With Ruizhe Cheng and Pete Stevenson | Current 2022
Imagine having access to metrics, events, and insights without code modification or application redeployment. Imagine visualizing delays and tracking down performance bottlenecks in your Kafka pipeline instantly with minimal performance overhead. In this session, we show all of this is possible with eBPF.
In a live demo, we will introduce an eBPF-based, always-on, CPU profiler to visualize what your Kafka applications are spending time on. We will analyze how much time the Kafka broker spends on handling different requests and responding to polling and how much time a Kafka consumer spends on polling the broker and processing the messages. Furthermore, we will see how to detect issues by measuring consumer lags in both offsets and seconds, and how to correlate the increasing consumer lag with the CPU flame graphs. We demonstrate how not only to detect issues quickly but also to pinpoint performance bottlenecks instantly in the Kafka pipeline: e.g. garbage collection and disk/network IO.
In addition, we will provide some unique insights with eBPF: e.g. topic-centric flow graphs, consumer rebalancing lags, and under-replicated partitions.
Collecting all the data with no instrumentation and low overhead is no easy task. we will conclude by revealing the magic of eBPF and discussing the design choices and technical challenges of our network traffic tracer and Java CPU profiler that empowered deep visibility into Kafka.
"WTF is Twisted? (or; owl amongst the ponies)" is a talk that introduces the Twisted asynchronous programming framework, how it works, and what uses it.
13. before we get this
horror show rolling
• kernels, drivers, glibc, and everything else
changes.
• code snips will differ from what you are
running on your machines.
• some things are simplified in the interest of
time.
14. bprobe
• boundary IPFIX flow meter
• collects flow data by sniffing packets with libpcap
• also collects low level NIC data from the driver
• packets tx/rx
• bytes tx/rx
• ethernet collisions
• ethernet errors
15. ethernet bonding (aka teaming)
• combine a group of physical NICs (eth0, eth1, ...)
into a single virtual device (bond0, bond1, ...).
• different modes
• active-passive
• round robin
• link aggregation
17. how does bonding work (on linux) ?
• at a high level...
• the bonding driver creates a “virtual device”
• when a packet is sent, bonding driver figures
out which physical NIC to transmit the packet
on.
• when a packet comes in, the NICs pass the
incoming packet up for the higher layers of the
network stack to figure out.
18. bprobe and bonding
• bprobe discovers bonded network
interfaces.
• uses libpcap to monitor the underlying
physical NICs instead of bond devices.
• detecting link failures, etc
22. Bug was filed...
• Debian Lenny, 64bit.
• Bonded ethernet interfaces.
•No incoming packets are showing up.
23. Step 0
•Take a step back.
•Breathe.
•Do not break the computer.
24. Step 1
• Examine our assumptions:
• The packets are making it to the kernel.
• The packets are being handed up from the
kernel to libpcap.
• libpcap doesn’t lose any packets before
bprobe examines them.
• bprobe has some weird bug in it.
32. Peel some layers away
• bprobe is really libpcap + packet analysis +
output.
• if this is a bug in the kernel or libpcap then
other programs that use libpcap (like
tcpdump) will also fail the same way.
• so, do they?
33.
34. tcpdump
• bonded ethernet interfaces (on linux) are virtual
devices created by combining other devices.
• for example:
• bond0
• eth0
• eth2
• eth4
• ...
35. First, sniff bond0...
% sudo tcpdump -i bond0 dst 172.16.209.136 and proto 1
12:57:26.275660 IP 172.16.209.1 > 172.16.209.136: ICMP
echo request, id 62831, seq 54, length 64
12:57:27.275731 IP 172.16.209.1 > 172.16.209.136: ICMP
echo request, id 62831, seq 55, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
37. Now eth0 (the active NIC in
bond0)
% sudo tcpdump -i eth0 dst
172.16.209.136 and proto 1
^C
0 packets captured
2 packets received by filter
0 packets dropped by kernel
51. Steps 3-5
• Dig until you see something you haven’t
seen before.
• Read all of the code and understand it.
• Go to step 2.
52. how are packets received?
• packets come in from the wire.
• a couple different ways for the kernel to
“know” about new packets.
• let’s just look at the simple case.
• an interrupt is raised when a packet arrives.
• both paths hand data up to the higher
layers in similar ways.
66. packet protocol family
(in the kernel)
libpcap
(in userland)
bprobe/tcpdump/etc
(in userland)
network device agnostic layer
(in the kernel)
67. libpcap (userland)
• creates a socket of type PF_PACKET
• two ways to get get packets from the kernel:
• one by one (slow)
• via shared memory (fast)
• libpcap tries to use the fast method
• if it fails, it falls back to slow.
73. packet protocol family
(in the kernel)
libpcap
(in userland)
bprobe/tcpdump/etc
(in userland)
network device agnostic layer
(in the kernel)
74. PF_PACKET (kernel)
• libpcap creates the PF_PACKET socket
• the PF_PACKET code in the kernel
(eventually) executes.
• this code does some initialization and
inserts a protocol hook...
75.
76. packet protocol family
(in the kernel)
libpcap
(in userland)
bprobe/tcpdump/etc
(in userland)
network device agnostic layer
(in the kernel)
77. network device agnostic layer
• pulls packets off the backlog queue.
• calls netif_receive_skb()
• has some logic to determine who the real
sender is when bonding is enabled.
• passes the packet through the protocol
hooks.
83. we now know the path packets take
so they can be examined by pcap apps.
84. packet protocol family
(in the kernel)
libpcap
(in userland)
bprobe/tcpdump/etc
(in userland)
network device agnostic layer
(in the kernel)
85.
86.
87. back to the bug
• so, the bug was that packeting sniffing
physical NICs on bonded hosts was not
revealing incoming packets.
• what do we now know about our
environment?
• what would be the best place to look to
track down this bug?
97. Bug
• We overwrite the packet’s device with the bond
device.
• The protocol hook check, checks to see if the hook is
for the device on the packet.
• It isn’t
• we are sniffing eth0
• skb->dev was overwritten to bond0.
• That’s why if you sniff “bond0” you see packets but if
you sniff “eth0” you see nothing.
107. Now eth0 (the active NIC in
bond0)
% sudo tcpdump -i eth0 dst
172.16.209.136 and proto 1
^C
0 packets captured
2 packets received by filter
0 packets dropped by kernel
121. Step 0
•Take a step back.
•Breathe.
•Do not break the computer.
122. Step 1
• Examine our assumptions:
• The kernel code is still broken.
• The incoming packets are being queued up for
libpcap to pull out of PF_PACKET properly.
• There probably isn’t bug in bProbe and
tcpdump.
126. i used apt-get source to
retrieve the official source for
debian lenny’s libpcap and I
found something
surprising.
127. old way of doing pcap
• debian lenny’s kernel supports the new way
of getting packets out of the kernel via
mmap.
•but, debian lenny’s libpcap is not new
enough and therefore uses the old way
to examine packets.
•this also means that unless i statically link
the libpcap version i want, my app will just
perform worse on lenny.
130. that if statement fails.
• we are sniffing packets on a physical device
• BUT in the kernel we are changing the
device a packet comes in on to the bond
device (remember in netif_receive_skb?)
131.
132. that if statement fails.
• the index of the bond device is different from
the index of the physical device we are sniffing
• so this if statement evaluates to TRUE
• libpcap returns without processing
the packet.
133. why?
this code exists to prevent a race condition
when sniffing packets the old way in some
kernels.
134. solution
• boot into our fixed debian lenny kernel.
• download a version of libpcap that is newer and
supports the mmap method for packet sniffing.
• new method doesn’t have this race condition
and has better performance.
• link bprobe/tcpdump/other pcap apps against it.
135. First, sniff bond0...
% sudo tcpdump -i bond0 dst 172.16.209.136 and proto 1
12:57:26.275660 IP 172.16.209.1 > 172.16.209.136: ICMP
echo request, id 62831, seq 54, length 64
12:57:27.275731 IP 172.16.209.1 > 172.16.209.136: ICMP
echo request, id 62831, seq 55, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
136. Next, sniff eth0...
% sudo tcpdump -i eth0 dst 172.16.209.136 and proto 1
12:57:26.275660 IP 172.16.209.1 > 172.16.209.136: ICMP
echo request, id 62831, seq 54, length 64
12:57:27.275731 IP 172.16.209.1 > 172.16.209.136: ICMP
echo request, id 62831, seq 55, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
140. summarize
• kernel bug when overwriting the device the
packet arrived on.
• fixed this bug, but bprobe/tcpdump still
failed.
• libpcap bug when pulling packets out the
kernel the old way
• can avoid this bug and get better
performance with a newer libpcap
141. Step 0
•Take a step back.
•Breathe.
•Do not break the computer.
142. Step 1-5
• Examine your assumptions.
• Start digging.
• Keep going until you see something you
haven’t seen before.
• Read all of the code and understand it.
• Go to step 2.
159. W A T
• We have 2 programs:
• Both link against libraries in /usr/local/lib/
• Only one works.
• The broken program’s library is in /usr/local/lib/
160.
161. Step 0
•Take a step back.
•Breathe.
•Do not break the computer.
162. Step 1
• Examine our assumptions:
• The programs and libraries are both 64bit.
• /usr/local/lib/ is in the library search path
174. Strange
• This is confusing.
• bprobe should fail.
• But, the shared libraries a particular binary
dynamically links to at runtime are built
into the binary itself.
• So....
183. ah ha!
• bprobe works and can link because the
binary is storing the library path inside of
itself.
• but, now there are 2 more questions:
• how did the rpath tag get there?
• why doesn’t ipfix_reader have one?