Failure cause analysis for MMAP issue with multiple process in Virtual Machines

1. MMAP failure with DPDK secondary application

2. Problem Overview
• During lab run; occasional MMAP failure is been observed.
• Primary is modeled for main packet processing. While secondary
handles configuration and special packet processing.
• Applications should run inside Virtual Machines with Address Space
Layout Randomization enabled.
• “EAL: Could not mmap <n> bytes in /dev/zero at [0x7fcedbc00000], got
[0x7fcea1800000] - please use '--base-virtaddr' option“.

3. What is ASLR?
• Address Space Layout Randomization (ASLR) is an exploit mitigation technique implemented in the majority of
modern operating systems. The idea behind ASLR is randomizing the process’ memory space in order to prevent
the attacker from finding the addresses of functions or gadgets (s)he might require to successfully complete the
exploit. Linux introduced ASLR with kernel 2.6.12 in 2015. ASLR can be configured in Linux using the
“/proc/sys/kernel/randomize_va_space” interface.
• The code segment (or text segment; .text) of the main binary is located at random locations only if the executable
has been compiled as a Position Independent Executable (PIE). A position independent executable is compiled in
such a way that can be located anywhere in memory and still execute properly without modification. This is
achieved through the use of PC relative addresses instead of absolute addresses. All shared objects (.so, libraries)
are compiled as PIE as it’s mandatory for them to work, thus they’re always at random memory addresses when
ASLR is enabled.
• Based on the above paragraph, we can assume that Linux executables not compiled as PIE are not effectively
protected by ASLR, even though it might be set to 2 (Full Randomization). The attacker could leverage the .text
segment, and other areas located within the main executable, such as GOT/PLT to build a successful exploit
against a non-PIE executable on a system with ASLR enabled. As a result, any non-PIE executable leaves the door
open to return-2-plt/GOT dereferencing and ROP attacks.

4. ASLR Overview
• Check library is randomized ldd <executables>
• Check text is randomized objdmp or return __builtin_return_address(0)-0x5;
• use of PIE is not widely embraced by the above Linux versions. 82.82% and 89.7% of binaries are not
effectively protected by ASLR in Linux systems.
• For libc randomization: for x in {1..5}; do grep 'r-xp .*/libc' /proc/self/maps; done

5. DPDK Multi-Process Overview
• DPDK processes running as a single application and using shared memory must have distinct core mask arguments. It is not possible to have a
primary and secondary instance, or two secondary instances, using any of the same logical cores. Attempting to do so can cause corruption of
memory pool caches, among other issues. The potential issues are caused by a dependence on the lcore_id internally by Intel DPDK data
structures, especially mempools. If two processes use the same lcore they will have the same lcore_id value, and will try and access the same
mempool cache which is not thread-safe. This will cause mempool corruption.
• NOTE: this applies only to co-operating processes, i.e. those run as primary and secondary processes. There are no mempool issues with running completely independent Intel DPDK
processes on the same cores, i.e. processes run using different "--file-prefix=" parameters, since those do not share any memory and data structures.
• Sample program multi_process:
• simple_mp & symmetric_mp proces is one binary bifurcated by proc_type).
• mp_ client/server: The server process performs the network port and data structure initialization much as the symmetric multi-process
application does when run as primary. Port configuration data in a memory zone in hugepage shared memory. In the same way that the server
process is designed to be run as a primary process instance only, the client processes are designed to be run as secondary instances only. There
are handles to all needed rings and memory pools are obtained via calls to rte_ring_lookup() and rte_mempool_lookup().
• Master-slave Multi-process : The master process calls the rte_eal_mp_remote_launch() EAL function to launch an application function for each
pinned thread through the pipe. Then, it waits to check if any slave processes have exited. If so, the process tries to re-initialize the resources that
belong to that slave and launch them in the pinned thread entry again.

6. DPDK Multi-Process Overview
• Deployment Models
• 1) Symmetric/Peer Processes: to create a set of peer processes where each process performs the same workload. This model is equivalent to
having multiple threads each running the same main-loop function
• 2) Asymmetric/Non-Peer Processes: have a single primary process instance that acts as a load-balancer or server distributing received packets
among worker or client threads, which are run as secondary processes. In this case, extensive use of rte_ring objects is made, which are located in
shared hugepage memory.
• Multi-process Limitations:
• 1) The multi-process feature requires that the exact same hugepage memory mappings be present in all applications. The Linux security feature -
Address-Space Layout Randomization (ASLR) can interfere with this mapping, so it may be necessary to disable this feature in order to reliably run
multi-process applications.
• 2) All DPDK processes running as a single application and using shared memory must have distinct coremask/corelist arguments. Attempting to do
so can cause corruption of memory pool caches, among other issues.
• 3) The delivery of interrupts, such as Ethernet* device link status interrupts, do not work in secondary processes.
• 4) The use of function pointers between multiple processes running based of different compiled binaries is not supported, since the location of a
given function in one process may be different to its location in a second.

7. Analysis
• In Primary Parameter, "--base-virtaddr“ can not be fixed “Address“; it varies from system to system.
• A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses
chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary
process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address.
The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base
address. But the virtual address of pci resources mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot
mmap"
- Disabling ASLR by adding those two lines to "/etc/sysctl.conf": # Disable Address Space Layout Randomization (ASLR) (needed by DPDK)
kernel.randomize_va_space = 0 is not a option.
- Getting virtual address of the first (the one with the minimum address value) memory segment returned from the function
"rte_eal_get_physmem_layout ()", called from a "dummy" primary application used only to get this address.
- - Passing the above virtual address as a parameter for the "real" primary application using the " --base-virtaddr= " dpdk command line option.
When secondary app starts, it all goes well with the specified base address.

8. Summary & Recommendation
1.Core mask has to be unique for Primary and Secondary – this can be done by using dummy
rte_eal_init to fetch system parameters (sample code is already shared).
2.Primary and Secondary share and inherit all shared libraries and dynamic linked libraries alike –
compile flags and code analysis can reveal the additional libraries.
3.Find the correct offset and pass value to primary – this can be done by using dummy rte_eal_init to
fetch the first huge page virtual address (sample code is already shared).
4.Make use of PIC flag for building PIE code base.