Understanding the virtual memory - Ixia Connect #2

2
• Intel 8086
• Intel 80286
• Intel 80386
• Memory Management Unit
• Meltdown
• Malloc
• IOMMU
O V E R V I E W

4
Memory BUS
20 address lines (wires)
2^20 = 1MB addressable RAM

5
1 MB
of
RAM
0x0
0x10.0000
int main()
{
int *a = malloc();
...
}
Memory BUS

6
• No isolation between applications, as each application was accessing the physical
memory directly
Directly accessing
the RAM

8
Memory
access
Memory
access
• First attempt to use Virtual Memory through a Memory Management Unit
• A Protected Mode was just added, using different memory Segments for each
application
• Not widely adopted because of problems in backwards compatibility with 8086

10
MMU access
Physical
access
• Different from 80286: introduction of Paging support in the MMU

11
MMU access
Physical
access
Hey, I got 4 GB of RAM!
I’m all yours.
1 MB
of
RAM
4 GB
of
RAM
32-bit Address Bus,
accessing max. 4GB
of RAM
True Virtualization:
hiding the hardware,
overcommitting

12
Virtual
Memory
access
1 MB
of
RAM4 GB
of
RAM
Physical
Memory
access
int main()
{
int *a = malloc();
...
}
0x0
0x10.0000
0x0
0x1.0000.0000

14
RAM
1 MB
Page 0
Page 1
.
.
.
.
.
.
.
.
Page 256
• The memory is split into pages:
• By default 4KB pages
• 64 KB
• 2 MB (Huge Pages)
• 1GB (Huge Pages)
Using the default 4KB
pages, the RAM will be
split in 256 pages

15
RAM
Virtual Memory
CPU
0 GB
4 GB
0 MB
1 MB

16
RAM
1
4
2
3
CPU
MMU
Process A
Virtual Memory
1
2
3
4
• One advantage is that the application
sees a contiguous portion of memory
(e.g. Process A, pages 3 and 4)
for (int *i = 0x1000; i < 0x3000; i++)
*i = 0;
0x1000
0x3000

17
CPU RAM
Process A
Virtual Memory
1
2
3
4
Process B
Virtual Memory
1
2
3
1
3
2
4
1
2
3
MMU

18
RAM
1
Process A Page Table
4
2
3
Process A Page Directory
32 bit Virtual Address
4KB
(max.
1024
entries)
Page Directory
4KB
(max.
1024
entries)
Page Table
CPU
The first 10 bits point to an
offset within the PD
10 bits means 1024 values
The PD entry will point to a
Page Table
The last 12 bits point to the
offset inside the Process’
page
The CPU makes a
read/write to a 32-bit
address
The next 10 bits will point to
an offset within the
Page Table Entry

19
Page Directory Page Table
CPU
Notice the 2 additional
memory accesses.
First
When cloning of a
process happens (a
new thread), it is
enough to copy the
entries in the PD to the
new PD
RAM
1
4
2
3
The third one actually
gets the desired data.
Second
The bottom entries
point to lower Virtual
Memory (0GB +)
The upper entries
point to higher Virtual
Memory (4GB)

20
Page Global Directory
(PGD)
Page Middle Directory
(PMD)
RAM
1
4
2
3
In the Linux kernel, we
have the following
terms:
PGD, PMD, PTE Page Table Entry
PTE

21
(PGD)
Page Middle Directory
(PMD)
PGD
RAM
1
4
2
3
How the kernel stores
information about a
process
Page Table Entry
PTE
task_struct
Each process has a
pointer to a PGD

22
The CR3 keeps the physical
memory address, so that the
MMU can use it directly
Tricky question:
Why sometimes the CR3
value is higher than the
RAM size?
The CR3 (Control Register 3)
stores the PGD pointer

23
RAM
1
3
2
4
1
2
3
• Each process has its own set of page
tables (also stored in RAM)
CPU MMU
Process A
Virtual Memory
1
2
3
4
CR3
Process A - PGD

24
RAM
Process B
Virtual Memory
1
2
3
1
3
2
4
1
2
3
CPU MMU
Process B - PGD
Process A - PGD
CR3
Process B

25
/proc/meminfo
Total physical
memory (1GB)
The amount of RAM
consumed to keep
the page tables of all
processes

28
Process A
Virtual Memory
1
2
3
4
Process B
Virtual Memory
1
2
3
1
2
3
1
2
3
The user address space
of a process should be
in the 0GB-3GB range
The kernel’s pages,
mapped in each
user-space process

29
1024
entries
Process A
Virtual Memory
1
2
3
4
Process B
Virtual Memory
1
2
3
1
2
3
1
2
3
RAM
1
3
2
4
1
2
3
Process B - PGD
Process A - PGD
KERNEL - PGD
MMU
A single Page Table
for the Kernel
256
entries
768
entries

30
The Kernel’s Page
Table mostly use Huge
Pages

32
Process A
Virtual Memory
1
2
3
4
1
2
3
int main()
{
int *a = 0xC000.0000;
printf(“%dn”, *a);
}
The MMU makes the
translation and detects that
the Pages have the System
bit set, but the Instruction
Pointer is in user-space
MMU
The program will receive
SIGSEGV but can handle the
signal and can continue its
execution
The MMU sends an
Exception to the CPU
and the program
receives SIGSEGV
CPU
The MMU already
forwarded the data
to the CPU’s cache
CPU cache
Using a side-channel
attack on the cache, the
application can read the
kernel memory data
A pointer to
somewhere in the
kernel address
space

33
Process A
Virtual Memory
1
2
3
4
1
2
3
User Program connect()
System Call User Space
Kernel Space
Kernel
During the System Call, the
CR3 does not change, as
the Page Table already
contains the mapping
required by the kernel to
execute its code and
access its data.

34
User Program connect()
System Call User Space
Kernel Space
Kernel
Process A
Virtual Memory
(in kernel-space)
1
2
3
4
1
2
3
Process A
Virtual Memory
(in user-space)
1
2
3
4
1
2
3
A very very small kernel
region is still mapped at
each fork() in the new
process Page Table
(VDSO, vSyscall)
After Meltdown, at each
System Call, Interrupt or
Exception, the CR3 register
is changed to point to the
kernel’s Page Table
The cost for this protection?
A new page to store
Process A’s kernel’s PGD
But the memory
consumption is not the
problem.
Flushing the TLB cache
when changing the CR3
takes a big penalty.
After Meltdown, the kernel
manages two sets of PGD
for each process

36
vma_struct task_struct
vma_struct
vma_struct
vma_struct
vma_struct
The Linux kernel
object to represent a
process
The Linux kernel
object to represent a
virtual memory area
(start, end)

37
vma_struct task_struct
vma_struct
vma_struct
vma_struct
vma_struct
A mmap() call will
allocate a new VMA
and link it in the list
MMAP() call here
vma_struct
PGD
The first page of the
Virtual Address space is
intentionally left
unmapped, to catch
NULL pointers
SegFault
Minor fault, the
PGD/PMD/PTE will be
updated.
When cloning a new
process, the list is
iterated and copied to
the new process (except
for the Stack vma_struct)

38
• Malloc() will request the kernel to increase the
Heap area region, using mmap() or brk() system
calls
• After a malloc() call, the Page Tables are not yet
updated
• Only after the first read/write operation, the Page
Tables are updated and a physical page is
assigned (Minor Page Faults)
• If the page was swapped to disk, the kernel can
bring it back in RAM (Major Page Faults)
• If a read/write request with an address in
between the Stack and the Heap will result in a
Segmentation Fault (caught by the MMU
because the Page Tables do not contain the
requested page)

39
The /proc/PID/maps
prints info about each
VMA region
132KB virtual size
12KB physical size
RSS = Resident Set Size

40
• It doesn’t fail (or rarely fails)
• As with any virtualization technique, the Virtual
Memory overcommits
• When running low on physical memory (RAM),
the application will most likely SIGSEGV or the
OOM (Out of Memory) killer starts stopping
processes
Best setting not to
overcommit memory
Make your process less
likely to be picked by the
OOM killer

41
How much VM is each
process using (in KB)
How much RAM is
actually used (in KB)
How many Page Table
Entries are used for each
process

43
• Similar to how processes
are isolated between
each other by the MMU,
in the same way multiple
Devices can be isolated
• Page table walks must
be performed when the
Devices read/writes to a
Virtual Memory address
• Example: SR-IOV lets
Virtual Machines see a
dedicated Network Card

Understanding the virtual memory - Ixia Connect #2

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Understanding the virtual memory - Ixia Connect #2

Similar to Understanding the virtual memory - Ixia Connect #2 (20)

Recently uploaded

Recently uploaded (20)

Understanding the virtual memory - Ixia Connect #2