Lord of the Rings• x86 processor has 4 layers of protection called Ring 0 – 3.• Privilege code (Kernel ) runs in Ring 0. Processor ensure that privilege instructions (like enable/disable interrupt, ) execute in kernel mode only.• User application runs in Ring 3.• Ring 1 is where the Hyperviser lives..
How system call works• Cannot directly enter kernel space using jmp or a call instruction.• When make a system call (like CreateFile, ReadFile) OS enter kernel mode (Ring 0) using instruction int 2E (it is called interrupt gate).• Code segment descriptor contain information about the ‘Ring’ at which the code can run. For kernel mode modules it will be always Ring 0. If a user mode program try to do ‘jmp <kernel mode address>’ it will cause access violation, because of the segment descriptor flag says processor should be in Ring 0.• The frequency of entering kernel mode is high (most of the Windows API call cause to enter kernel mode) sysenter is the new optimized instruction to enter kernel mode.
System Call continued..• Windows maintains a system service dispatch table which is similar to the IDT. Each entry in system service table point to kernel mode system call routine.• The int 2E probe and copy parameters from user mode stack to thread’s kernel mode stack and fetch and execute the correct system call procedure from the system service table.• There are multiple system service tables. One table for NT Native APIs, one table for IIS and GDI etc.
Lets try it in WinDBG..• NtWriteFile: mov eax, 0x0E ; build 2195 system service number for NtWriteFile mov ebx, esp ; point to parameters int 0x2E ; execute system service trap ret 0x2C ; pop parameters off stack and return to caller
Software Interrupt Request Levels (IRQLs)• Windows has its own interrupt priority schemes know as IRQL.• IRQL levels from 0 to 31, the higher the number means higher priority interrupt level.• HAL map hardware interrupts to IRQL 3 (Device 1) - IRQL 31 (High)• When higher priority interrupt occur, it mask the all lower interrupts and execute the ISR for the higher interrupt.• After executing the ISR, kernel lower the interrupt levels and execute the lower interrupt ISR.• ISR routine should do minimal work and it should defer the major chunk of work to Deferred Procedure Call (DPC) which run at lower IRQL 2.
IRQL and DPC• DPC concept is similar to other OS, in Linux it is called bottom half.• DPC is per processor, means a duel processor SMP box contains two DPC Qs.• The ISR routine generally fetch data from hardware and queue a DPC for further processing.• IRQL priority is different from thread scheduling priority.
IRQL and DPC• The scheduler (dispatcher) also runs at IRQL 2.• So a code that execute on or above IRQL 2(dispatch level) cannot preempt.• From the Diagram, see only hardware interrupts and some higher priority interrupts like clock, power fail are above IRQL 2.• Most of the time OS will be in IRQL 0(Passive level)• All user programs and most of the kernel code execute on Passive level only.
IRQL continued..• Scheduler runs at IRQL 2, so what happen if my driver try to wait on or above dispatch level ?.• Simple system will crash with ‘Blue Screen’, usually with the bug check ID IRQL_NOT_LESSTHAN_EQUAL.• Because if wait above dispatch level, no one there to come and switch the thread.• What happen if try to access a PagedPool in above dispatch level ?.• If the pages are on disk, then a page fault exception will happen, the current thread need to wait and page fault handler will read the pages from page file to page frames in memory.• If page fault happen above the dispatch level, no one there to stop the current thread and schedule the page fault handler. Thus cannot access PagedPool on or above dispatch level.
IRQL 1 - APCs• Asynchronous Procedure Call (APC) run at IRQL 1.• The main duty of APC is to send the data to user thread context.• APC Q is thread specific, each thread has its own APC Q.• User space thread initiate the read operation from a device and either it wait to finish it or continue with another job.• The IO may finish sometime later, now the buffer need to send to the calling thread’s process context. It is the duty of APC.
App issue ReadFile NtReadFile User Land Kernel LandIO Manager IO Mgr create IRP Packet, IRP send to driver stack File System Volume Manager Disk Class Driver Hardware Driver
What is IO Request Packet (IRP)• IO Operation passes thru, – Different stages. – Different threads. – Different drivers.• IRP Encapsulate the IO request.• IRP is thread independent.
IO Request Packet (IRP)• When a thread initiate an IO operation, IO Manager create a data structure call IO Request Packet (IRP).• The IRP contains all information about the request.• IO Manager send the IRP to the top device in the driver stack.• Demo : !irpfind to see all current IRPs. Demo : !irp <irp address> to see information about one IRP.
IRP Continued..• Compare IRP with Windows Messages -MSG structure.• Each driver in the stack do its own task, finally forward the IRP to the lower driver in the stack.• IRP can be processed synchronously or asynchronously.
IRP Continued..• Usually lower level hardware driver takes more time. H/W driver can mark the IRP for pending and return.• When H/W finish IO, H/W driver complete the IRP by calling IoCompleteRequest().• IoCompleteRequest() call IO completion routine set by drivers in stack and complete the IO.
Structure of IRP• Fixed IRP Header IRP Header• Variable Stack locations – – One sub stack per driver Stack Location 1 Stack Location 2 Stack Location 3 Stack Location N
Flow of IRP IRP for Storage Stack Storage Stack IRP Header File System Stack Location 1 Volume Manager Stack Location 2 Disk Class Driver Stack Location 3 Hardware Driver Stack Location 4Forward IRP to lowerdriver in the stack
Flow of IRP Completion IRP for Storage Stack Storage Stack IRP Header File System – Stack Location 1 Completion Routine Volume Manager – Stack Location 2 Completion Routine Disk Class Driver – Stack Location 3 Completion Routine Hardware Driver – Stack Location 4 Complete the IRPCall the completion routine whilecompleting the IRP
IRP Header• IO buffer Information.• Flags – Page IO Flag – No Caching IO flag• IO Status – On Completion set this to IO Completed.• IRP cancel routine
IRP Stack Location• IO Manager get the driver count in the stack from the top device in the stack.• While creating IRP, IO manager allocate the IO stack locations equal to the device count from the top device object.
Contents of IO Stack Location• IO Completion routine specific to the driver.• File object specific to the request.
Asynchronous IO• CreateFile(…, FILE_FLAG_OVERLAPPED ,..), ReadFile(.., LPOVERLAPPED)• When complete the IO operation, IO Mgr signal the EVENT in LPOVERLAPPED.
How Async IO work in Kernel• Lower layer driver complete IRP in arbitrary thread context.• IO Mgr call IO Completion routine in reverse order.• If operation is Async, IO Mgr queue an APC specific to the initiator thread.• This APC has complete info of buffer, size info.• This APC get executed later in the context of initiator thread, which copy buffer to user space, trigger the event set by App.
Common issues related IRP• After forward the IRP down, don’t touch it (except from IO completion routine).• If lower driver mark the IRP for pending, all top layer driver should do the same.• If a middle level driver need to keep the IRP for further processing after completing it by lower driver, it can return STATUS_MORE_PROCESSING REQUIRED from completion routine.• Middle layer driver should complete it later.• See ReactOS source code (instead of reading 20 page doc)• FastIO - Concepts
Locality Theory• If access page/cluster n, high possibility to access blocks near to n.• All memory based computing system working on this principle.• Windows has registry keys to configure pre-fetch how many blocks/pages.• Application specific memory manager like Databases, multimedia workload, have application aware pre-fetching.
Virtual Memory Manager (VMM)• Apps feels memory is unlimited – magic done by VMM.• Multiple apps run concurrently with out interfering other apps data.• Apps feel the entire resource is mine.• Protect OS memory from apps.• Advanced app may need to share memory. Provide solution to memory sharing easily.
VMM Continued..• VMM reserve certain amount of memory to Kernel.• 32 bit box , 2GB for Kernel and 2GB for User apps.• Specific area in Kernel memory reserved to store process specific data like PDE, PTE etc called Hyper Space
Segmentation and Paging• X86 processor has segmentation and paging support.• Can disable or enable paging, but segmentation is enabled by default.• Windows uses paging.• Since not able to disable segmentation, it consider the entire memory for segments (also called ‘flat segments’).
Paging• Divide entire physical memory in to equal size pages (4K size for x86 platforms). This is called ‘page frames’ and list called ‘page frame database’ (PF DB).• PF DB also contains flags stating, read/write underway , shared page , etc.
VMM Continued..• Upper 2GB Kernel space is common for all process.• What is it mean – Half of PDE is common to all process !.• Experiment – See the PDE of two process and make sure half of the PDE is same
Physical to Virtual address translation• Address translation in both direction – When write PF to pagefile, VMM need to update proper PDE/PTE stating page is in disk.• Done by – Memory Management Unit (MMU) of the processor. – The VMM help MMU.• VMM keep the PDE/PTE info and pass to MMU during process context switch.• MMU translate virtual address to physical address.• CR3 register
Translation Lookaside Buffer (TLB)• Address translation is costly operation• It happen frequently – when even touches virtual memory.• TLB keeps a list containing most frequent address translations.• The list is tagged by process ID.• TLB is a generic OS concept - implementation is architecture dependent.• Before doing the address translation MMU search TLB for the PF.
Address Translation• In x86 32 bit address – 10 bits of MSB points to the PTE offset in PDE. Thus PDE size of process is 1024 bytes.• Next 10 bits point to the PF starting address in PTE. Thus each PTE contains 1024 bytes.• Remaining 12 bits to address the location in the PF. Thus page size is 4K.
What is a Zero Page• Page frames not specific to apps.• If App1 write sensitive data to PF1, and later VMM push the page to page file, attach PF 1 to App2. App2 can see these sensitive info.• It’s a big security flaw, VMM keep a Zero Page list.• Cannot clean the page while freeing memory – it’s a performance problem.• VMM has dedicated thread who activate when system under low memory situation and pick page frames from free PF list, clean it and push to zero page list.• VMM allocate memory from zero page list.
Arbitrary Thread Context• Top layer of the driver stack get the request (IRP) in the same process context.• Middle or lower layer driver MAY get the request in any thread context (Ex: IO completion), the current running thread context.• The address in the IRP is specific to the PDE/PTE in the original process context.
Arbitrary Thread Context continued..• How to solve the issue ?.• Note the half of the PDE (Kernel area) is common in all process.• If some how map to the kernel memory (Upper half of PDE), the buffer is accessible from all process.
Mapping buffer to Kernel space• Allocate kernel pool from the calling process context, copy user buffer to this Kernel space.• Memory Descriptor List (MDL) – Most commonly used mechanism to keep data in Kernel space.
Standby list• To reclaim pages from a process, VMM first move pages to Standby list.• VMM keep it there for a pre-defined ticks.• If process refer the same page, VMM remove from standby list and assign to process.• VMM free the pages from Standby list after the timeout expire.• Pages in standby list is not free, not belong to a process also.• VMM keep a min and max value for free and standby page count. If its out of the limits, appropriate events will signaled and adjust the appropriate lists.
Miscellaneous VMM Terms• Paged Pool• Non Paged Pool• Copy on write (COW)
Cache Manager concepts• If disk heads run in the speed of super sonic jets, Cache Manager not required.• Disk access is the main bottleneck that reduce the system performance. Faster CPU and Memory, but disk is still in stone age.• Common concept in Operating Systems, Unix flavor called ‘buffer cache’.
What Cache Manager does• Keep the system wide cache data of frequently used secondary storage blocks.• Facilitate read ahead , write back to improve the overall system performance.• With write-back, cache manager combine multiple write requests and issue single write request to improve performance. There is a risk associated with write-back.
How Cache Manager works• Cache Manager implement caching using Memory Mapping.• The concept is similar to an App uses memory mapped file.• CreateFile(…dwFlagsAndAttributes ,..)• dwFlagsAndAttributes == FILE_FLAG_NO_BUFFERING means I don’t want cache manager.
How Cache Manager works..• Cache Manager reserve area in higher 2GB (x86 platform) system area.• The Cache Manager reserved page count adjust according to the system memory requirement.• If system has lots of IO intensive tasks, system dynamically increase the cache size.• If system under low memory situation, reduce the buffer cache size.
How cached read operation works User Space Kernel SpaceCached Read (1) Page Fault (4) VMM Do Memory Mapping (3) Get the Pages From Cache Manager File System CM (2) Get the blocks from disk (5) Disk stack (SCSI/Fibre Channel)
How cached write operation works User Space Kernel SpaceCached Write (1) Modified Page Writer Thread VMM of VMM Write to disk Do Memory Mapping (3), later(4) Copy data to VMM pages. Copy Pages to CM (2) Cache Manager File System Write the blocks to disk (5) Disk stack (SCSI/Fibre Channel)