Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Dave Probert, Ph.D. - Windows Kernel Architect Microsoft Windows Division Copyright Microsoft Corporation
  2. 2. About Me <ul><li>Ph.D. in Computer Engineering (Operating Systems w/o Kernels) </li></ul><ul><li>Kernel Architect at Microsoft for over 13 years </li></ul><ul><ul><li>Managed platform-independent kernel development in Win2K/XP </li></ul></ul><ul><ul><li>Working on multi-core & heterogeneous parallel computing support </li></ul></ul><ul><ul><ul><li>Architect for UMS in Windows 7 / Windows Server 2008 R2 </li></ul></ul></ul><ul><li>Co-instigator of the Windows Academic Program </li></ul><ul><ul><li>Providing kernel source and curriculum materials to universities </li></ul></ul><ul><ul><li>http://microsoft.com/WindowsAcademic or [email_address] </li></ul></ul><ul><ul><li>Wrote the Windows material for leading OS textbooks </li></ul></ul><ul><ul><ul><li>Tanenbaum, Silberschatz, Stallings </li></ul></ul></ul><ul><ul><ul><li>Consulted on others, including a successful OS textbook in China </li></ul></ul></ul>
  3. 3. UNIX vs NT Design Environments Copyright Microsoft Corporation Environment which influenced fundamental design decisions UNIX [1969] Windows (NT) [1989] 16-bit program address space Kbytes of physical memory Swapping system with memory mapping Kbytes of disk, fixed disks Uniprocessor State-machine based I/O devices Standalone interactive systems Small number of friendly users 32-bit program address space Mbytes of physical memory Virtual memory Mbytes of disk, removable disks Multiprocessor (4-way) Micro-controller based I/O devices Client/Server distributed computing Large, diverse user populations
  4. 4. Effect on OS Design Copyright Microsoft Corporation NT vs UNIX Although both Windows and Linux have adapted to changes in the environment, the original design environments (i.e. in 1989 and 1969) heavily influenced the design choices: Unit of concurrency: Process creation: I/O: Namespace root: Security: Threads vs processes CreateProcess() vs fork() Async vs sync Virtual vs Filesystem ACLs vs uid/gid Addr space, uniproc Addr space, swapping Swapping, I/O devices Removable storage User populations
  5. 5. Today’s Environment [2009] Copyright Microsoft Corporation <ul><ul><ul><ul><ul><li>64-bit addresses </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>GBytes of physical memory </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>TBytes of rotational disk </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>New Storage hierarchies (SSDs) </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Hypervisors, virtual processors </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Multi-core/Many-core </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Heterogeneous CPU architectures, Fixed function hardware </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>High-speed internet/intranet, Web Services </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Media-rich applications </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Single user, but vulnerable to hackers worldwide </li></ul></ul></ul></ul></ul>Convergence: Smartphone / Netbook / Laptop / Desktop / TV / Web / Cloud
  6. 6. Windows Architecture hardware interfaces (buses, I/O devices, interrupts, interval timers, DMA, memory cache control, etc., etc.) System Service Dispatcher Object Mgr. Windows USER, GDI File System Cache I/O Mgr Environment Subsystems System Processes Services Applications System Threads NTDLL.DLL Device & File Sys. Drivers Session Manager POSIX Windows DLLs Plug and Play Mgr. Power Mgr. Security Reference Monitor Virtual Memory Processes & Threads Local Procedure Call Graphics Drivers Kernel Hardware Abstraction Layer (HAL) (kernel mode callable interfaces) Configura- tion Mgr (registry) OS/2 Windows Copyright Microsoft Corporation Task Manager Explorer SvcHost.Exe WinMgt.Exe SpoolSv.Exe Service Control Mgr. LSASS User Application Subsystem DLLs User Mode Kernel Mode WinLogon Services.Exe
  7. 7. Kernel-mode Architecture of Windows Copyright Microsoft Corporation NT API stubs (wrap sysenter) -- system library (ntdll.dll) user mode kernel mode NTOS executive layer Trap/Exception/Interrupt Dispatch CPU mgmt: scheduling, synchr, ISRs/DPCs/APCs Drivers Devices, Filters, Volumes, Networking, Graphics Hardware Abstraction Layer (HAL): BIOS/chipset details firmware/ hardware CPU, MMU, APIC, BIOS/ACPI, memory, devices NTOS kernel layer Caching Mgr Security Procs/Threads Virtual Memory IPC glue I/O Object Mgr Registry Copyright Microsoft Corporation
  8. 8. Kernel/Executive layers <ul><li>Kernel layer – ntos/ke – ~ 5% of NTOS source) </li></ul><ul><ul><li>Abstracts the CPU </li></ul></ul><ul><ul><ul><li>Threads, Asynchronous Procedure Calls (APCs) </li></ul></ul></ul><ul><ul><ul><li>Interrupt Service Routines (ISRs) </li></ul></ul></ul><ul><ul><ul><li>Deferred Procedure Calls (DPCs – aka Software Interrupts) </li></ul></ul></ul><ul><ul><li>Providers low-level synchronization </li></ul></ul><ul><li>Executive layer </li></ul><ul><ul><li>OS Services running in a multithreaded environment </li></ul></ul><ul><ul><li>Full virtual memory, heap, handles </li></ul></ul><ul><ul><li>Extensions to NTOS: drivers, file systems, network, … </li></ul></ul>Copyright Microsoft Corporation
  9. 9. NT (Native) API examples <ul><li>NtCreateProcess (&ProcHandle, Access, SectionHandle , DebugPort, ExceptionPort, …) </li></ul><ul><li>NtCreateThread (&ThreadHandle, ProcHandle , Access, ThreadContext, bCreateSuspended, …) </li></ul><ul><li>NtAllocateVirtualMemory ( ProcHandle , Addr, Size, Type, Protection, …) </li></ul><ul><li>NtMapViewOfSection (SectionHandle, ProcHandle , Addr, Size, Protection, …) </li></ul><ul><li>NtReadVirtualMemory ( ProcHandle , Addr, Size, …) </li></ul><ul><li>NtDuplicateObject ( srcProcHandle , srcObjHandle, dstProcHandle , dstHandle, Access, Attributes, Options) </li></ul>Copyright Microsoft Corporation
  10. 10. Windows Vista Kernel Changes <ul><li>Kernel changes mostly minor improvements </li></ul><ul><ul><li>Algorithms, scalability, code maintainability </li></ul></ul><ul><ul><li>CPU timing: Uses Time Stamp Counter (TSC) </li></ul></ul><ul><ul><ul><li>Interrupts not charged to threads </li></ul></ul></ul><ul><ul><ul><li>Timing and quanta are more accurate </li></ul></ul></ul><ul><ul><li>Communication </li></ul></ul><ul><ul><ul><li>ALPC: Advanced Lightweight Procedure Calls </li></ul></ul></ul><ul><ul><ul><li>Kernel-mode RPC </li></ul></ul></ul><ul><ul><ul><li>New TCP/IP stack (integrated IPv4 and IPv6) </li></ul></ul></ul><ul><ul><li>I/O </li></ul></ul><ul><ul><ul><li>Remove a context switch from I/O Completion Ports </li></ul></ul></ul><ul><ul><ul><li>I/O cancellation improvements </li></ul></ul></ul><ul><ul><li>Memory management </li></ul></ul><ul><ul><ul><li>Address space randomization (DLLs, stacks) </li></ul></ul></ul><ul><ul><ul><li>Kernel address space dynamically configured </li></ul></ul></ul><ul><ul><li>Security: BitLocker, DRM, UAC, Integrity Levels </li></ul></ul>Copyright Microsoft Corporation
  11. 11. Windows 7 Kernel Changes <ul><li>Miscellaneous kernel changes </li></ul><ul><ul><li>MinWin </li></ul></ul><ul><ul><ul><li>Change how Windows is built </li></ul></ul></ul><ul><ul><ul><li>Lots of DLL refactoring </li></ul></ul></ul><ul><ul><ul><li>API Sets (virtual DLLs) </li></ul></ul></ul><ul><ul><li>Working-set management </li></ul></ul><ul><ul><ul><li>Runaway processes quickly start reusing own pages </li></ul></ul></ul><ul><ul><ul><li>Break up kernel working-set into multiple working-sets </li></ul></ul></ul><ul><ul><ul><ul><li>System cache, paged pool, pageable system code </li></ul></ul></ul></ul><ul><ul><li>Security </li></ul></ul><ul><ul><ul><li>Better UAC, new account types, less BitLocker blockers </li></ul></ul></ul><ul><ul><li>Energy efficiency </li></ul></ul><ul><ul><ul><li>Trigger-started background services </li></ul></ul></ul><ul><ul><ul><li>Core Parking </li></ul></ul></ul><ul><ul><ul><li>Timer-coalescing, tick skipping </li></ul></ul></ul><ul><li>Major scalability improvements for large server apps </li></ul><ul><ul><li>Broke apart last two major kernel locks, >64p </li></ul></ul><ul><li>Kernel support for ConcRT </li></ul><ul><ul><li>User-Mode Scheduling (UMS) </li></ul></ul>Copyright Microsoft Corporation
  12. 12. MinWin <ul><li>MinWin is first step at creating architectural partitions </li></ul><ul><ul><li>Can be built, booted and tested separately from the rest of the system </li></ul></ul><ul><ul><li>Higher layers can evolve independently </li></ul></ul><ul><ul><li>An engineering process improvement, not a microkernel NT! </li></ul></ul><ul><li>MinWin was defined as set of components required to boot and access network </li></ul><ul><ul><li>Kernel, file system driver, TCP/IP stack, device drivers, services </li></ul></ul><ul><ul><li>No servicing, WMI, graphics, audio or shell, etc, etc, etc </li></ul></ul><ul><li>MinWin footprint: </li></ul><ul><ul><li>150 binaries, 25MB on disk, 40MB in-memory </li></ul></ul>
  13. 13. MinWin Layering Shell, Graphics, Multimedia, Layered Services, Applets, Etc. Kernel, HAL, TCP/IP, File Systems, Drivers, Core System Services MinWin
  14. 14. Timer Coalescing <ul><li>Secret of energy efficiency: Go idle and Stay idle </li></ul><ul><li>Staying idle requires minimizing timer interrupts </li></ul><ul><li>Before, periodic timers had independent cycles even when period was the same </li></ul><ul><li>New timer APIs permit timer coalescing </li></ul><ul><ul><li>Application or driver specifies tolerable delay </li></ul></ul><ul><ul><li>Timer system shifts timer firing </li></ul></ul>MarkRuss
  15. 15. Broke apart the Dispatcher Lock <ul><li>Scheduler Dispatcher lock hottest on server workloads </li></ul><ul><ul><li>Lock protects all thread state changes (wait, unwait) </li></ul></ul><ul><ul><li>Very lock at >64x </li></ul></ul><ul><li>Dispatcher lock broken up in Windows 7 / Server 2008 R2 </li></ul><ul><ul><li>Each object protected by its own lock </li></ul></ul><ul><ul><li>Many operations are lock-free </li></ul></ul>Copyright Microsoft Corporation
  16. 16. Removed PFN Lock <ul><li>Windows tracks the state of pages in physical memory </li></ul><ul><ul><li>In use: in working sets: </li></ul></ul><ul><ul><li>Not assigned: on paging lists: freemodified, standby, … </li></ul></ul><ul><li>Before, all page state changes protected by global PFN (Physical Frame Number) lock </li></ul><ul><li>As of Windows 7 the PFN lock is gone </li></ul><ul><ul><li>Pages are now locked individually </li></ul></ul><ul><ul><li>Improves scalability for large memory applications </li></ul></ul>Copyright Microsoft Corporation
  17. 17. The Silicon Power Wall <ul><li>The situation: </li></ul><ul><li>Power 2 ∝ Clock frequency </li></ul><ul><li>Voltage ∝ Power 2 </li></ul><ul><ul><li>Clock frequency and Voltage offset each other </li></ul></ul><ul><li>Clock frequency inversely proportional to logic path length </li></ul><ul><li>Bad News: </li></ul><ul><li>Power is about as low as it can go </li></ul><ul><li>Logic paths between clocked elements are pretty short </li></ul><ul><li>Good News: </li></ul><ul><li>Moore’s Law continues (# transistors doubles ~22 months) </li></ul><ul><li>All that parallel computational theory is going into practice </li></ul><ul><li>Transistors going into more cores, not faster cores! </li></ul><ul><li>Software subject to Amdahl’s Law, not Moore’s Law </li></ul><ul><ul><li>(or Gustafson’s Law </li></ul></ul><ul><ul><li>– if my wife can find large enough datasets she cares about) </li></ul></ul>
  18. 18. Approaches to HW parallelism <ul><li>Homogeneous </li></ul><ul><li>More big superscalar cores </li></ul><ul><li>Extend with private (or shared) SIMD engines (SSE on steroids) </li></ul><ul><li>( Maybe) not very energy efficient </li></ul><ul><li>A few more big, cores and lots of smaller, slower, cooler cores </li></ul><ul><li>Use SIMD for performance </li></ul><ul><li>Shutoff idle small cores for energy efficiency (but leakage?) </li></ul><ul><li>Lots of little fully programmable cores, all the same </li></ul><ul><li>Nobody has ever gotten this to work – more on this later </li></ul><ul><li>Heterogeneous </li></ul><ul><li>Programmable Accelerators (e.g. GPUs) </li></ul><ul><li>Attach loosely-coupled, specialized (non-x86), energy-efficient cores </li></ul><ul><li>Fixed-function Accelerators </li></ul><ul><li>Very energy-efficient, device-like computational units for very-specific tasks </li></ul>
  19. 19. User Mode Scheduling (UMS) <ul><li>Improve support for efficient cooperative multithreaded scheduling of small tasks (over-decomposition) </li></ul><ul><ul><ul><li>Want to schedule tasks in user-mode </li></ul></ul></ul><ul><ul><ul><li>Use NT threads to simulate CPUs, multiplex tasks onto these threads </li></ul></ul></ul><ul><li>When a task calls into the kernel and blocks, the CPU may get scheduled to a different app </li></ul><ul><ul><ul><li>If a single NT thread per CPU, when it blocks it blocks. </li></ul></ul></ul><ul><ul><ul><li>Could have extra threads, but then kernel and user-mode are competing to schedule the CPU </li></ul></ul></ul><ul><li>Tasks run arbitrary Win32 code (but only x64/IA64) </li></ul><ul><ul><ul><li>Assumes running on an NT thread (TEB, kernel thread) </li></ul></ul></ul><ul><li>Used by ConcRT (Visual Studio 2010’s Concurrency Run-Time) </li></ul>Copyright Microsoft Corporation
  20. 20. Windows 7 User-Mode Scheduling <ul><li>UMS breaks NT thread into two parts: </li></ul><ul><ul><li>UT: user-mode portion (TEB, ustack, registers ) </li></ul></ul><ul><ul><li>KT: kernel-mode portion (ETHREAD, kstack, registers) </li></ul></ul><ul><li>Three key properties: </li></ul><ul><ul><li>User-mode scheduler switches UTs w/o ring crossing </li></ul></ul><ul><ul><li>KT switch is lazy: at kernel entry (e.g. syscall, pagefault) </li></ul></ul><ul><ul><li>CPU returned to user-mode scheduler when KT blocks </li></ul></ul><ul><li>KT “returns” to user-mode by queuing completion </li></ul><ul><ul><li>User-mode scheduler schedules corresponding UT </li></ul></ul><ul><ul><li>(similar to scheduler activations, etc) </li></ul></ul>Copyright Microsoft Corporation
  21. 21. Normal NT Threading kernel user KT 0 KT 1 KT 2 UT 2 UT 1 UT 0 Kernel-mode Scheduler NTOS executive trap code <ul><li>NT Thread is Kernel Thread (KT) and User Thread (UT) </li></ul><ul><li>UT/KT form a single logical thread representing NT thread in user or kernel </li></ul><ul><ul><li>KT: ETHREAD, KSTACK, link to EPROCESS </li></ul></ul><ul><ul><li>UT: TEB, USTACK </li></ul></ul>x86 core Copyright Microsoft Corporation
  22. 22. User-Mode Scheduling (UMS) kernel user Thread Parking KT 0 KT 1 KT 2 UT Completion list Primary Thread UT 0 UT 1 UT 0 User-mode Scheduler trap code NTOS executive KT 0 blocks Only primary thread runs in user-mode Trap code switches to parked KT KT blocks  primary returns to user-mode KT unblocks & parks  queue UT completion Copyright Microsoft Corporation
  23. 23. UMS <ul><li>Based on NT threads </li></ul><ul><ul><ul><li>Each NT thread has user & kernel parts (UT & KT) </li></ul></ul></ul><ul><ul><ul><li>When a thread becomes UMS, KT never returns to UT </li></ul></ul></ul><ul><ul><ul><ul><li>(Well, sort of) </li></ul></ul></ul></ul><ul><ul><ul><li>Instead, the primary thread calls the USched </li></ul></ul></ul><ul><li>USched </li></ul><ul><ul><ul><li>Switches between UTs, all in user-mode </li></ul></ul></ul><ul><ul><ul><li>When a UT enters kernel and blocks, the primary thread will hand CPU back to the USched declaring UT blocked </li></ul></ul></ul><ul><ul><ul><li>When UT unblocks, kernel queues notification </li></ul></ul></ul><ul><ul><ul><li>USched consumes notifications, marks UT runnable </li></ul></ul></ul><ul><li>Primary Thread </li></ul><ul><ul><ul><li>Self-identified by entering kernel with wrong TEB </li></ul></ul></ul><ul><ul><ul><li>So UTs can migrate between threads </li></ul></ul></ul><ul><ul><ul><li>Affinities of primaries and KTs are orthogonal issues </li></ul></ul></ul>Copyright Microsoft Corporation
  24. 24. UMS Thread Roles <ul><li>Primary threads: represent CPUs, normal app threads enter the USched world and become primaries, primaries also can be created by UScheds to allow parallel execution </li></ul><ul><ul><li>Primaries represent concurrent execution </li></ul></ul><ul><li>UMS threads (UT/KTs): allow blocking in the kernel without losing the CPU </li></ul><ul><ul><li>UMS thread represent concurrent blocking in kernel </li></ul></ul>Copyright Microsoft Corporation
  25. 25. Thread Scheduling vs UMS Core 2 Core 1 Core 2 Core 1 MarkRuss Thread 3 Non-running threads Thread 4 Thread 5 Thread 1 Thread 2 Thread 6 User Thread 2 Kernel Thread 2 User Thread 1 Kernel Thread 1 User Thread 3 Kernel Thread 3 User Thread 4 Kernel Thread 4 User Thread 5 Kernel Thread 5 User Thread 6 Kernel Thread 6
  26. 26. Win32 compat considerations <ul><li>Why not Win32 fibers? </li></ul><ul><li>TEB issues </li></ul><ul><ul><ul><li>Contains TLS and Win32-specific fields (incl LastError) </li></ul></ul></ul><ul><ul><ul><li>Fibers run on multiple threads, so TEB state doesn’t track </li></ul></ul></ul><ul><li>Kernel thread issues </li></ul><ul><ul><ul><li>Visibility to TEB </li></ul></ul></ul><ul><ul><ul><li>I/O is queued to thread </li></ul></ul></ul><ul><ul><ul><li>Mutexes record thread owner </li></ul></ul></ul><ul><ul><ul><li>Impersonation </li></ul></ul></ul><ul><ul><ul><li>Cross-thread operations expect to find threads and IDs </li></ul></ul></ul><ul><ul><ul><li>Win32 code has thread and affinity awareness </li></ul></ul></ul>Copyright Microsoft Corporation
  27. 27. Futures: Master/Slave UMS? remote kernel Remote x86 Thread Parking KT 0 KT 1 KT 2 UT 2 UT 1 Remote Scheduler trap code NTOS executive Kernel-mode Scheduler Syscall Completion Queue Syscall Request Queue UT 0 x86 core UTs (can) run on accelerators or x86s KTs run on x86s, syscalls remoted/batched Pagefaults are just like syscalls Accelerator never “loses the CPU” (implicit primary) Copyright Microsoft Corporation
  28. 28. Operating Systems Futures <ul><li>Many-core challenge </li></ul><ul><ul><li>New driving force in software innovation: </li></ul></ul><ul><ul><ul><ul><li>Amdahl’s Law overtakes Moore’s Law as high-order bit </li></ul></ul></ul></ul><ul><ul><li>Heterogeneous cores? </li></ul></ul><ul><li>OS Scalability </li></ul><ul><ul><li>Loosely –coupled OS: mem + cpu + services? </li></ul></ul><ul><ul><li>Energy efficiency </li></ul></ul><ul><li>Shrink-wrap and Freeze-dry applications? </li></ul><ul><li>Hypervisor/Kernel/Runtime relationships </li></ul><ul><ul><li>Move kernel scheduling (cpu/memory) into run-times? </li></ul></ul><ul><ul><li>Move kernel resource management into Hypervisor? </li></ul></ul>Copyright Microsoft Corporation
  29. 29. Windows Academic Program <ul><li>Windows Kernel Internals </li></ul><ul><ul><li>Windows kernel in source (Windows Research Kernel – WRK) </li></ul></ul><ul><ul><li>Windows kernel in PowerPoint (Curriculum Resource Kit – CRK) </li></ul></ul><ul><li>Based on Windows Server 2008 Service Pack 1 </li></ul><ul><ul><li>Latest kernel at time of release </li></ul></ul><ul><ul><li>First kernel release with AMD64 support </li></ul></ul><ul><li>Joint program between Windows Product Group and MS Academic Groups </li></ul><ul><ul><li>Program directed by Arkady Retik (Need a DVD? Have questions?) </li></ul></ul><ul><ul><ul><li>Information available at </li></ul></ul></ul><ul><ul><ul><li>http://microsoft.com/WindowsAcademic OR </li></ul></ul></ul><ul><ul><ul><li>[email_address] </li></ul></ul></ul><ul><li>Microsoft Academic Contacts in Buenos Aires </li></ul><ul><ul><ul><li>Miguel Saez ( [email_address] ) or </li></ul></ul></ul><ul><ul><ul><li>Ezequiel Glinsky ( [email_address] ) </li></ul></ul></ul>Copyright Microsoft Corporation
  30. 30. muchas gracias