Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand. The document discusses Windows memory management, NUMA architecture, SQL Server memory management, and differences between AWE and 64-bit systems. It provides information on optimizing SQL Server performance through effective memory configuration and management.

2. "Technology is dominated by two types of
people: those who understand what they do
not manage, and those who manage what
they do not understand.“
“Putt's Law and the Successful Technocrat
How to Win in the Information Age”
Archibald Putt

3. DAT322: SQL Server
2005 Memory Internal
Geyzerskiy Dmitriy
IT Architect, Senior Technologies Consultant
dima@performit.co.il

4. Session Objectives and Agenda
• Windows Memory Management
• NUMA Architecture
• SQL Server Memory Management
• AWE vs. 64-bit

5. Myths

10. Windows Memory Overview
• Virtual Memory
• 32 bits: 4GB
• 64 bits: 17,179,869,184 GB
• VM states: Committed, Reserved, Free
• Page size
• 4KB on x86, EM64T, AMD64
• 8KB on Itanium
• Large pages: 4MB on x64, 16MB on Itanium
• Paging
• Page Faults
• Working Sets
• Kernel Memory And User Memory

11. Retrieving system information using
Win32 API and Windbg

12. 3GB Process Space Option
Application Memory Application Memory
2GB 3GB
System Memory System Memory
2GB 1GB
• /3GB switch in BOOT.INI
• /USERVA (between 2048 and 3072, in 128MB increments)
• .EXE must be linked with LARGEADDRESSAWARE flag

13. 64 bit Address Space
x64 IA64
8TB 7TB
User User
6TB 6TB
System System
• Map more data into the address space
• The application “speed” is the same on 32 bit and 64 bit
• OS needs 2GB of memory to hold pointers to 16GB or
more physical memory

15. Paging Dynamics
Standby
List
Working Zero
Sets Free List
Modified List
List

16. Exploring Page Lists using Windows
Kernel Debugger

18. Sizing the Page File
• More RAM should mean smaller page file!
• Crash dump settings affect
• Full: size of RAM
• Kernel dump: much smaller
• To size correctly, review what goes there
• Minimum should = commit charge peak
• Maximum could be a multiple of this

20. Measuring memory performance

21. Agenda
• Windows Memory Management
• NUMA Architecture
• SQL Server Memory Management
• AWE vs. 64-bit

22. NUMA

23. What is SMP
Memory
Front-bus
CPU 0 CPU 1 ... CPU n
• SMP – Symmetric Multi-Processing
• Front-bus point of contention
• Difficult to scale beyond 32 CPU

24. What is NUMA
NUMA (Non-Uniformed Memory Access)
Memory Memory
Local Memory Local Memory
Access Access
CPU 0 CPU 1 CPU 2 CPU 3 CPU … CPU … CPU … CPU n

25. What is Interleaved-NUMA
Memory Memory
CPU 0 CPU 1 CPU 2 CPU 3 CPU … CPU … CPU … CPU n
Local Memory Access Foreign Memory Access
• Enable NUMA hardware to behave as SMP
• Memory are used by all CPUs
• Each CPU’s cache line access slice of memory from all nodes
• SQL Server 2000 should use interleaved-NUMA

26. What is Soft-NUMA
• Activates custom NUMA configuration on top of
any of hardware
• Provides greater performance, scalability,
and manageability on SMP as well as on real
NUMA hardware.
• Provides one I/O thread and one lazy writer
thread per NUMA node.

27. Soft-NUMA Configuration Example
We have:
NUMA system with 2 nodes and 4 CPU per
Node
We need:
2 CPUs for loading application and the rest
of CPUs for queries.

28. Soft-NUMA Configuration Example
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
Server90NodeConfigurationNode0]
"CPUMask"=dword:0000000F
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
Server90NodeConfigurationNode1]
"CPUMask"=dword:00000030
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
Server90NodeConfigurationNode2]
"CPUMask"=dword:000000C0
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
ServerMSSQL.1MSSQLServerSuperSocketNetLibTcpIPAll]
"TcpPort"="1433[0x3],2000[0x4]"
"TcpDynamicPorts"=""
"DisplayName"="Any IP Address"

29. But don’t try to configure Soft-NUMA on the
system with 4 or less CPUs and at least 1 node
has exactly 1 CPU

30. Agenda
• Windows Memory Management
• NUMA Architecture
• SQL Server Memory Management
• AWE vs. 64-bit

31. SQL Server 2005 Architecture
Protocols
Query Processor
Database Query
Parser Optimizer SQL Manager
SQLOS API
Manager Executor
Access Methods
External Components (CLR/MDAC)
Transaction Services
Storage Engine Managers for:
Lock Manager Row Operations
Indexes
File Manager Utilities:
Pages
Bulk Load DBCC
Allocations
Buffer Manager Backup/Restore
Versions
SQLOS API
SQLOS Hosting API
Deadlock
Monitor
SQLOS Lock
Manager
Resource
Monitor Synchronization
Lazy Buffer
Writer Scheduling I/O
Pool
Scheduler
Monitor Memory Manager
= thread

32. SQL Server Memory Management
High Physical Internal / External
Resource Low VAS
Monitor Low Physical Internal / External
Memory Pressure
Generic Cache Memory Buffer Pool CLR Memory
Memory Clerk Clerk Memory Clerk Clerk
Memory Node

33. CP CacheStore
insert t values (2.1, 3.12) insert t values (3.33, 4.1)
Auto-parameterization
(@1 numeric(2,1), @2 numeric(3,2)) (@1 numeric(3,2), @2 numeric(2,1))
insert [t] values(@1 , @2) insert [t] values(@1 , @2)
Bucket 0 Bucket 1 Bucket 2
SQL Server 2005 RTM SQL Server 2005 SP2

34. Monitoring Memory with SQL Server DMVs

35. Agenda
• Windows Memory Management
• NUMA Architecture
• SQL Server Memory Management
• AWE vs. 64-bit

36. Address Windowing Extensions (AWE)
• Access more than 4GB of physical memory.
• Is ignored on systems with less than 3GB of physical memory.
• Is never swapped to disk.
1 Allocate the physical memory (Lock Pages in Memory)
Create a region in the process address space to serve as
2 a window for mapping views of this physical memory
3 Map a view of the physical memory into the virtual
memory window

37. SQL Server Process Address Space with AWE
0xFFFFFFFF
Operating System
0xC0000000
SQL Server or OS
(/3GB switch)
0x80000000
(Buffer Pool)
Plan Cache
Workspace
SQL Server
DB Page
Locks
Cache
Other
Query
AWE
Memory
Thread stacks
MemToLeave area
0x00000000

38. SQL Server 2005 32-bit AWE Memory
• Right OS version
• Windows Server 2003 Standard and up
• /PAE in boot.ini enables 32-bit OS
to address more than 4GB memory
• SQL Server Edition
• Enterprise Edition
• Developer Edition
• sp_configure ‘awe enabled’

39. Lock Pages In Memory Option
A significant part of sql server process memory has been paged out.
This may result in a performance degradation. Duration: 0 seconds.
Working set (KB): 1086400, committed (KB): 2160928, memory
utilization: 50%.
• Entry in the SQLERROR log
• 64 bit: Using locked pages for buffer pool
• 32 bit: Address Windowing Extensions is enabled
• Discarded in Standard Edition
• The Local System account has the 'lock pages in
memory' privilege by default

41. SQL Server 2005 64 bit vs. 32 bit
• The only way to get virtual memory > 3GB
• What is different from 32-bit?
• All pointers are 64-bit
• SQL Server commits ‘min server memory’ memory at startup
• Some internal memory-related data-structure constants larger
• 64-bit alignment of data structures

42. SQL Server 2005 64 bit vs. 32 bit
• What is the same?
• No on-disk database format changes
• No differences in buffer pool policy / algorithms from 32-bit
• All uses of memory can use additional 64-bit memory
• DB Page Cache, Query Workspace Memory, Plan Cache,
Locks, External uses, Utilities, …

43. 32-Bit and 64-Bit System Limitations
Architectural 64-bit Windows 32-bit Windows
component
Virtual memory 16 TB 4 GB
Paging file size 512 TB 16 TB
Hyperspace 8 GB 4 MB
Paged pool 128 GB 470 MB
Non-paged pool 128 GB 256 MB
System cache 1 TB 1 GB
System PTEs 128 GB 660 MB

44. Resources
• Blogs
• Slava Oks’ blog: http://blogs.msdn.com/slavao
• SQL Programmability & API Development Team Blog:
http://blogs.msdn.com/sqlprogrammability/
• External Links:
• NUMA FAQ: http://lse.sourceforge.net/numa/faq
• Books:
• Eldad Eilam: Reversing: Secrets of Reverse Engineering
• Ken Henderson: SQL Server 2005 Practical
Troubleshooting The Database Engine
• Kalen Delaney: Inside Microsoft SQL Server 2005 The
Storage Engine

45. ‫כדאי למלא משוב!‬
‫איך ממלאים?‬
‫בעקבות מייל שישלח בסיום כל יום,‬ ‫•‬
‫ב- ‪ Business Center‬במתחם ‪,HP‬‬ ‫•‬
‫בעמדות האינטרנט במלונות הילטון ודן‬ ‫•‬
‫מילאת משוב? מגיעה לך חולצת ‪!Live It‬‬
‫מלאת משוב בשלושת ימי הכנס?‬
‫יש לך הזדמנות לזכות בכרטיס טיסה לתאילנד מתנת סוכנות‬
‫‪ ,BTC‬מכשיר בלאק ג'ק מתנת סמסונג, מכשיר ‪ HTC‬מתנת‬
‫ניופאן, מדיה סנטר מתנת ‪ DataSafe‬ועוד...‬

46. Summary
• It pays to understand SQL Server memory management
• A number of performance issues either originate or manifest as
memory issues
• Memory-based performance tuning is a very useful technique
• Significant internal and external changes in SQL Server 2005
• Consider NUMA for your next large-scale project
• Upgrade your system to 64 bit

47. ‫כמה מילים על.... המסיבה!‬
‫• החל משעה 03:02 יוצאות הסעות מבתי המלון למסיבה בנמל‬
‫אילת.‬
‫• ארוחת הערב תוגש במסיבה עצמה.‬
‫• שימו לב! בשל בעיות אבטחה‬
‫כבדות בנמל אין אפשרות להגיע‬
‫עם רכבים פרטיים או מוניות. לא‬
‫נוכל להבטיח כניסה למי שיבחר‬
‫להגיע שלא באמצעות ההסעות.‬

49. © 2008 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only.
MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.