SQL PASS 臺灣分部十一月份聚會-Take a trip down memory lane


Published on

講師:Po Chen
講師簡介:美商金融業 Senior Database Architect / Engineer
分享主題:Walking down memory lane, something about SQL Server memory
深入淺出 SQL Server 記憶體管理

SQL Server memory management past, present and hidden gem.

* Memory Concept

* SQLOS and memory management

* Large Page

* Memory Corruption

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • BPool – Single page allocationMTL: – Multipage allocation for non buffer pool. –g only affect on the 32-bit -> aka VAS reservation , default 256MBPAE: OS concept , use 36 point instead of 32-bit point to allow OS to see more than 4GB memory up to 64GBUse 3GB for User space inside the VAS. /USERVA: use between 2-3GB as user space inside the VAS
  • The subset of the VAS of a process that resides in physical memory is known as the working set.
  • Show thread in DMVShow thread in the command line : * open taskmgr to find the spid * open command line : pslist –d spid * open the procexp -> click spid -> thread tab , any thread starting with sqldk ( 2012) are the worker threads Each Thread use 2MB DWA: Direct Windows , extend store procedure, OLE automation ( sp_OA ) , Link server--- Show the VAS--- Work for both SQL 2008/2012SELECT physical_memory_in_use_kb / ( 1024 ) AS sql_physical_mem_in_use_mb, locked_page_allocations_kb / ( 1024 ) AS awe_memory_mb, total_virtual_address_space_kb / ( 1024 ) AS max_vas_mb, virtual_address_space_committed_kb / ( 1024 ) AS sql_committed_mb, memory_utilization_percentage AS working_set_percentage, virtual_address_space_available_kb / ( 1024 ) AS vas_available_mb, process_physical_memory_low AS is_there_external_pressure, process_virtual_memory_low AS is_there_vas_pressureFROM sys.dm_os_process_memory ;--- Show number of threadselect COUNT(*) from sys.dm_os_workers;select * from sys.dm_os_workers;gosp_configure "max worker threads";select max_workers_count from sys.dm_os_sys_info;--- Switch to VM show the thread from OS --- DWAselect * from sys.dm_os_loaded_modulesselect * from sys.dm_os_loaded_modules where company <> 'Microsoft Corporation'
  • There were 2 memory manager pre 2012: Buffer Pool and MPA
  • Compare the performance counter changeMemory Manager replace the buffer poolKB replace page
  • SQLServer binary changed from 2008R2 to 2012
  • Bring up the Windbg -- *DO NOT do this on the production system* windbg -y srv*c:\public_symbols*http://msdl.microsoft.com/download/symbols sqlservr.exe -sSGC3SQL2K12-- Display all the modulex *! -- Setup the break point -- KERNELBASE, KERNEK32 are the windows API.-- SQLDK is the SQL Server dll. bpKERNELBASE!AdjustTokenPrivilegesbpKERNELBASE!VirtualAllocbp kernel32!QueryMemoryResourceNotification bp kernel32!GetLargePageMinimum bpsqldk!MemoryNode::GetTotalPhysicalMemorybpsqldk!MemoryClerkInternal::AllocateLargePageBufferOnlybpsqldk!MemoryClerkInternal::AllocateLargePageBuff-- type g to continue the debug process-- When the breakpoint is hit. Type kb to print out the call stack-- To disable the breakpoint. bd [number] , EX bd 0 means disable the break point 0 disable-- When complete the debug, type .detach to detach the Windbg from SQL server process ( SQL server would continue to run)-- If you type “q”, windbg would be closed and shutdown the SQL Server*DO NOT do this on the production system*
  • Show SSMS AWE check box between 2012 and 2008windbg -y srv*c:\public_symbols*http://msdl.microsoft.com/download/symbols sqlservr.exe -sSGC3SQL2K12On SQL2008R2. When AWE enable, SQL Server use “kernel32!AllocateUserPhysicalPages” to allocate the memory.Bpkernel32!AllocateUserPhysicalPagesOn SQL2012. When LPIM enable, SQL Server use “kernel32!AllocateUserPhysicalPagesNuma” to allocate the memory.Bpkernel32!AllocateUserPhysicalPages This also prove the SQL2012 is using AWE API to allocate the memory even there is no AWE option in the SQL2012.
  • 2012 does not have use large page for buffer pool is because BP is just the clerk . 834 does for 2012 is pre allocate all the memory upfront during the startup.
  • SQL Server read the page from disk , its called clean page. SQL engine will periodic check the page's check SUM ( especially before the page is being modified and mark it as dirty). If the check SUM does not match. 832 error occur. ECC RAMWindows server 2012SQL server 2012 -- DEMO for 834-- Demo 824 errorUSE [master];GOIF DATABASEPROPERTYEX (N'Corrupt824', N'Version') > 0BEGINALTER DATABASE [Corrupt824] SET SINGLE_USERWITH ROLLBACK IMMEDIATE;DROP DATABASE [Corrupt824];ENDGOCREATE DATABASE [Corrupt824];USE [Corrupt824];GOCREATE TABLE [t1] ([c1] INT IDENTITY,[c2] CHAR (8000) DEFAULT 'a');GOINSERT INTO [t1] DEFAULT VALUES;GO 10-- List the pages in the tableDBCC IND (N'Corrupt824', N't1', -1);GO-- Pick a page to corruptALTER DATABASE [Corrupt824] SET SINGLE_USER;GODBCC WRITEPAGE (N'Corrupt824', 1, 284, 0, 2, 0x0000, 1);GOALTER DATABASE [Corrupt824] SET MULTI_USER;GO-- Clean the error log and suspect_pagesDELETE FROM [msdb].[dbo].[suspect_pages];EXEC sp_cycle_errorlog;GOselect * from [msdb].[dbo].[suspect_pages];-- Demo script for I/O Errors demo.-- Run the Setup-IOErrors.sql script-- Trip the I/O errorSELECT*FROM[Corrupt824].[dbo].[t1];GO-- Look in error log using Object Explorer.-- Look in the Windows Application event log.-- Look in suspect_pages table.SELECT*FROM[msdb].[dbo].[suspect_pages];GO
  • SQL PASS 臺灣分部十一月份聚會-Take a trip down memory lane

    1. 1. Take a trip down memory lane SQL Server memory management past, present and hidden gem Po
    2. 2. Who am I ? Carnegie Mellon University, MS in Information Technology Banking and Financial industry. Manage more than 4,000 database instances environment including, MSSQL, MSAS, Sybase, DB2 and Oracle. Manage the Database client application deployment to more than 100,000 desktops/Servers. Blog: http://sqlpanda.com Twitter: https://twitter.com/SQLPANDA Linkedin: http://www.linkedin.com/pub/po-chen/2/249/791 2
    3. 3. Today’s Topic What we will cover today ? SQLOS and memory management Large Page Database Corruption … Mostly 64-bit What we will not cover today ? NUMA WOW64 Column store Index SQL Server in Virtualization environment SQL Server 2014 (Hekaton, BP extension ..) Memory pressure
    4. 4. Why do we care about the memory? 5
    5. 5. Memory acronym Acronym • • • • • • • • • • 6 VAS PAE AWE 3GB/USERVA MTL LPIM SPA MPA ASPA LPA Full Name Virtual Address Space Physical Address Extension Address Windowing Extensions 4GB Tuning Memory To Leave Lock Page In Memory Single Page Allocation Multi-Page allocation Any Size Page Allocation Large Page Allocation
    6. 6. VAS ? 7
    7. 7. Max Memory now means Max Memory (Almost …) 8
    8. 8. Memory Manager Change in SQL 2012 / • • • • 9 HeapAlloc AllocateUserPhysicalPages AllocateUserPhysicalPagesNuma VitrualAlloc
    9. 9. Performance Counter SQL 2008R2 Memory Manager Reserved Memory Memory Manager Reserved Memory In Use Buffer Pool Target (Pages) Buffer Pool Committed (Pages) Buffer Pool Stolen (Pages) + Buffer Pool Database Buffer Pool Reserved (Pages) Buffer Pool Free (Pages) Buffer Pool Stolen Potential (Pages) 1 0 SQL 2012 Memory Manager Large Pages Allocated Memory Manager Emergency Memory Memory Manager Emergency Memory In Use Memory Manager Target Committed (KB) Memory Manager Current Committed (KB) Memory Manager Pages Allocated (KB) Memory Manager Pages Reserved (KB) Memory Manager Pages Free (KB) Memory Manager Pages In Use Memory Manager Page Alloc Potential (KB)
    10. 10. Memory Manager type and error log Conventional memory • Default • Using conventional memory in the memory manager. AWE Large Page • LPIM • “Using locked pages in the memory manager. ” OR “Address Windowing Extensions enabled." • Host has RAM > 8GB • LPIM • SQL Enterprise Edition • “Using locked pages in the memory manager. ” and “Large Page Allocated: 32MB” Error Log will show the physical memory installed on the host. EX: Detected 8499 MB of RAM. This is an informational message; no user action is required.  New in 2012 1 1
    11. 11. SQLOS Pre SQL Server 2012 (2008R2) • Sqlserver.exe -> 60MB • Sqlos.dll -> 16KB 1 2 SQL Server 2012 Sqlserver.exe -> 188KB Sqldk.dll -> 2MB Sqlos.dll -> 26KB Sqlmin.dll -> 30MB Sqllang.dll -> 33MB
    12. 12. SQLOS Use Windows API to • Check LPIM privilege: AdjustTokenPrivileges() • Allocate the memory: VirtualAlloc() • Monitor the memory: QueryMemoryResourceNotification() Lets do some debug : ( Windbg ) -> DO NOT do this on the production system 1 3
    13. 13. LPIM & AWE Lets do some debug : ( Windbg ) -> DO NOT do this on the production system ever!! 1 4
    14. 14. LPIM & LPA LPIM LPIM Host has RAM > 8GB SQL Enterprise Edition -T 834 1 5 2012: “Using locked pages in the memory manager” 2008: “Address Windowing Extensions enabled” 2012: Using locked pages in the memory manager. Large Page Allocated: 32MB 2008: Large Page Extensions enabled. Large Page Granularity: 2097152 Large Page Allocated: 32MB locked pages for buffer pool. 2012: Same as before 2008: Using large pages for buffer pool. DO NOT use 834 if you have column store index in SQL 2012
    15. 15. Database Corruption: 823, 824, 825, 832 823: Hard IO error, Can not read page at all 824: Soft IO error -> Check SUM fail during the data read. 825: IO Error, Retry success 832 error -> memory Corruption Error: 832, Severity: 24, State: 1. A page that should have been constant has changed (expected checksum: <expected value>, actual checksum: <actual value>, database <dbid>, file '<filename>', page <pageno>). This usually indicates a memory failure or other hardware or OS corruption. Application Assisted Memory Error Recovery 1 6
    16. 16. JOIN US for our second annual event to get the best learning for analyzing, managing, and sharing business information and insights through the Microsoft Data Platform of technologies.
    17. 17. SQLSaturday Events – Nov/Dec Upcoming International Events Upcoming North America Events • Nov 16 #265 • Dec 7 #233 Oregon Washington DC • • • • • Nov 30 Dec 7 Dec 7 Dec 13 Dec 21 #261 #259 #266 #264 #274 Visit www.sqlsaturday.com to register for an event near you! Moscow Kharkov Lima Ancona Slovenia
    18. 18. Stay Involved! & Happy Holiday • Sign up for a free membership today at sqlpass.org • • • • Linked In: Professional Association for SQL Server Facebook: Professional Association for SQL Server Group Twitter: @SQLPASS The PASS Blog: sqlpass.org
    19. 19. Extensive reading • How to fake the NUMA( Not soft NUMA): http://www.sqlpanda.com/2013/06/build-mssql-fake-numa-testenvironment.html • SQL Server Large Page: http://www.sqlpanda.com/2013/11/sqlserver-large-page.html • 832 error video: http://h20324.www2.hp.com/SDP/Content/ContentDetails.aspx?Port alID=1&ID=3397&booth=63&tag=687&goback=%2Egmp_1434987 %2Egde_1434987_member_142079817 2 0