Another in TheSQLGuru’s very popular « Deep Dive » series, this time covering indexing. You will learn about the internal structures of heaps, clustered indexes and nonclustered indexes. We will examine included columns and covering indexes, filtered indexes, indexed views, indexing DMVs and finding the right indexes. There are some incredibly helpful graphics as well as some nice samples to help you transition theory to practice. Session présentée lors du SQLSaturday Paris 2014

1. Indexing Deep Dive
Beyond heaps, clustered indexes,
and nonclustered indexes

2. Rejoignez la communauté SQL Server
Webcasts, Conférences, Afterworks
Session donnée lors du
http://GUSS.pro
@GUSS_FRANCE
/GUSS
/GUSS.FR

3. Who am I??
Kevin G. Boles
SQL Server Consultant Extraordinaire
“The Wolf” for the Relational Engine
Indicium Resources, Inc.
AIM/Twitter: TheSQLGuru
Gtalk/GMail: TheSQLGuru@gmail.com
MCITP, MCT, MVP 2007-2012, yada-yada-yada
© 2011, Hugo Kornelis

4. What do I do?
• World-class Better than average relational
engine expert 
• ~45,000 man-hours invested in SQL Server
• Absolutely LOVE doing targeted performance
analysis and tuning work
• VERY good at making others better at
interacting with SQL Server
• Couldn’t SSAS my way out of a paper bag
… with both ends open! 
© 2011, Hugo Kornelis

5. Topics covered
• Heap, clustered index, nonclustered index
– (Quick overview)
• Included columns
• Filtered indexes
• Indexed views
• Finding the right indexes
• Over-indexing
© 2011, Hugo Kornelis

6. Topics NOT covered
• Index maintenance
• Indexing options (YOU NEED TO USE THESE!)
• XML indexes
• Free-text indexes
• Spatial indexes
• Columnstore Indexes
• In-Memory Indexes
© 2011, Hugo Kornelis

7. Topics covered
• Heap, clustered index, nonclustered index
– (Quick overview)
• Included columns
• Filtered indexes
• Indexed views
• Finding the right indexes
• Over-indexing
© 2011, Hugo Kornelis

8. Data Storage Pages
© 2011, Hugo Kornelis

9. Table Structure, 30000 Foot View
© 2011, Hugo Kornelis

10. IAM Page
(If size of data file exceeds 4GB)
100000110000000
001010010000010
000000000010000
000000000000000
001100000000100
000000100100000
000000000000001
prev next
Structure of a heap
Data page
ID FName LName Other
1 Jan Jansen …
7 Wilma Boer, de …
xxxxx
16 Els Rekels …
IAM Page
000000100010000
000000000000000
000000000000000
000000000100000
000000000000000
000000000000000
000000000000000
prev next
Data page
ID FName LName Other
16 Els Rekels …
9 Klaas Jansen …
81 Annet Visser …
Data page
ID FName LName Other
3 Klaas Visser …
4 Wim Blok …
forwarding pointer
back-pointer
IAM Page
001000000000000
000010100100000
000000000000000
000100001100000
000000000000000
000000011100000
000000000000000
prev next
prev next prev next prev next
© 2011, Hugo Kornelis

11. Navigating a heap: table scan
Data page
ID FName LName Other
1 Jan Jansen …
7 Wilma Boer, de …
xxxxx
16 Els Rekels …
IAM Page
000000100010000
000000000000000
000000000000000
000000000100000
000000000000000
000000000000000
000000000000000
prev next
Data page
ID FName LName Other
16 Els Rekels …
9 Klaas Jansen …
81 Annet Visser …
Data page
ID FName LName Other
3 Klaas Visser …
4 Wim Blok …
forwarding pointer
prev next prev next prev next
© 2011, Hugo Kornelis

12. Structure of a clustered index
Leaf page
ID page
NULL xxxx
9 xxxx
ID FName LName Other
1 Jan Jansen …
3 Klaas Visser …
4 Wim Blok …
7 Wilma Boer, de …
Root level
ID page
NULL xxxx
17 xxxx
Leaf page
ID FName LName Other
9 Klaas Jansen …
16 Els Rekels …
Intermediate
prev next
prev next
prev next
prev next
Leaf page
ID FName LName Other
25 Els Rekels …
81 Annet Visser …
prev next
Intermediate
prev next
ID page
17 xxxx
IAM Page
000000100010000
000000000000000
000000000000000
000000000100000
000100000000000
000000000001000
000000100000000
prev next
© 2011, Hugo Kornelis

13. Structure of a nonclustered index
FName LName page
NULL NULL xxxx
Jansen Jan xxxx
Leaf page
FName LName ID
Blok Wim 4
Boer, de Wilma 7
Jansen Klaas 9
Root level
FName LName page
NULL NULL xxxx
Visser Klaas xxxx
Leaf page
FName LName ID
Jansen Jan 1
Rekels Els 16
Rekels Els 25
Visser Annet 81
Intermediate
prev next
prev next
prev next
prev next
Leaf page
FName LName ID
Visser Klaas 3
prev next
Intermediate
prev next
FName LName page
Visser Klaas xxxx
IAM Page
000000100010000
000000000000000
000000000000000
000000000100000
000100000000000
000000000001000
000000100000000
prev next
© 2011, Hugo Kornelis

14. Navigating an index: scan (ordered)
Leaf page
ID page
NULL xxxx
9 xxxx
ID FName LName Other
1 Jan Jansen …
3 Klaas Visser …
4 Wim Blok …
7 Wilma Boer, de …
Root level
ID page
NULL xxxx
17 xxxx
Leaf page
ID FName LName Other
9 Klaas Jansen …
16 Els Rekels …
Intermediate
prev next
prev next
prev next
prev next
Leaf page
ID FName LName Other
25 Els Rekels …
81 Annet Visser …
prev next
Intermediate
prev next
ID page
17 xxxx
IAM Page
000000100010000
000000000000000
000000000000000
000000000100000
000100000000000
000000000001000
000000100000000
prev next
© 2011, Hugo Kornelis

15. Navigating an index: scan (unordered)
LName FName page
NULL NULL xxxx
Jansen Jan xxxx
Leaf page
LName Fname ID
Blok Wim 4
Boer, de Wilma 7
Jansen Klaas 9
Root level
LName FName page
NULL NULL xxxx
Visser Klaas xxxx
Leaf page
LName FName ID
Jansen Jan 1
Rekels Els 16
Rekels Els 25
Visser Annet 81
Intermediate
prev next
prev next
prev next
prev next
Leaf page
LName FName ID
Visser Klaas 3
prev next
Intermediate
prev next
LName Fname page
Visser Klaas xxxx
IAM Page
000000100010000
000000000000000
000000000000000
000000000100000
000100000000000
000000000001000
000000100000000
prev next
© 2011, Hugo Kornelis

16. Index scan: ordered or unordered?
• Case A: Plan requires an ordered scan
– Operator property Ordered = True
– Scan will always be ordered
• Case B: Plan does not require ordered scan
– Operator property Ordered = False
– Table lock or no locks  unordered scan
• But only if table size exceeds 64 pages!
– Row locks, page locks  always ordered scan!
Advanced Indexing - © 2011, Hugo Kornelis

17. Navigating an index: seek
Leaf page
ID page
NULL xxxx
9 xxxx
ID Fname LName Other
1 Jan Jansen …
3 Klaas Visser …
4 Wim Blok …
7 Wilma Boer, de …
Root level
ID page
NULL xxxx
17 xxxx
Leaf page
ID FName LName Other
9 Klaas Jansen …
16 Els Rekels …
Intermediate
prev next
prev next
prev next
prev next
Leaf page
ID FName LName Other
25 Els Rekels …
81 Annet Visser …
prev next
Intermediate
prev next
ID page
17 xxxx
IAM Page
000000100010000
000000000000000
000000000000000
000000000100000
000100000000000
000000000001000
000000100000000
prev next
© 2011, Hugo Kornelis

18. Topics covered
• Heap, clustered index, nonclustered index
– (Quick overview)
• Included columns
• Filtered indexes
• Indexed views
• Finding the right indexes
• Over-indexing
© 2011, Hugo Kornelis

19. NC index seek + Key Lookup
ID page
NULL xxxx
9 xxxx
Leaf page
ID FName LName Other
1 Jan Jansen …
3 Klaas Visser …
4 Wim Blok …
7 Wilma Boer, de …
Root level
Intermediate
ID page
NULL xxxx
17 xxxx
Leaf page
ID FName LName Other
9 Klaas Jansen …
16 Els Rekels …
prev next
prev next
prev next
prev next
Leaf page
ID FName LName Other
25 Els Rekels …
81 Annet Visser …
prev next
Intermediate
prev next
ID page
17 xxxx
LName FName page
NULL NULL xxxx
Jansen Jan xxxx
Visser Annet xxxx
LName FName ID
Blok Wim 4
Boer, de Wilma 7
Jansen Klaas 9
Root level
LName FName page
NULL NULL xxxx
Visser Klaas xxxx
Intermediate
Leaf page
LName FName ID
Jansen Jan 1
Rekels Els 16
Rekels Els 25
Visser Annet 81
prev next
prev next
prev next
prev next
Leaf page
LName FName ID
Visser Klaas 3
Xavier Bill 98
prev next
Intermediate
prev next
LName FName page
Visser Klaas xxxx
Leaf page
© 2011, Hugo Kornelis

20. NC index seek + RID Lookup
LName FName page
NULL NULL xxxx
Jansen Jan xxxx
Leaf page
LName FName RID
Blok Wim 95,3
Boer, de Wilma 18,2
Jansen Klaas 31,3
Root level
LName FName page
NULL NULL xxxx
Visser Klaas xxxx
Intermediate
Leaf page
LName FName RID
Jansen Jan 18,1
Rekels Els 31,2
Rekels Els 18,4
Visser Annet 31,4
prev next
prev next
prev next
prev next
Leaf page
LName FName RID
Visser Klaas 18,3
prev next
Intermediate
prev next
LName FName page
Visser Klaas xxxx
Data page 18
ID FName LName Other
1 Jan Jansen …
7 Wilma Boer, de …
xxxxx (95,2)
16 Els Rekels …
Data page 31
ID FName LName Other
16 Els Rekels …
9 Klaas Jansen …
81 Annet Visser …
Data page 95
ID FName Lname Other
3 Klaas Visser …
4 Wim Blok …
prev next prev next prev next
© 2011, Hugo Kornelis

21. Avoiding lookups: Covering indexes
• A nonclustered index that has all data a
query needs is said to cover this query
– Benefit: No lookup required
• What data is in a nonclustered index?
– Indexed columns (in all pages)
– Clustered index columns (in leaf pages)
– Optionally: included columns (in leaf pages)
• Requires SQL Server 2005 or later
© 2011, Hugo Kornelis

22. Effect of including a column
FName LName page
NULL NULL xxxx
Jansen Jan xxxx
Leaf page
FName LName ID
Blok Wim 4
Boer, de Wilma 7
Jansen Klaas 9
Root level
FName LName page
NULL NULL xxxx
Visser Klaas xxxx
Leaf page
FName LName ID
Jansen Jan 1
Rekels Els 16
Rekels Els 25
Visser Annet 81
Intermediate
prev next
prev next
prev next
prev next
Leaf page
FName LName ID
Visser Klaas 3
prev next
Intermediate
prev next
FName LName page
Visser Klaas xxxx
Mail
x@y
x@y
x@y
Mail
x@y
x@y
x@y
x@y
Mail
x@y
© 2011, Hugo Kornelis

23. Effect of including a column
FName LName page
NULL NULL xxxx
Jansen Klaas xxxx
Rekels Els xxxx
Leaf page
Root level
FName LName page
NULL NULL xxxx
Visser Klaas xxxx
Leaf page
Intermediate
prev next
prev next
prev next
prev next
Leaf page
prev next
Intermediate
prev next
FName LName page
Visser Annet xxxx
FName LName ID Mail
Blok Wim 4 x@y
Boer, de Wilma 7 x@y
FName LName ID Mail
Jansen Klaas 9 x@y
Jansen Jan 1 x@y
FName LName ID Mail
Rekels Els 16 x@y
Rekels Els 25 x@y
Leaf page
FName LName ID Mail
Visser Annet 81 x@y
Visser Klaas 3 x@y
prev next
© 2011, Hugo Kornelis

24. Included columns
• Only in CREATE INDEX, not in ALTER INDEX
• Only for nonclustered indexes
• Number or size of columns not restricted
• All data types except text, ntext, image
CREATE NONCLUSTERED INDEX index_name
ON table_name ( column [ ,...n ] )
INCLUDE ( column [ ,...n ] ) ;
© 2011, Hugo Kornelis

25. Included columns
DEMO
© 2011, Hugo Kornelis

26. Topics covered
• Heap, clustered index, nonclustered index
– (Quick overview)
• Included columns
• Filtered indexes
• Indexed views
• Finding the right indexes
• Over-indexing
© 2011, Hugo Kornelis

27. What is a filtered index?
• Usually, indexes include entries for all rows
• Filtered indexes (SQL Server 2008 and up)
index only a subset of all rows
• Benefits
– Performance (demo to follow)
– Enforce uniqueness on subset of rows (often
where column is NULLable)
– Better plan when data values are skewed
– Smaller storage space
© 2011, Hugo Kornelis

28. How is a filtered index created?
• Only on nonclustered index
• No computed columns, hierarchyid, spatial,
or user-defined data type in predicate
• Only simple predicates allowed
• AND and IN allowed; OR and NOT not
CREATE NONCLUSTERED INDEX index_name
ON table_name ( column [ ,...n ] )
WHERE filter_predicate ;
© 2011, Hugo Kornelis

29. When is a filtered index used?
• If query predicate is subset of index
predicate, cost of lookup may prevent
filtered index usage
• Filtered index may also not be chosen
because of QO “functionality gap”.
– INCLUDE filter column often helps
– Or you can use hints…
© 2011, Hugo Kornelis

30. Lots of Issues, Limitations, etc.
• http://sqlperformance.com/2013/04/t-sql-queries/
optimizer-limitations-with-filtered-indexes
• http://sqlperformance.com/2014/06/sql-plan/
filtered-index-side-effect
• http://sqlperformance.com/2013/04/t-sql-queries/
filtered-indexes
Advanced Indexing - © 2011, Hugo Kornelis

31. Filtered indexes
DEMO
© 2011, Hugo Kornelis

32. Topics covered
• Heap, clustered index, nonclustered index
– (Quick overview)
• Included columns
• Filtered indexes
• Indexed views
• Finding the right indexes
• Over-indexing
© 2011, Hugo Kornelis

33. What is a (non indexed) view?
• Compare to macro in 3GL languages
CREATE VIEW dbo.VisView
AS SELECT FirstName
FROM dbo.Persons
WHERE LastName LIKE 'Vis%';
SELECT MAX(FirstName)
FROM dbo.VisView AS v;
(SELECT FirstName
FROM dbo.Persons
WHERE LastName LIKE 'Vis%') AS v;
© 2011, Hugo Kornelis

34. What is an indexed view?
• An indexed view is quite different!
– Upon creation, SELECT * FROM view is
executed and the results are stored
• Hence, indexed view is also called persisted view
– Future modifications in base table are
immediately reflected in persisted view
• Common use case: Aggregates
© 2011, Hugo Kornelis

35. How is an indexed view created?
• First: create the view
– Reference only base tables, no views
– Two-part naming required
– Must be schema-bound
– No GROUP BY without COUNT_BIG(*)
– Various T-SQL elements not allowed
• Necessary to keep automatic updates fast
– See Books Online for full list of requirements
© 2011, Hugo Kornelis

36. How is an indexed view created?
• Next: Create the index
– Must be a clustered index
– Must be a unique index
– Only columns in GROUP BY (if any) allowed
CREATE UNIQUE CLUSTERED INDEX index_name
ON view_name ( column [ ,...n ] ) ;
• Optional: create extra nonclustered indexes
– But why????
© 2011, Hugo Kornelis

37. When is an indexed view used?
• Steps in query optimization
1. View expansion
– All editions
2. Optimizer may choose to use indexed view
– Enterprise Edition only
• View expansion can be overridden
– Results in error if view is not indexed
FROM view_name WITH (NOEXPAND)
© 2011, Hugo Kornelis

38. When is an indexed view used?
• Enterprise Edition:
– View used with NOEXPAND hint
• Indexed view will always be used
– View used without NOEXPAND hint
• Indexed view may be used. Not used if alternative is
cheaper, or if optimizer finds “fast enough” plan
before indexed view matching
– View not used
• Indexed view may still be used!!
© 2011, Hugo Kornelis

39. When is an indexed view used?
• Other editions:
– View used with NOEXPAND hint
• Indexed view will always be used
– View used without NOEXPAND hint
• Indexed view will never be used
– View not used
• Indexed view will never be used
© 2011, Hugo Kornelis

40. Indexed views
DEMO
© 2011, Hugo Kornelis

41. Topics covered
• Heap, clustered index, nonclustered index
– (Quick overview)
• Included columns
• Filtered indexes
• Indexed views
• Finding the right indexes
• Over-indexing
© 2011, Hugo Kornelis

42. Finding the right indexes
• Understanding indexes, optimization and
YOUR DATA is key
• There is no silver bullet
• DMVs can offer some guidance
– Reset when service restarts
– Check after a few days uptime, or after
executing a representative test run
– Beware of incidental long-running jobs
© 2011, Hugo Kornelis

43. Index usage stats
• Most interesting columns:
– user_scans, user_seeks, user_lookups = reads
– user_updates = writes
• Reads < writes?
– Suspect, but not necessarily bad
• Reads > writes?
– Not suspect, but not necessarily good
• No reads in 2 years uptime – C YA! 
© 2011, Hugo Kornelis

44. Missing indexes
• Added when QO finds query that could
benefit from an index
– Indication of estimated cost saving
– No indication of cost of keeping index updated
– Best index for that query only
– Loves to INCLUDE columns
• Try to combine indexes into one index that
can help a lot of queries
© 2011, Hugo Kornelis

45. Index-related DMVs
DEMO
© 2011, Hugo Kornelis

46. Topics covered
• Heap, clustered index, nonclustered index
– (Quick overview)
• Included columns
• Filtered indexes
• Indexed views
• Finding the right indexes
• Over-indexing
© 2011, Hugo Kornelis

47. Over-indexing
• Indexes hurt performance in multiple ways
– Extra work on all data modifications
– Increased compilation time and less chance of
finding the best plan
– Increased backup size
– Extra index maintenance required
– LOCKING/BLOCKING/DEADLOCKING!
• BEWARE Database Tuning Advisor!!!!
© 2011, Hugo Kornelis

48. Over-indexing
DEMO
© 2011, Hugo Kornelis

49. T H E E N D
• Kevin G. Boles
• Relational Engine Wizard
• TheSQLGuru@gmail.com/@TheSQLGuru
Questions?
• http://www.sqlperformance.com/2013/04/t-sql-queries/filtered-indexes
• http://blogs.msdn.com/b/timchapman/archive/2012/08/27/the-drawback-of-using-filtered-indexes.aspx
© 2011, Hugo Kornelis

50. Slide Deck and samples by SQL Server MVP Hugo
Kornelis. This material is for personal consumption
only. Please do not share or distribute it.
• hugo@perFact.info
• @Hugo_Kornelis
• http://www.perFact.info
• http://sqlblog.com/blogs/hugo_kornelis
© 2011, Hugo Kornelis