Your SlideShare is downloading. ×
Sql server – query optimization – remove bookmark lookup – remove rid lookup
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Sql server – query optimization – remove bookmark lookup – remove rid lookup

480
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
480
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... Home All Articles SQL Interview Q & A Blog Stats Contact Tools SQL Books >>Search<< SQL S ERVER J OURNEY WITH SQL Feeds: Posts Comments A UTHORITY Personal Notes of Pinal Dave « SQL SERVER – Interesting Observation SQL SERVER – Query Optimization – COMMUNITY INITIATIVES – Query Hint – FORCE ORDER Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2 » SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup October 7, 2009 by pinaldave Today, I would like to share one very quick tip about how to remove bookmark lookup or RID lookup. Let us first understand Bookmark lookup or RID lookup. Please note that from SQL Server 2005 SP1 onwards, Bookmark look up is known as Key look up. When a small number of rows are requested by a query, the SQL Server optimizer will try to use a non-clustered index on the column or columns contained in the WHERE clause to retrieve the data requested by the query. If the query requests data from columns not present in the non-clustered index, SQL Server must go back to the data pages to get the data in those columns. Even if the table has a clustered index or not, the query will still have to return to the table or clustered index to retrieve the data. In the above scenario, if table has clustered index, it is called bookmark lookup (or key lookup); if the table does not have clustered index, but a non-clustered index, it is called RID lookup. This operation is very expensive. To optimize any query containing bookmark lookup or RID lookup, it should be removed from the execution plan to improve performance. There are two different ways to remove bookmark/RID lookup. Before we understand these two methods, we will create sample table without clustered index and simulate RID lookup. RID Lookup is a bookmark lookup on a heap that uses a supplied row identifier (RID). USE tempdb GOhttp://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 2. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... -- Create Table OneIndex with few columns CREATE TABLE OneIndex (ID INT, FirstName VARCHAR(100), LastName VARCHAR(100), City VARCHAR(100)) GO -- Insert One Hundred Thousand Records INSERT INTO OneIndex (ID,FirstName,LastName,City) SELECT TOP 100000 ROW_NUMBER() OVER (ORDER BY a.name) RowID, Bob, CASE WHEN ROW_NUMBER() OVER (ORDER BY a.name)%2 = 1 THEN Smith ELSE Brown END, CASE WHEN ROW_NUMBER() OVER (ORDER BY a.name)%1000 = 1 THEN Las Vegas WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 1 THEN New York WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 5 THEN San Marino WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 3 THEN Los Angeles ELSE Houston END ABOUT PINAL DAVE FROM sys.all_objects a Pinal Dave is a Technology CROSS JOIN sys.all_objects b Evangelist. He has written over GO 2200 articles on the subject on his blog at Now let us run following select statement and check the execution plan. http://blog.sqlauthority.com. SELECT ID, FirstName Along with 8+ years of hands on experience he holds a FROM OneIndex Masters of Science degree and WHERE City = Las Vegas a number of certifications, GO including MCTS, MCDBA and MCAD (.NET). He is co-author of three SQL Server books - SQL Server Programming, SQL Wait Stats and SQL Server Interview Questions and Answers. His past work experiences includes Technology Evangelist at Microsoft and Sr. Consultant at SolidQ. Prior to joining Microsoft he was awarded Microsoft MVP award for three continuous years for his contribution in community.http://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 3. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... As there is no index on table, scan is performed over the table. We will create a clustered index on the table and check the execution plan once again. -- Create Clustered Index CREATE CLUSTERED INDEX [IX_OneIndex_ID] ON [dbo].[OneIndex] ( [ID] ASC ) ON [PRIMARY] Follow @pinaldave GO Send +Pinal Dave an email at pinal@sqlauthority.com Now, run following select on the table once again. SELECT ID, FirstName FROM OneIndex BLOG STATS WHERE City = Las Vegas 47,000,371 (46 Million+) GO EMAIL SUBSCRIPTION Enter your email address to subscribe to this blog and receive notifications of new posts by email. Join 32,364 other followers Sign me up! SQL IN SIXTY SECONDS <!--[if gte mso 9]> 800x600 <![endif]--> <!--[if gte mso 9]> Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 <![endif]--><!--[if gte mso 9]> <![endif]--> It is clear from execution plan that as a clustered index is created on the table, table scan is now converted to clustered index scan. In either case, base table is completely scanned and there is no seek on the table. BOOKS I AUTHORED Now, let us see the WHERE clause of our table. From our basic observation, if we create an index on the column that contains the clause, a performance improvement may be obtained. Let us create non-clustered index on the table and then check the execution plan. -- Create Index on Column City As that is used in where condition CREATE NONCLUSTERED INDEX [IX_OneIndex_City] ON [dbo].[OneIndex] ( [City] ASC ) ON [PRIMARY]http://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 4. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... GO After creating the non-clustered index, let us run our select statement again and check the execution plan. . Amazon | Kindle SELECT ID, FirstName Flipkart | Indiaplaza FROM OneIndex WHERE City = Las Vegas GO . Amazon | Kindle Flipkart | Indiaplaza <!--[if gte mso 9]> 800x600 <![endif]--> As we have an index on the WHERE clause, the SQL Server query execution engine uses the non-clustered index to retrieve data from the table. However, the columns used in the SELECT clause are still not part of the index, and to display those columns, the engine will have to go to the base table again and retrieve those columns. This particular behavior is known as bookmark lookup or key Amazon | Kindle lookup. Flipkart | Indiaplaza There are two different methods to resolve this issue. I have demonstrated both the methods together; however, it is recommended that you use any one of these methods for removing key lookup. I prefer Method 2. Method 1: Creating non-clustered cover index.http://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 5. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... In this method, we will create non-clustered index containing the columns, which are used in the SELECT statement, along with the column which is used in the Amazon | 1 | 2 | 3 | 4 | 5 WHERE clause. Flipkart | Indiaplaza CREATE NONCLUSTERED INDEX [IX_OneIndex_Cover] ON [dbo].[OneIndex] ( City, FirstName, ID FUNNY INDEX VIDEO ) ON [PRIMARY] GO Once the above non-clustered index, which covers all the columns in query, is created, let us run the following SELECT statement and check our execution plan. SELECT ID, FirstName FROM OneIndex WHERE City = Las Vegas GO From the execution plan, we can confirm that key lookup is removed the only Login index seek is happening. As there is no key lookup, the SQL Server query engine does not have to go to retrieve the data from data pages and it obtains all the necessary data from index itself. SQLAuthori on Facebook Like Con 21,525 people like SQLAuthority.co 21,524 people like SQLAuthority.co . Method 2: Creating an included column non-clustered index. Naresh Facebook social plugin Here, we will create non-clustered index that also includes the columns, which are used in the SELECT statement, along with the column used in the WHERE clause. In this method, we will use new syntax introduced in SQL Server 2005. An index with included nonkey columns can significantly improve query performance when SQLAUTHORITY LINKS Subscribe to Newsletter all columns in the query are included in the index. My Homepage CREATE NONCLUSTERED INDEX [IX_OneIndex_Include] ON [dbo].[OneIndex] Windows Live Blog ( -------------------- City Top Downloadshttp://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 6. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... ) INCLUDE (FirstName,ID) ON [PRIMARY] PDF Downloads GO Script Downloads From the execution plan, it is very clear that this method also removes the key Script Bank lookup as well. Favorite Scripts All Scripts - 1 All Scripts - 2 All Scripts - 3 Top Articles Best Articles Favorite Articles - 1 Favorite Articles - 2 -------------------- > SQL Interview Q & A < SQL Coding Standards SQL FAQ Download -------------------- Jobs @ SQLAuthority In summary, Key lookup, Bookmark lookup or RID lookup reduces the performance of query, and we can improve the performance of query by using included column index or cover index. ABOUT NUPUR DAVE I will cover few additional concepts related to the optimal method in another Nupur Dave loves technology article. simply because it makes life more convenient. She is Reference : Pinal Dave (http://blog.sqlauthority.com) devoted to technology because Related Post: it touches our heart makes our daily lives easier. Among the SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID many technological programs Lookup – Remove Key Lookup she uses and embraces SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Windows Live most because she can do lots of things with Lookup – Remove Key Lookup – Part 2 ease – from photo management SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID to movies; business emails to Lookup – Remove Key Lookup – Part 3 personal social media connections. DISCLAIMER This is a personal weblog. The opinions expressed here represent my own and not those of my employer. For accuracy and official reference refer to MSDN/ TechNet/ BOL. My employer do not endorse any tools, applications, books, or concepts mentioned on the blog. I have documented my personal experience on this blog.http://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 7. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... Share: TOP 3 COMMENTERS 2397 - Madhivanan Like this: �Like Be the first to like this. 474 - Imran Mohammed Posted in Pinal Dave, SQL, SQL Authority, SQL Index, SQL Optimization, SQL Performance, 301 - Ramdas Jaya SQL Query, SQL Scripts, SQL Server, SQL Tips and Tricks, SQLServer, T SQL, Technology | 28 Comments 28 Responses Brad Schulz on October 7, 2009 at 11:13 am | Reply No need to add ID to your nonclustered index… that’s redundant. Since you have a clustered index on ID, then ID will automatically be included in any nonclustered index you create (so that it could do the bookmark lookup). So your nonclustered index examples would be to either create an index on (City, FirstName) or an index on (City) INCLUDE (FirstName). –Brad Pradip on October 7, 2009 at 11:29 am | Reply I have a table customer in which cstid of BIGINT data type with identity. As table have identity column I do not want any type of index on it. Being cstid as primary key of table, is it possible without any index? Michael on October 7, 2009 at 12:23 pm | Reply Dear Pinal, good post, very clear sample, but I would like to see a remark, that Key- Lookups could be in some situations, special with very wide tables, and the SELECT includes a lot or all of the columns, the much much better way. Think about a table containing 11.000.000 customers with 200 columns an the avg size of a row of 30kb. The SELECT need to return the whole row-data. Seach is done within the lastname, no leading placeholder to be able to use an Index. So an additional small index only containing the lastname and the key and then doing an Key Lookup is much much faster instead of using the index, even clustered index where every row nees severals data blocks. In case of IO we see differences in the factor of 500 times more IO. Regards, Michael DBTEAM on November 17, 2009 at 7:21 pm | Reply Hi pinal, We have a requirement to Merge six 12millions data tables into single Table.All the tables are having duplicate Emails.So We put the Primary key for email column using EnterPrise Manager and We r trying to merge by writinghttp://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 8. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... the following insert query, But it’s taking more than 20hrs…:-( Insert into Total( Email,FirstName,LastName,Address,City,State,Zipcode,Gender,DOB , Phone,WebAddress1,Webaddress2,IPAddress,DateTime,Interest ) Select Email,FirstName,LastName,Address,City,State,Zipcode,Gender,DOB , Phone,WebAddress1,Webaddress2,IPAddress,DateTime,Interest From [Set4] Where (not exists(select email from Total where Total.email=[Set4].email)) How can we do this most efficiently??? Please Help Us…… Thanks in advance… pinaldave on October 7, 2009 at 1:03 pm | Reply Brad Schulz, You are correct, however, I am going to cover that particular concept in different blog post. This post is written to show the concept of covering Index. In next blog post, I am going to show as there is clustered index, your non clustered index will not require that key and it can be still covering Index. Kind Regards, Pinal stanly on October 21, 2011 at 2:24 pm | Reply how can i remove duplicate records in a table and after deleting duplicate records i need one distinct record madhivanan on October 24, 2011 at 9:22 pm | Reply Search for “delete duplicate values” in this site. You will get related articles Ramdas on October 7, 2009 at 7:37 pm | Reply Hi Pinal, Nice explanation of the covering index concept. What is your take on the comment posted by Michael about the customer table situation, where one needs to return a lot of columns. Thank youhttp://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 9. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... dave p on October 7, 2009 at 9:04 pm | Reply Pinal, Why not just remove, from the query, the column that’s causing the key lookup. This is really much simpler I’m sure you’ll agree. You’ve written about best practices for database design in the past, and taught me that fewer columns per table are always better. So if you’re getting lots of key lookups, it follows that we should split up our tables and make them smaller. This is the query optimizer’s way of telling us that we have messed up our design! Thank you for continuing to educate us on how best to use our database. pinaldave on October 7, 2009 at 10:33 pm | Reply Hi, We can not remove the columns which are needed in query from SELECT statement. That would not be best practices. dave p on October 7, 2009 at 11:58 pm | Reply But what if you don’t need those columns. Then you can remove them. This example proves that you shouldn’t add columns to a query that you don’t need. That’s never a best practice (at least in my experience). You revealed in another post that we should keep indexes as narrow as possible, and I have been following this rule religiously! Since starting down this path I have managed to get almost every index in our database down to a single column. So far this has worked very nicely as far as I can tell, but I’m not really sure how to collect query performance numbers. Would you mind helping me so that I can send you a report and you can show readers what great progress can be made if we follow your methods. pinaldave on October 8, 2009 at 6:28 am | Reply Hi, If we do not need those columns in query for SURE they need to be removed. I agree with you there. I will send you other details soon. Kind Regards, Pinal SQL SERVER – Query Optimization – on October 8, 2009 at 7:01 am | Reply Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2 Journey to SQL Authority with Pinal Dave [...] 8, 2009 by pinaldave This article is follow up of my previous article SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RIDhttp://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 10. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... Lookup &#821…. Please do read my previous article before continuing [...] SQL SERVER – Query Optimization – on October 12, 2009 at 7:01 am | Reply Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3 Journey to SQL Authority with Pinal Dave [...] SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lo… [...] jfbergeron on October 13, 2009 at 4:41 pm | Reply Hi Pinal, I’m curious to know why you have included the ID column in the covering index, as well as in the nc index with the includes. I thought all columns in the Clustered index were already included in the non clustered index. Is there a reason why you did that? Thanks, pinaldave on October 13, 2009 at 5:28 pm | Reply Hi jfbergeron, Please read my follow up post here : SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3 http://blog.sqlauthority.com/2009/10/12/sql-server-query-optimization-remove- bookmark-lookup-remove-rid-lookup-remove-key-lookup-part-3 Kind Regards, Pinal Barbara37 on October 23, 2009 at 4:24 pm | Reply And these new notes were not just plain bank notes. , SQL SERVER – Removing Key Lookup – on November 9, 2009 at 7:01 am | Reply Seek Predicate – Predicate – An Interesting Observation Related to Datatypes Journey to SQL Authority with Pinal Dave [...] SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lo… [...] SQL SERVER – Remove Bookmark Key on November 22, 2009 at 7:02 am | Reply Lookup – 4 Different Ideas Journey to SQL Authority with Pinal Dave [...] SQL SERVER – Query Optimization – Remove Bookmark Lookup –http://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 11. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... Remove RID Lookup – Remove Key Lo… [...] Mike Chapman on November 24, 2009 at 1:19 am | Reply Awesome. Managed to knock down a query from 5-6 minutes down to 5 seconds or so. Ronald Lee on January 10, 2010 at 12:40 pm | Reply Excellent Post!!! Thank you for the information. This helps me improve my SP from 5 seconds to 0.xx second. Thanks alot! SQL SERVER – Four Posts on Removing the on May 13, 2010 at 7:01 am | Reply Bookmark Lookup – Key Lookup Journey to SQL Authority with Pinal Dave [...] SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lo… [...] Sridhar Nalluri on June 15, 2010 at 2:13 pm | Reply Hi Pinal, Nice post, very impressive and understandable…Thank You. Nick Duckstein on November 2, 2010 at 5:23 am | Reply Dave, I’m confused why the best recommendation isn’t to create a clustered index. In our situation we have a 5 Billion row table without a clustered index. The RID lookups are killing us. The need for a lot of extra non-clustered indexes is also killing performance because this is on a replication server. Finally the writes to a clustered index are dramatically less than to a non- clustered index with a HEAP. See: http://msdn.microsoft.com/en-us/library/cc917672.aspx#EFAA How can anyone justify a large table without a clustered index? Nick Nidhi on March 28, 2011 at 8:36 pm | Reply HI Pinal, Thanks for this post.Its very easy to understand.Good post. SQL SERVER – Interview Questions and on July 19, 2011 at 7:00 am | Reply Answers – Frequently Asked Questions – Day 19 of 31 Journey tohttp://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 12. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... SQLAuthority [...] In the above scenario, if table has clustered index, it is called bookmark lookup (or key lookup); if the table does not have clustered index, but a non-clustered index, it is called RID lookup. (Read more here) [...] SQL SERVER – An Important Part of Most on January 5, 2012 at 7:02 am | Reply SELECT statement – WHERE clause – Quiz – Puzzle – 4 of 31 « SQL Server Journey with SQL Authority [...] Questions and Answers ISBN: 1466405643 Page#38-40 Does Order of Column in WHERE clause Matter? Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup Logical Query Processing Phases – Order of Statement [...] Hemant K on January 9, 2012 at 11:31 am | Reply Hi, If you are saying, if table has clustered index, it is called bookmark lookup (or key lookup); if the table does not have clustered index, but a non-clustered index, it is called RID lookup then you again wrote that as we have an index on the WHERE clause, the SQL Server query execution engine uses the non-clustered index to retrieve data from the table. However, the columns used in the SELECT clause (id, firstname) are still not part of the index that means (neither cluster index nor non-cluster index has applied on it)and to display those columns, the engine will have to go to the base table again and retrieve those columns. so how we can say this particular behavior is known as bookmark lookup or key lookup Comments RSS Leave a Reply Email (required) (Address never made public) Name (required)http://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]
  • 13. SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author... Website Notify me of follow-up comments via email. Post Comment Notify me of new posts via email. Blog at WordPress.com. Theme: Customized MistyLook by WPThemes.http://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]