SlideShare a Scribd company logo
1 of 41
Working with
Large Lists in
SharePoint 2013
WHY YOU NEED TO PAY ATTENTION TO THRESHOLDS AND METADATA
Steve Stewart
Accudata Systems, Inc.
7906 North Sam Houston Pkwy W | Suite 300 | Houston, TX 77064
Direct: 281.897.6394 | Fax: 281.897.5001
www.accudatasystems.com
blog.get-spsteve.com
@SharePointFly
sstewart@accudatasystems.com
Services we offer:
• Advisory Services
• Data Center Infrastructure
• Security and Mobility
• Managed Services
• Enterprise Applications
• Network
• Unified Communications
• Assessment & Compliance
Feedback & Other Important Stuff
 Use #HoustonSPTF
 Register to use the Mobile App at SPTF2015.com to give presentation
feedback
 Afternoon Reception runs from 2:45-3:30 p.m. in the Exhibit Hall.
Refreshments are free and there will be prizes!
Have you ever seen this?
List View Threshold
 The List View Threshold is set by default to 5,000 items for a web
application in Central Administration
 Administrators with special permissions have a default list view
threshold of 20,000 items
 Must be administrator on local server and on farm
 Several other thresholds also set (List View Lookup, Permission, etc.)
 Administrative Window – Time period for working on large lists
without limit
Central Admin - Resource Throttling
Resource throttling settings are accessed in Central Admin from
Application Management>Manage Web Applications>Web Application>Settings
CA Resource Throttling Settings
Some Important Notes
 The List View Threshold calculation includes items in the
Recycle Bin. If you delete 1,000 items, they still count
against the item limit until they are deleted from the
recycle bin.
 Search is not affected by the threshold. You can always
search for items and get results. Even if the list is well over
the threshold
 A common misunderstanding is that a view that has
been filtered to contain less than the threshold number
of items will display items when the list exceeds the
threshold. Only views filtered with indexed columns work
this way.
What Actions are Blocked When
the Threshold is Exceeded?
Operation Description
Add/remove/update a list column
All columns, including lookup and calculated columns, in addition
to many kinds of updates, such as a type change or a uniqueness
change. Some updates, such as a name change, are not
blocked because they do not affect every item in the list.
Add/remove/Update a list content type
Affects every item in the list so it is blocked for any list that has
more items than the list view threshold.
Create/remove an index
Affects every item in the list so it is blocked for any list that has
more items than the list view threshold.
Manage files which have no checked-in version
A non-indexed recursive query that fails for any list that has more
items than the list view threshold.
Non-indexed recursive queries
Includes filters and some sorts. This operation fails when the list size
is greater than the list view threshold. Because there is no index, it
does a full scan against the entire list. Also, it returns all items, and
it ignores folders.
Cross list query
Includes queries by the Content Query Web Part and follows the
list view threshold setting for auditors and administrators, which by
default is 20,000 items. If the operation involves more than 20,000
items, the query fails.
Blocked Operations, cont.
Operation Description
Lookup columns that enforce relationship behavior
You cannot create lookup columns that enforce relationship behavior
when the list that is referenced contains more items than the list view
threshold.
Deleting a list
Affects every item in the list so it is blocked for any list that has more items
than the list view threshold.
Deleting a site
If the sum of all items in a site is greater than the list view threshold,
deleting the site is prevented because it affects too many items.
Save list as template with data
Affects every item in the list so it is blocked for any list that has more items
than the list view threshold.
Showing totals in list views
Performs a query against every item in the list so it is blocked for any list
that has more items than the list view threshold.
Enable/disable attachments in a list
Affects every item in the list so it is blocked for any list that has more items
than the list view threshold.
What’s Happening in a Nutshell
 List View Threshold is a SQL limitation, not a SharePoint limitation
 To ensure accurate updates without affecting other users accessing the
same table, SQL locks the row being updated
 When more than 5,000 rows need to be locked due to the same query,
it is more efficient for SQL to lock the entire table … that is all the rows.
 It’s there to ensure optimized performance for ALL users
 So think of it this way. SQL normally eats one tiny piece of pie at a time
except when there are more than 5,000 pieces ordered. In this case,
SQL eats the entire pie!
“
”
So, just up the
number. SharePoint
can handle it right?
Actually, it’s not really a good idea to increase the
threshold number.
List View Threshold Throughput
0 5 10 15 20 25 30 35 40 45 50
5,000
10000
50000
100000
Request per Second
ListViewThreshold
RPS*
*Values are approximate and for illustration only.
As the size of the list threshold increases,
the number of requests per second
SharePoint can handle decreases
Does this mean my list can’t be larger
than 5,000 items? Of course not.
Feature Max Value Type
Number of items in a list
or library
30 million Supported Limit
Size of an individual file
item or item attachment
2 Gigabytes Boundary
Boundaries, Thresholds, and
Supported Limits
 Boundary - An absolute that cannot be exceeded by
design
 Threshold - A value that is set to a default and cannot be
exceeded unless the default is changed
 Supported Limit - Values determined by testing and
which represent known limits of the product. Exceeding
these limits may result in performance degradation,
errors, and other problems.
Thresholds and Limits
Threshold
or Limit
Default
value
Description
List View Threshold 5,000
Specifies the maximum number of list or library items that a database operation, such
as a query, can process at one time. Operations that exceed this limit are blocked.
To give you time to make alternative plans, You are warned on the List Settings page
when your list has exceeded 3,000 items. The warning contains a help link to this topic.
Unique permissions
limit
50,000
Specifies the maximum number of unique permissions allowed for a list or library.
Every time you break the inheritance of permissions for an item or folder, it is counted
as 1 unique permission toward this limit. If you try to add an item that would lead to
exceeding this limit, you are prevented from doing so.
Row size limit 6
Specifies the maximum number of table rows internal to the database used for a list
or library item. To accommodate wide lists with many columns, each item is wrapped
over several internal table rows, up to 6 rows and up to a total of 8,000 bytes
(excluding attachments).
For example, if you have a list with many small columns, one that contains hundreds
of Yes/No columns, then you could exceed this limit, in which case you would not be
able to add more Yes/No columns to the list, but you still may be allowed to add
columns of a different type.
Note Administrators can only set this limit by using the object model, not through the
user interface.
Thresholds and Limits
Threshold
or Limit
Default
value
Description
List View Lookup
Threshold
8
Specifies the maximum number of join operations, such as those based on lookup,
Person/Group, or workflow status columns.
If the query uses more than eight columns, the operation is blocked. However, it is
possible to programmatically select which columns to use by using maximal view,
which can be set through the object model.
List View Threshold
size for auditors and
administrators
20,000
Specifies the maximum number of list or library items that a database operation, such
as a query, can process at one time when performed by an auditor or administrator
with appropriate permissions. This setting works in conjunction with Allow Object
Model Override.
Allow Object Model
Override
Y
Specifies whether or not developers can perform database operations, such as
queries, that request an override of the List View Threshold to the higher limit specified
by the List View Threshold for auditors and administrators. An administrator must
enable an object model override, and then developers with appropriate permission
may then programmatically request that their query use the higher List View Threshold
to take advantage of it.
Daily Administrative Time Window
 Specifies a time period during which resource thresholds and limits
are ignored. An administrator can configure a time window during
"off-peak" hours in 15 minute increments and up to 24 hours, such as,
6:00 PM to 10:00 PM or 1:30 AM to 5:15 AM.
 Note A database operation or query started within the daily time
window continues until completion (or failure) even if it doesn't finish
within the specified time window.
 Time can be used to run queries and conduct other administrative tasks
such as indexing columns, creating views and deleting large numbers of
items.
 The administrative window is set in Central Administration for a web
application
Is the Threshold Really All or Nothing
for the Web Application
 Yes and no.
 You can only change the threshold for the entire web application using
Central Admin
 But, you can also use PowerShell to disable the threshold for a single list
$web = Get-SPWeb https://domain.com/web/
$list = $web.Lists[“List Name"]
$list.EnableThrottling = $false
Additional PowerShell commands are available for setting List thresholds at the web
application level. See MSDN for more information:
https://msdn.microsoft.com/en-
us/library/office/Microsoft.SharePoint.Administration.SPWebApplication_properties.aspx
REST API
 You can use the REST API to quickly view the list view threshold
settings for a particular site collection without opening Central
Admin by accessing the following URL:
 <site collection URL>/_api/site/maxItemsPerThrottledOperation

 Results will be returned that look like this:
Demonstrations
Views with Indexed Columns
Supported Column Types Unsupported Column Types
Single line of text Multiple lines of text
Choice (single value) Choice (multi-valued)
Number Calculated
Currency Hyperlink or Picture
Date and Time Custom Columns
Lookup (single value)
Indexing the column in the other
list or library does not improve
performance of the lookup
operation.
Lookup (multi-valued)
Person or Group (single value) Person or Group (multi-valued)
Yes/No External data
Managed Metadata
Filter Ideas
To filter by: Index the: For example:
Recently changed items Modified column
To view only items that have changed in the past week, apply the filter
Modified is greater than [Today]-7.
New items Created column
To create a view of only those items that were added in the past week,
apply the filter Created is greater than [Today]-7.
My items Created By column
To create a view of only those items that you added, apply the filter
Created By is equal to [Me].
Items due today
Due Date column
(created by you in a list
or library)
To create a view of only those items with a due date of today, apply
the filter Due Date is equal to [Today].
Discussion board updates Last Updated column
To create a view of only the discussions that were updated in the past
month, apply the filter Last Updated is greater than [Today]-30.
Archive files in a document library Date Modified
To create a view of documents not changed since the year 2006 (that
you then want to archive), apply the filter Date Modified is less than
December 31st 2006.
Find a subset of financial data
Region, Year (as two
simple indexes)
To create a view of financial data for the NorthEast region in 2008,
apply the filter Region equals "NE" AND Year equals 2008 because the
Region column is likely to have less values than the Year column.
Index a Column and Create a
View Based on it (Demo)
1. Go to list > List Settings
2. Scroll to the bottom of the
columns list
3. Click Indexed columns
4. Click Create a new index
5. Select Columns to index
6. Normally, you will pick only a
primary column from the drop
down. However, you may with
some fields select a secondary to
create a compound Index. For
example, Modified by and
Modified (date). Compound
indexes can be used with
managed metadata.
7. Click Create (max 20 indexes)
8. Indexed columns appear in View
builder filter area in the Indexed
Columns group.
Create Index Columns
Using InfoPath to Break the Rules
Trick: You cannot use a
calculated column as an indexed
column, but you can calculate a
value in InfoPath and use a rule to
transfer the value to a supported
column type.
Use Key Filters
 Key Filters work with managed metadata to help parse a list
 Prerequisite for using this feature is activation of the Metadata
Navigation and Filtering site feature.
 Fields available for Key Filtering are:
 Content Type
 Choice Field
 Managed Metadata Field
 Person or Group Field
 Date and Time Field
 Number Field
Steps to Create a Key
Filter
1. Go to List
Settings
2. Click Metadata
Navigation
Settings
3. Add Navigation
Hierarchies
4. Add Key Filters
5. Click OK
Folders – mainly for libraries
 You can divide items into folders to get around view thresholds
 This is best suited for document libraries
 Either use initial view with all folders visible or create navigation (quick
launch) to go directly to folder
 Use Per Location View Settings to customize default and available views
in folders
 In lists there is no default way to move items into folders after creation
 Document Center
Key Filters Get Us Part Way There,
But an Index is Still Needed
Datasheet
1. From the Ribbon, click Create View
2. Select Datasheet View as the view type
3. Under View Settings, expand folders and select Show all items
without folders
4. Expand Item Limit and select either Display All Items or Limit the
total number of items returned to option.
5. Type 30 in the box below it (this number must be below the list view
threshold)
6. Save the view
7. Click OK when you receive the warning about flat views.
8. Edit the view again, now change
Data Sheet View Settings
 Remember unless you are referencing indexed columns, no filters,
sorting, totals, etc., can be selected.
Export to Excel
 From the List/Library Ribbon click Export to Excel
 You can use the Export to Excel feature to create a read only
version of your list which can be used to perform queries
 Remove unwanted columns from the Excel worksheet without
affecting your view or the list – columns will not reappear upon
refresh
 Use the Refresh button to get new additions to the list.
 You can use Excel tools to generate reports, etc.
Open in Access
 You need Access 2013 to use this feature in SharePoint 2013
 From the Ribbon click Open in Access to create a client that can be
used for:
 CRUD (Create, Read, Update, Delete)
 Creating forms
 Queries
 Reporting
Even More Powerful – Access Views
1. From the Ribbon select Create View
2. Select Access View
Select View Style
3. Select an Access View Style
4. Click OK
5. Make adjustments to the form
and list design, as desired.
6. Use Access to update lists.
Save the Access View App
Anywhere
 Users can update your list from another site collection
 You can control permission based on the location where the
document is saved
 Think about adding a link to the app on the Quick Launch
 Users can download a copy for more efficiency
Questions
References
 Designing large lists and maximizing list
performance
 Techniques for managing large lists
 Manage lists and libraries with many items
 SharePoint 2010/2013: List view Lookup threshold
uncovered
 Software Boundaries and Limits for SharePoint
2013
 © Copyright 2015 Stephen D. Stewart. All rights reserved.

More Related Content

What's hot

Aiimi Project Management Office
Aiimi Project Management OfficeAiimi Project Management Office
Aiimi Project Management Office
AiimiLtd
 
Getting The Most Out Of SP Search SPSTC
Getting The Most Out Of SP Search SPSTCGetting The Most Out Of SP Search SPSTC
Getting The Most Out Of SP Search SPSTC
John Ross
 
Designing Corporate News Application Using SharePoint 2013 Web Content Manage...
Designing Corporate News Application Using SharePoint 2013 Web Content Manage...Designing Corporate News Application Using SharePoint 2013 Web Content Manage...
Designing Corporate News Application Using SharePoint 2013 Web Content Manage...
Nik Patel
 

What's hot (20)

SharePoint for Beginners
SharePoint for BeginnersSharePoint for Beginners
SharePoint for Beginners
 
Understanding SharePoint site structure what's inside
Understanding SharePoint site structure  what's insideUnderstanding SharePoint site structure  what's inside
Understanding SharePoint site structure what's inside
 
SPCA2013 - Best Practices Document Management in SharePoint (Online) 2013
SPCA2013 - Best Practices Document Management in SharePoint (Online) 2013SPCA2013 - Best Practices Document Management in SharePoint (Online) 2013
SPCA2013 - Best Practices Document Management in SharePoint (Online) 2013
 
Exploring the New Search in SharePoint 2013 - What can you do now?
Exploring the New Search in SharePoint 2013 - What can you do now?Exploring the New Search in SharePoint 2013 - What can you do now?
Exploring the New Search in SharePoint 2013 - What can you do now?
 
Understand the SharePoint Basics
Understand the SharePoint BasicsUnderstand the SharePoint Basics
Understand the SharePoint Basics
 
Aiimi Project Management Office
Aiimi Project Management OfficeAiimi Project Management Office
Aiimi Project Management Office
 
Share point development 101
Share point development 101Share point development 101
Share point development 101
 
SharePoint Beginner Training for End Users
SharePoint Beginner Training for End UsersSharePoint Beginner Training for End Users
SharePoint Beginner Training for End Users
 
So you’re building an intranet
So you’re building an intranetSo you’re building an intranet
So you’re building an intranet
 
Document Management in SharePoint without folders - Introduction to Metadata
Document Management in SharePoint without folders - Introduction to MetadataDocument Management in SharePoint without folders - Introduction to Metadata
Document Management in SharePoint without folders - Introduction to Metadata
 
SharePoint 5000 Item List view Threshold Checklist and Best Practices
SharePoint 5000 Item List view Threshold Checklist and Best PracticesSharePoint 5000 Item List view Threshold Checklist and Best Practices
SharePoint 5000 Item List view Threshold Checklist and Best Practices
 
SharePoint Tutorial and SharePoint Training - Introduction
SharePoint Tutorial and SharePoint Training - IntroductionSharePoint Tutorial and SharePoint Training - Introduction
SharePoint Tutorial and SharePoint Training - Introduction
 
Designing a great SharePoint Online intranet in Office 365
Designing a great SharePoint Online intranet in Office 365Designing a great SharePoint Online intranet in Office 365
Designing a great SharePoint Online intranet in Office 365
 
SharePoint Document Sets - SharePoint Saturday Denver 2013
SharePoint Document Sets - SharePoint Saturday Denver 2013SharePoint Document Sets - SharePoint Saturday Denver 2013
SharePoint Document Sets - SharePoint Saturday Denver 2013
 
Building an Extranet with Office 365
Building an Extranet with Office 365Building an Extranet with Office 365
Building an Extranet with Office 365
 
Getting The Most Out Of SP Search SPSTC
Getting The Most Out Of SP Search SPSTCGetting The Most Out Of SP Search SPSTC
Getting The Most Out Of SP Search SPSTC
 
SharePoint Folders & Metadata
SharePoint Folders & MetadataSharePoint Folders & Metadata
SharePoint Folders & Metadata
 
Designing Corporate News Application Using SharePoint 2013 Web Content Manage...
Designing Corporate News Application Using SharePoint 2013 Web Content Manage...Designing Corporate News Application Using SharePoint 2013 Web Content Manage...
Designing Corporate News Application Using SharePoint 2013 Web Content Manage...
 
Office 365 Tip: Create a team site on SharePoint
Office 365 Tip: Create a team site on SharePointOffice 365 Tip: Create a team site on SharePoint
Office 365 Tip: Create a team site on SharePoint
 
Share point 2016 end user training module 1 - introduction
Share point 2016 end user training   module 1 - introductionShare point 2016 end user training   module 1 - introduction
Share point 2016 end user training module 1 - introduction
 

Viewers also liked

SharePoint Folders vs. Metadata Best Practices
SharePoint Folders vs. Metadata Best PracticesSharePoint Folders vs. Metadata Best Practices
SharePoint Folders vs. Metadata Best Practices
Chris Woodill
 

Viewers also liked (20)

Office Delve for Office 365 Administrators
Office Delve for Office 365 AdministratorsOffice Delve for Office 365 Administrators
Office Delve for Office 365 Administrators
 
SharePoint Folders vs. Metadata
SharePoint Folders vs. MetadataSharePoint Folders vs. Metadata
SharePoint Folders vs. Metadata
 
10 Reasons to Avoid Folders in SharePoint 2013/2010
10 Reasons to Avoid Folders in SharePoint 2013/201010 Reasons to Avoid Folders in SharePoint 2013/2010
10 Reasons to Avoid Folders in SharePoint 2013/2010
 
Introduction to SharePoint Information Architecture
Introduction to SharePoint Information ArchitectureIntroduction to SharePoint Information Architecture
Introduction to SharePoint Information Architecture
 
SharePoint Folders vs. Metadata Best Practices
SharePoint Folders vs. Metadata Best PracticesSharePoint Folders vs. Metadata Best Practices
SharePoint Folders vs. Metadata Best Practices
 
5 Ways to Manage Documents in SharePoint & Office 365
5 Ways to Manage Documents in SharePoint & Office 3655 Ways to Manage Documents in SharePoint & Office 365
5 Ways to Manage Documents in SharePoint & Office 365
 
The 7 Deadly Sins of SharePoint: Planning Successful Implementations and Avoi...
The 7 Deadly Sins of SharePoint: Planning Successful Implementations and Avoi...The 7 Deadly Sins of SharePoint: Planning Successful Implementations and Avoi...
The 7 Deadly Sins of SharePoint: Planning Successful Implementations and Avoi...
 
SharePoint Jumpstart #1 Creating a SharePoint Strategy
SharePoint Jumpstart #1 Creating a SharePoint StrategySharePoint Jumpstart #1 Creating a SharePoint Strategy
SharePoint Jumpstart #1 Creating a SharePoint Strategy
 
Guiding a Successful SharePoint Implementation
Guiding a Successful SharePoint ImplementationGuiding a Successful SharePoint Implementation
Guiding a Successful SharePoint Implementation
 
SharePoint Document Types
SharePoint Document TypesSharePoint Document Types
SharePoint Document Types
 
How to implement SharePoint in your organization
How to implement SharePoint in your organizationHow to implement SharePoint in your organization
How to implement SharePoint in your organization
 
SharePoint Tips and Tricks you cannot live without
SharePoint Tips and Tricks you cannot live withoutSharePoint Tips and Tricks you cannot live without
SharePoint Tips and Tricks you cannot live without
 
Utilizing SharePoint for Project Management
Utilizing SharePoint for Project ManagementUtilizing SharePoint for Project Management
Utilizing SharePoint for Project Management
 
SharePoint 2010 Managed Metadata
SharePoint 2010 Managed MetadataSharePoint 2010 Managed Metadata
SharePoint 2010 Managed Metadata
 
SharePoint 2013 Search Topology and Optimization
SharePoint 2013 Search Topology and OptimizationSharePoint 2013 Search Topology and Optimization
SharePoint 2013 Search Topology and Optimization
 
SharePoint Performance Optimization In 10 Steps for the IT Professional
SharePoint Performance Optimization In 10 Steps for the IT ProfessionalSharePoint Performance Optimization In 10 Steps for the IT Professional
SharePoint Performance Optimization In 10 Steps for the IT Professional
 
I am Not your SharePoint Guru
I am Not your SharePoint GuruI am Not your SharePoint Guru
I am Not your SharePoint Guru
 
SharePoint Training They Won't Forget
SharePoint Training They Won't ForgetSharePoint Training They Won't Forget
SharePoint Training They Won't Forget
 
An SDLC for SharePoint
An SDLC for SharePointAn SDLC for SharePoint
An SDLC for SharePoint
 
Best Practices in Managing SharePoint Projects
Best Practices in Managing SharePoint ProjectsBest Practices in Managing SharePoint Projects
Best Practices in Managing SharePoint Projects
 

Similar to Working with Large Lists in SharePoint 2013 - Why You Need to Pay Attention to Thresholds and Metadata

1 Week 6 - What Well Be Working On This Week In th.docx
1 Week 6 - What Well Be Working On This Week  In th.docx1 Week 6 - What Well Be Working On This Week  In th.docx
1 Week 6 - What Well Be Working On This Week In th.docx
dorishigh
 
Siebel CTMS Best Practices
Siebel CTMS Best PracticesSiebel CTMS Best Practices
Siebel CTMS Best Practices
Perficient
 
Sql interview question part 5
Sql interview question part 5Sql interview question part 5
Sql interview question part 5
kaashiv1
 
Performance Tuning for Visualforce and Apex
Performance Tuning for Visualforce and ApexPerformance Tuning for Visualforce and Apex
Performance Tuning for Visualforce and Apex
Salesforce Developers
 

Similar to Working with Large Lists in SharePoint 2013 - Why You Need to Pay Attention to Thresholds and Metadata (20)

Managing lists and libraries with more than 5000 items
Managing lists and libraries with more than 5000 itemsManaging lists and libraries with more than 5000 items
Managing lists and libraries with more than 5000 items
 
Optimize access
Optimize accessOptimize access
Optimize access
 
1 Week 6 - What Well Be Working On This Week In th.docx
1 Week 6 - What Well Be Working On This Week  In th.docx1 Week 6 - What Well Be Working On This Week  In th.docx
1 Week 6 - What Well Be Working On This Week In th.docx
 
Introduction to Lists SharePoint 2010 - EPC Group
Introduction to Lists SharePoint 2010  - EPC GroupIntroduction to Lists SharePoint 2010  - EPC Group
Introduction to Lists SharePoint 2010 - EPC Group
 
Filtered Indexes In Sql 2008
Filtered Indexes In Sql 2008Filtered Indexes In Sql 2008
Filtered Indexes In Sql 2008
 
Tableau Basic Questions
Tableau Basic QuestionsTableau Basic Questions
Tableau Basic Questions
 
Transaction
TransactionTransaction
Transaction
 
Siebel CTMS Best Practices
Siebel CTMS Best PracticesSiebel CTMS Best Practices
Siebel CTMS Best Practices
 
Database Basics
Database BasicsDatabase Basics
Database Basics
 
SQL Tunning
SQL TunningSQL Tunning
SQL Tunning
 
SharePoint 2010 list improvements
SharePoint 2010 list improvementsSharePoint 2010 list improvements
SharePoint 2010 list improvements
 
Learning How to Shape and Configure an OData Service for High Performing Web ...
Learning How to Shape and Configure an OData Service for High Performing Web ...Learning How to Shape and Configure an OData Service for High Performing Web ...
Learning How to Shape and Configure an OData Service for High Performing Web ...
 
ASP.NET 10 - Data Controls
ASP.NET 10 - Data ControlsASP.NET 10 - Data Controls
ASP.NET 10 - Data Controls
 
Ebook5
Ebook5Ebook5
Ebook5
 
Sql interview question part 5
Sql interview question part 5Sql interview question part 5
Sql interview question part 5
 
Views
ViewsViews
Views
 
Performance Tuning for Visualforce and Apex
Performance Tuning for Visualforce and ApexPerformance Tuning for Visualforce and Apex
Performance Tuning for Visualforce and Apex
 
Bi requirements checklist
Bi requirements checklistBi requirements checklist
Bi requirements checklist
 
Chapter12 (1)
Chapter12 (1)Chapter12 (1)
Chapter12 (1)
 
Sas training in hyderabad
Sas training in hyderabadSas training in hyderabad
Sas training in hyderabad
 

Recently uploaded

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 

Working with Large Lists in SharePoint 2013 - Why You Need to Pay Attention to Thresholds and Metadata

  • 1. Working with Large Lists in SharePoint 2013 WHY YOU NEED TO PAY ATTENTION TO THRESHOLDS AND METADATA
  • 2. Steve Stewart Accudata Systems, Inc. 7906 North Sam Houston Pkwy W | Suite 300 | Houston, TX 77064 Direct: 281.897.6394 | Fax: 281.897.5001 www.accudatasystems.com blog.get-spsteve.com @SharePointFly sstewart@accudatasystems.com
  • 3. Services we offer: • Advisory Services • Data Center Infrastructure • Security and Mobility • Managed Services • Enterprise Applications • Network • Unified Communications • Assessment & Compliance
  • 4. Feedback & Other Important Stuff  Use #HoustonSPTF  Register to use the Mobile App at SPTF2015.com to give presentation feedback  Afternoon Reception runs from 2:45-3:30 p.m. in the Exhibit Hall. Refreshments are free and there will be prizes!
  • 5. Have you ever seen this?
  • 6. List View Threshold  The List View Threshold is set by default to 5,000 items for a web application in Central Administration  Administrators with special permissions have a default list view threshold of 20,000 items  Must be administrator on local server and on farm  Several other thresholds also set (List View Lookup, Permission, etc.)  Administrative Window – Time period for working on large lists without limit
  • 7. Central Admin - Resource Throttling Resource throttling settings are accessed in Central Admin from Application Management>Manage Web Applications>Web Application>Settings
  • 9. Some Important Notes  The List View Threshold calculation includes items in the Recycle Bin. If you delete 1,000 items, they still count against the item limit until they are deleted from the recycle bin.  Search is not affected by the threshold. You can always search for items and get results. Even if the list is well over the threshold  A common misunderstanding is that a view that has been filtered to contain less than the threshold number of items will display items when the list exceeds the threshold. Only views filtered with indexed columns work this way.
  • 10. What Actions are Blocked When the Threshold is Exceeded? Operation Description Add/remove/update a list column All columns, including lookup and calculated columns, in addition to many kinds of updates, such as a type change or a uniqueness change. Some updates, such as a name change, are not blocked because they do not affect every item in the list. Add/remove/Update a list content type Affects every item in the list so it is blocked for any list that has more items than the list view threshold. Create/remove an index Affects every item in the list so it is blocked for any list that has more items than the list view threshold. Manage files which have no checked-in version A non-indexed recursive query that fails for any list that has more items than the list view threshold. Non-indexed recursive queries Includes filters and some sorts. This operation fails when the list size is greater than the list view threshold. Because there is no index, it does a full scan against the entire list. Also, it returns all items, and it ignores folders. Cross list query Includes queries by the Content Query Web Part and follows the list view threshold setting for auditors and administrators, which by default is 20,000 items. If the operation involves more than 20,000 items, the query fails.
  • 11. Blocked Operations, cont. Operation Description Lookup columns that enforce relationship behavior You cannot create lookup columns that enforce relationship behavior when the list that is referenced contains more items than the list view threshold. Deleting a list Affects every item in the list so it is blocked for any list that has more items than the list view threshold. Deleting a site If the sum of all items in a site is greater than the list view threshold, deleting the site is prevented because it affects too many items. Save list as template with data Affects every item in the list so it is blocked for any list that has more items than the list view threshold. Showing totals in list views Performs a query against every item in the list so it is blocked for any list that has more items than the list view threshold. Enable/disable attachments in a list Affects every item in the list so it is blocked for any list that has more items than the list view threshold.
  • 12. What’s Happening in a Nutshell  List View Threshold is a SQL limitation, not a SharePoint limitation  To ensure accurate updates without affecting other users accessing the same table, SQL locks the row being updated  When more than 5,000 rows need to be locked due to the same query, it is more efficient for SQL to lock the entire table … that is all the rows.  It’s there to ensure optimized performance for ALL users  So think of it this way. SQL normally eats one tiny piece of pie at a time except when there are more than 5,000 pieces ordered. In this case, SQL eats the entire pie!
  • 13. “ ” So, just up the number. SharePoint can handle it right? Actually, it’s not really a good idea to increase the threshold number.
  • 14. List View Threshold Throughput 0 5 10 15 20 25 30 35 40 45 50 5,000 10000 50000 100000 Request per Second ListViewThreshold RPS* *Values are approximate and for illustration only. As the size of the list threshold increases, the number of requests per second SharePoint can handle decreases
  • 15. Does this mean my list can’t be larger than 5,000 items? Of course not. Feature Max Value Type Number of items in a list or library 30 million Supported Limit Size of an individual file item or item attachment 2 Gigabytes Boundary
  • 16. Boundaries, Thresholds, and Supported Limits  Boundary - An absolute that cannot be exceeded by design  Threshold - A value that is set to a default and cannot be exceeded unless the default is changed  Supported Limit - Values determined by testing and which represent known limits of the product. Exceeding these limits may result in performance degradation, errors, and other problems.
  • 17. Thresholds and Limits Threshold or Limit Default value Description List View Threshold 5,000 Specifies the maximum number of list or library items that a database operation, such as a query, can process at one time. Operations that exceed this limit are blocked. To give you time to make alternative plans, You are warned on the List Settings page when your list has exceeded 3,000 items. The warning contains a help link to this topic. Unique permissions limit 50,000 Specifies the maximum number of unique permissions allowed for a list or library. Every time you break the inheritance of permissions for an item or folder, it is counted as 1 unique permission toward this limit. If you try to add an item that would lead to exceeding this limit, you are prevented from doing so. Row size limit 6 Specifies the maximum number of table rows internal to the database used for a list or library item. To accommodate wide lists with many columns, each item is wrapped over several internal table rows, up to 6 rows and up to a total of 8,000 bytes (excluding attachments). For example, if you have a list with many small columns, one that contains hundreds of Yes/No columns, then you could exceed this limit, in which case you would not be able to add more Yes/No columns to the list, but you still may be allowed to add columns of a different type. Note Administrators can only set this limit by using the object model, not through the user interface.
  • 18. Thresholds and Limits Threshold or Limit Default value Description List View Lookup Threshold 8 Specifies the maximum number of join operations, such as those based on lookup, Person/Group, or workflow status columns. If the query uses more than eight columns, the operation is blocked. However, it is possible to programmatically select which columns to use by using maximal view, which can be set through the object model. List View Threshold size for auditors and administrators 20,000 Specifies the maximum number of list or library items that a database operation, such as a query, can process at one time when performed by an auditor or administrator with appropriate permissions. This setting works in conjunction with Allow Object Model Override. Allow Object Model Override Y Specifies whether or not developers can perform database operations, such as queries, that request an override of the List View Threshold to the higher limit specified by the List View Threshold for auditors and administrators. An administrator must enable an object model override, and then developers with appropriate permission may then programmatically request that their query use the higher List View Threshold to take advantage of it.
  • 19. Daily Administrative Time Window  Specifies a time period during which resource thresholds and limits are ignored. An administrator can configure a time window during "off-peak" hours in 15 minute increments and up to 24 hours, such as, 6:00 PM to 10:00 PM or 1:30 AM to 5:15 AM.  Note A database operation or query started within the daily time window continues until completion (or failure) even if it doesn't finish within the specified time window.  Time can be used to run queries and conduct other administrative tasks such as indexing columns, creating views and deleting large numbers of items.  The administrative window is set in Central Administration for a web application
  • 20. Is the Threshold Really All or Nothing for the Web Application  Yes and no.  You can only change the threshold for the entire web application using Central Admin  But, you can also use PowerShell to disable the threshold for a single list $web = Get-SPWeb https://domain.com/web/ $list = $web.Lists[“List Name"] $list.EnableThrottling = $false Additional PowerShell commands are available for setting List thresholds at the web application level. See MSDN for more information: https://msdn.microsoft.com/en- us/library/office/Microsoft.SharePoint.Administration.SPWebApplication_properties.aspx
  • 21. REST API  You can use the REST API to quickly view the list view threshold settings for a particular site collection without opening Central Admin by accessing the following URL:  <site collection URL>/_api/site/maxItemsPerThrottledOperation   Results will be returned that look like this:
  • 23. Views with Indexed Columns Supported Column Types Unsupported Column Types Single line of text Multiple lines of text Choice (single value) Choice (multi-valued) Number Calculated Currency Hyperlink or Picture Date and Time Custom Columns Lookup (single value) Indexing the column in the other list or library does not improve performance of the lookup operation. Lookup (multi-valued) Person or Group (single value) Person or Group (multi-valued) Yes/No External data Managed Metadata
  • 24. Filter Ideas To filter by: Index the: For example: Recently changed items Modified column To view only items that have changed in the past week, apply the filter Modified is greater than [Today]-7. New items Created column To create a view of only those items that were added in the past week, apply the filter Created is greater than [Today]-7. My items Created By column To create a view of only those items that you added, apply the filter Created By is equal to [Me]. Items due today Due Date column (created by you in a list or library) To create a view of only those items with a due date of today, apply the filter Due Date is equal to [Today]. Discussion board updates Last Updated column To create a view of only the discussions that were updated in the past month, apply the filter Last Updated is greater than [Today]-30. Archive files in a document library Date Modified To create a view of documents not changed since the year 2006 (that you then want to archive), apply the filter Date Modified is less than December 31st 2006. Find a subset of financial data Region, Year (as two simple indexes) To create a view of financial data for the NorthEast region in 2008, apply the filter Region equals "NE" AND Year equals 2008 because the Region column is likely to have less values than the Year column.
  • 25. Index a Column and Create a View Based on it (Demo) 1. Go to list > List Settings 2. Scroll to the bottom of the columns list 3. Click Indexed columns 4. Click Create a new index 5. Select Columns to index 6. Normally, you will pick only a primary column from the drop down. However, you may with some fields select a secondary to create a compound Index. For example, Modified by and Modified (date). Compound indexes can be used with managed metadata. 7. Click Create (max 20 indexes) 8. Indexed columns appear in View builder filter area in the Indexed Columns group.
  • 27. Using InfoPath to Break the Rules Trick: You cannot use a calculated column as an indexed column, but you can calculate a value in InfoPath and use a rule to transfer the value to a supported column type.
  • 28. Use Key Filters  Key Filters work with managed metadata to help parse a list  Prerequisite for using this feature is activation of the Metadata Navigation and Filtering site feature.  Fields available for Key Filtering are:  Content Type  Choice Field  Managed Metadata Field  Person or Group Field  Date and Time Field  Number Field
  • 29. Steps to Create a Key Filter 1. Go to List Settings 2. Click Metadata Navigation Settings 3. Add Navigation Hierarchies 4. Add Key Filters 5. Click OK
  • 30. Folders – mainly for libraries  You can divide items into folders to get around view thresholds  This is best suited for document libraries  Either use initial view with all folders visible or create navigation (quick launch) to go directly to folder  Use Per Location View Settings to customize default and available views in folders  In lists there is no default way to move items into folders after creation  Document Center
  • 31. Key Filters Get Us Part Way There, But an Index is Still Needed
  • 32. Datasheet 1. From the Ribbon, click Create View 2. Select Datasheet View as the view type 3. Under View Settings, expand folders and select Show all items without folders 4. Expand Item Limit and select either Display All Items or Limit the total number of items returned to option. 5. Type 30 in the box below it (this number must be below the list view threshold) 6. Save the view 7. Click OK when you receive the warning about flat views. 8. Edit the view again, now change
  • 33. Data Sheet View Settings  Remember unless you are referencing indexed columns, no filters, sorting, totals, etc., can be selected.
  • 34. Export to Excel  From the List/Library Ribbon click Export to Excel  You can use the Export to Excel feature to create a read only version of your list which can be used to perform queries  Remove unwanted columns from the Excel worksheet without affecting your view or the list – columns will not reappear upon refresh  Use the Refresh button to get new additions to the list.  You can use Excel tools to generate reports, etc.
  • 35. Open in Access  You need Access 2013 to use this feature in SharePoint 2013  From the Ribbon click Open in Access to create a client that can be used for:  CRUD (Create, Read, Update, Delete)  Creating forms  Queries  Reporting
  • 36. Even More Powerful – Access Views 1. From the Ribbon select Create View 2. Select Access View
  • 37. Select View Style 3. Select an Access View Style 4. Click OK 5. Make adjustments to the form and list design, as desired. 6. Use Access to update lists.
  • 38. Save the Access View App Anywhere  Users can update your list from another site collection  You can control permission based on the location where the document is saved  Think about adding a link to the app on the Quick Launch  Users can download a copy for more efficiency
  • 40. References  Designing large lists and maximizing list performance  Techniques for managing large lists  Manage lists and libraries with many items  SharePoint 2010/2013: List view Lookup threshold uncovered  Software Boundaries and Limits for SharePoint 2013
  • 41.  © Copyright 2015 Stephen D. Stewart. All rights reserved.

Editor's Notes

  1. Talk about when you see these. Then let audience know you will guide them in some ways to effectively work with large lists.
  2. Explain Thresholds – (default values – best practice); Supported Limits (may or may not work past this); Boundaries – Absolutes, can’t go over this, even if we want to
  3. For example, the 2 Gb file upload size limit is a boundary, the 5,000 list item view limit is a threshold and the 200 Gb size of a content database is a supported limit.
  4. Talk about AND versus OR on filters
  5. Demonstrate how to create an Indexed Column (Create Catalog Value) Demonstrate how this can be used to create a view (High Value – Filter on Catalog Value over $10)
  6. Show an example if you have time – Create concat column of Catalog and Year in InfoPath
  7. Add a key filter (Catalog Value) and then go to the default view to show the “Only Newest Items” Message. Use Key filter with index Comment about list design and thinking about your columns in advance
  8. Demo: Create Key Filters for Catalog Value and Catalog Year Filter on Catalog Value at least $10.00 and catalog year equals 2015
  9. Can use content and structure manager to move items to folders
  10. Point out how adding the filter presented most recent results Filter CV 500 – Note no results and then go to B and do same filter – Note the Country Index filter in place already
  11. Demonstrate the instructions above.
  12. DEMO Show export to excel from view that exceeds threshold. Remove columns Refresh Add Slicer
  13. Demo Show open with access Update an item Delete an item
  14. Create Access View Select Split View Save DB Ensure Control Wizard is active for access controls Add Button to Form to Add Records Add new item
  15. Go through the process of uploading the app to a new site collection. Add link on quick launch in secondary site. Show how items can be updated from there.