Working with Large Lists in SharePoint 2013 - Why You Need to Pay Attention to Thresholds and Metadata

1. Working with Large Lists in SharePoint 2013 WHY YOU NEED TO PAY ATTENTION TO THRESHOLDS AND METADATA

2. Steve Stewart Accudata Systems, Inc. 7906 North Sam Houston Pkwy W | Suite 300 | Houston, TX 77064 Direct: 281.897.6394 | Fax: 281.897.5001 www.accudatasystems.com blog.get-spsteve.com @SharePointFly sstewart@accudatasystems.com

3. Services we offer: • Advisory Services • Data Center Infrastructure • Security and Mobility • Managed Services • Enterprise Applications • Network • Unified Communications • Assessment & Compliance

4. Feedback & Other Important Stuff  Use #HoustonSPTF  Register to use the Mobile App at SPTF2015.com to give presentation feedback  Afternoon Reception runs from 2:45-3:30 p.m. in the Exhibit Hall. Refreshments are free and there will be prizes!

5. Have you ever seen this?

6. List View Threshold  The List View Threshold is set by default to 5,000 items for a web application in Central Administration  Administrators with special permissions have a default list view threshold of 20,000 items  Must be administrator on local server and on farm  Several other thresholds also set (List View Lookup, Permission, etc.)  Administrative Window – Time period for working on large lists without limit

7. Central Admin - Resource Throttling Resource throttling settings are accessed in Central Admin from Application Management>Manage Web Applications>Web Application>Settings

8. CA Resource Throttling Settings

9. Some Important Notes  The List View Threshold calculation includes items in the Recycle Bin. If you delete 1,000 items, they still count against the item limit until they are deleted from the recycle bin.  Search is not affected by the threshold. You can always search for items and get results. Even if the list is well over the threshold  A common misunderstanding is that a view that has been filtered to contain less than the threshold number of items will display items when the list exceeds the threshold. Only views filtered with indexed columns work this way.

10. What Actions are Blocked When the Threshold is Exceeded? Operation Description Add/remove/update a list column All columns, including lookup and calculated columns, in addition to many kinds of updates, such as a type change or a uniqueness change. Some updates, such as a name change, are not blocked because they do not affect every item in the list. Add/remove/Update a list content type Affects every item in the list so it is blocked for any list that has more items than the list view threshold. Create/remove an index Affects every item in the list so it is blocked for any list that has more items than the list view threshold. Manage files which have no checked-in version A non-indexed recursive query that fails for any list that has more items than the list view threshold. Non-indexed recursive queries Includes filters and some sorts. This operation fails when the list size is greater than the list view threshold. Because there is no index, it does a full scan against the entire list. Also, it returns all items, and it ignores folders. Cross list query Includes queries by the Content Query Web Part and follows the list view threshold setting for auditors and administrators, which by default is 20,000 items. If the operation involves more than 20,000 items, the query fails.

11. Blocked Operations, cont. Operation Description Lookup columns that enforce relationship behavior You cannot create lookup columns that enforce relationship behavior when the list that is referenced contains more items than the list view threshold. Deleting a list Affects every item in the list so it is blocked for any list that has more items than the list view threshold. Deleting a site If the sum of all items in a site is greater than the list view threshold, deleting the site is prevented because it affects too many items. Save list as template with data Affects every item in the list so it is blocked for any list that has more items than the list view threshold. Showing totals in list views Performs a query against every item in the list so it is blocked for any list that has more items than the list view threshold. Enable/disable attachments in a list Affects every item in the list so it is blocked for any list that has more items than the list view threshold.

12. What’s Happening in a Nutshell  List View Threshold is a SQL limitation, not a SharePoint limitation  To ensure accurate updates without affecting other users accessing the same table, SQL locks the row being updated  When more than 5,000 rows need to be locked due to the same query, it is more efficient for SQL to lock the entire table … that is all the rows.  It’s there to ensure optimized performance for ALL users  So think of it this way. SQL normally eats one tiny piece of pie at a time except when there are more than 5,000 pieces ordered. In this case, SQL eats the entire pie!

13. “ ” So, just up the number. SharePoint can handle it right? Actually, it’s not really a good idea to increase the threshold number.

14. List View Threshold Throughput 0 5 10 15 20 25 30 35 40 45 50 5,000 10000 50000 100000 Request per Second ListViewThreshold RPS* *Values are approximate and for illustration only. As the size of the list threshold increases, the number of requests per second SharePoint can handle decreases

15. Does this mean my list can’t be larger than 5,000 items? Of course not. Feature Max Value Type Number of items in a list or library 30 million Supported Limit Size of an individual file item or item attachment 2 Gigabytes Boundary

16. Boundaries, Thresholds, and Supported Limits  Boundary - An absolute that cannot be exceeded by design  Threshold - A value that is set to a default and cannot be exceeded unless the default is changed  Supported Limit - Values determined by testing and which represent known limits of the product. Exceeding these limits may result in performance degradation, errors, and other problems.

17. Thresholds and Limits Threshold or Limit Default value Description List View Threshold 5,000 Specifies the maximum number of list or library items that a database operation, such as a query, can process at one time. Operations that exceed this limit are blocked. To give you time to make alternative plans, You are warned on the List Settings page when your list has exceeded 3,000 items. The warning contains a help link to this topic. Unique permissions limit 50,000 Specifies the maximum number of unique permissions allowed for a list or library. Every time you break the inheritance of permissions for an item or folder, it is counted as 1 unique permission toward this limit. If you try to add an item that would lead to exceeding this limit, you are prevented from doing so. Row size limit 6 Specifies the maximum number of table rows internal to the database used for a list or library item. To accommodate wide lists with many columns, each item is wrapped over several internal table rows, up to 6 rows and up to a total of 8,000 bytes (excluding attachments). For example, if you have a list with many small columns, one that contains hundreds of Yes/No columns, then you could exceed this limit, in which case you would not be able to add more Yes/No columns to the list, but you still may be allowed to add columns of a different type. Note Administrators can only set this limit by using the object model, not through the user interface.

18. Thresholds and Limits Threshold or Limit Default value Description List View Lookup Threshold 8 Specifies the maximum number of join operations, such as those based on lookup, Person/Group, or workflow status columns. If the query uses more than eight columns, the operation is blocked. However, it is possible to programmatically select which columns to use by using maximal view, which can be set through the object model. List View Threshold size for auditors and administrators 20,000 Specifies the maximum number of list or library items that a database operation, such as a query, can process at one time when performed by an auditor or administrator with appropriate permissions. This setting works in conjunction with Allow Object Model Override. Allow Object Model Override Y Specifies whether or not developers can perform database operations, such as queries, that request an override of the List View Threshold to the higher limit specified by the List View Threshold for auditors and administrators. An administrator must enable an object model override, and then developers with appropriate permission may then programmatically request that their query use the higher List View Threshold to take advantage of it.

19. Daily Administrative Time Window  Specifies a time period during which resource thresholds and limits are ignored. An administrator can configure a time window during "off-peak" hours in 15 minute increments and up to 24 hours, such as, 6:00 PM to 10:00 PM or 1:30 AM to 5:15 AM.  Note A database operation or query started within the daily time window continues until completion (or failure) even if it doesn't finish within the specified time window.  Time can be used to run queries and conduct other administrative tasks such as indexing columns, creating views and deleting large numbers of items.  The administrative window is set in Central Administration for a web application

20. Is the Threshold Really All or Nothing for the Web Application  Yes and no.  You can only change the threshold for the entire web application using Central Admin  But, you can also use PowerShell to disable the threshold for a single list $web = Get-SPWeb https://domain.com/web/ $list = $web.Lists[“List Name"] $list.EnableThrottling = $false Additional PowerShell commands are available for setting List thresholds at the web application level. See MSDN for more information: https://msdn.microsoft.com/en- us/library/office/Microsoft.SharePoint.Administration.SPWebApplication_properties.aspx

21. REST API  You can use the REST API to quickly view the list view threshold settings for a particular site collection without opening Central Admin by accessing the following URL:  <site collection URL>/_api/site/maxItemsPerThrottledOperation   Results will be returned that look like this:

22. Demonstrations

23. Views with Indexed Columns Supported Column Types Unsupported Column Types Single line of text Multiple lines of text Choice (single value) Choice (multi-valued) Number Calculated Currency Hyperlink or Picture Date and Time Custom Columns Lookup (single value) Indexing the column in the other list or library does not improve performance of the lookup operation. Lookup (multi-valued) Person or Group (single value) Person or Group (multi-valued) Yes/No External data Managed Metadata

24. Filter Ideas To filter by: Index the: For example: Recently changed items Modified column To view only items that have changed in the past week, apply the filter Modified is greater than [Today]-7. New items Created column To create a view of only those items that were added in the past week, apply the filter Created is greater than [Today]-7. My items Created By column To create a view of only those items that you added, apply the filter Created By is equal to [Me]. Items due today Due Date column (created by you in a list or library) To create a view of only those items with a due date of today, apply the filter Due Date is equal to [Today]. Discussion board updates Last Updated column To create a view of only the discussions that were updated in the past month, apply the filter Last Updated is greater than [Today]-30. Archive files in a document library Date Modified To create a view of documents not changed since the year 2006 (that you then want to archive), apply the filter Date Modified is less than December 31st 2006. Find a subset of financial data Region, Year (as two simple indexes) To create a view of financial data for the NorthEast region in 2008, apply the filter Region equals "NE" AND Year equals 2008 because the Region column is likely to have less values than the Year column.

25. Index a Column and Create a View Based on it (Demo) 1. Go to list > List Settings 2. Scroll to the bottom of the columns list 3. Click Indexed columns 4. Click Create a new index 5. Select Columns to index 6. Normally, you will pick only a primary column from the drop down. However, you may with some fields select a secondary to create a compound Index. For example, Modified by and Modified (date). Compound indexes can be used with managed metadata. 7. Click Create (max 20 indexes) 8. Indexed columns appear in View builder filter area in the Indexed Columns group.

26. Create Index Columns

27. Using InfoPath to Break the Rules Trick: You cannot use a calculated column as an indexed column, but you can calculate a value in InfoPath and use a rule to transfer the value to a supported column type.

28. Use Key Filters  Key Filters work with managed metadata to help parse a list  Prerequisite for using this feature is activation of the Metadata Navigation and Filtering site feature.  Fields available for Key Filtering are:  Content Type  Choice Field  Managed Metadata Field  Person or Group Field  Date and Time Field  Number Field

29. Steps to Create a Key Filter 1. Go to List Settings 2. Click Metadata Navigation Settings 3. Add Navigation Hierarchies 4. Add Key Filters 5. Click OK

30. Folders – mainly for libraries  You can divide items into folders to get around view thresholds  This is best suited for document libraries  Either use initial view with all folders visible or create navigation (quick launch) to go directly to folder  Use Per Location View Settings to customize default and available views in folders  In lists there is no default way to move items into folders after creation  Document Center

31. Key Filters Get Us Part Way There, But an Index is Still Needed

32. Datasheet 1. From the Ribbon, click Create View 2. Select Datasheet View as the view type 3. Under View Settings, expand folders and select Show all items without folders 4. Expand Item Limit and select either Display All Items or Limit the total number of items returned to option. 5. Type 30 in the box below it (this number must be below the list view threshold) 6. Save the view 7. Click OK when you receive the warning about flat views. 8. Edit the view again, now change

33. Data Sheet View Settings  Remember unless you are referencing indexed columns, no filters, sorting, totals, etc., can be selected.

34. Export to Excel  From the List/Library Ribbon click Export to Excel  You can use the Export to Excel feature to create a read only version of your list which can be used to perform queries  Remove unwanted columns from the Excel worksheet without affecting your view or the list – columns will not reappear upon refresh  Use the Refresh button to get new additions to the list.  You can use Excel tools to generate reports, etc.

35. Open in Access  You need Access 2013 to use this feature in SharePoint 2013  From the Ribbon click Open in Access to create a client that can be used for:  CRUD (Create, Read, Update, Delete)  Creating forms  Queries  Reporting

36. Even More Powerful – Access Views 1. From the Ribbon select Create View 2. Select Access View

37. Select View Style 3. Select an Access View Style 4. Click OK 5. Make adjustments to the form and list design, as desired. 6. Use Access to update lists.

38. Save the Access View App Anywhere  Users can update your list from another site collection  You can control permission based on the location where the document is saved  Think about adding a link to the app on the Quick Launch  Users can download a copy for more efficiency

39. Questions

40. References  Designing large lists and maximizing list performance  Techniques for managing large lists  Manage lists and libraries with many items  SharePoint 2010/2013: List view Lookup threshold uncovered  Software Boundaries and Limits for SharePoint 2013