1.
Tips and tricks for complex
migrations to SharePoint Online
Office 365 and SharePoint Connect – Community Edition, October 2019

2.
Are you considering, preparing or executing a
large-scale migration to Office 365?

3.
Expect to run into some problems along the way

4.
This session aims to help you with tips and
tricks to increase your project’s success rate

5.
Are you in the right session?
• Expect a mix of technical and tactical topics that will help you to
design and execute your large-scale migration project
• Large-scale migrations as in Terabytes and Petabytes
• We’re talking documents, file shares and SharePoint, not Exchange
• Some live-demo’s and walk-throughs
• Not really a business session
• Up-tempo due to time constraints (45 minutes).

6.
Agenda
• Introductions and objectives
• Typical migration challenges
• Migration means rationalization
• Moving from classic to modern
• Choosing the right migration tool
• Bulk migrations using pipelines
• Maximize automation for large-scale

7.
A brief personal introduction
• Working in technology since 1999
• Architect for Wortell (via ETTU) since March 2015
• Previously: Pink Elephant, PinkRoccade, Getronics,
Ordina & KPN Consulting
• Crazy about technology that connects people
• Focus on Business & IT Alignment:
 Governance & security
 Use of Best practices
 Certified Microsoft Office 365 technology expert
 Addicted to large-scale migrations 
Andries.den.Haan@wortell.nl
Twitter: @awdhaan

8.
Organizations have built up large collections
of “Dark data” over the years…

9.
Typical challenges for large-scale migrations
• Organizations with multiple divisions and offices
• File shares existing for a long time, supporting various processes
• Also SharePoint Server environments with multiple site collections
• Variety in file structures and authorizations
• Difficult to track responsible owner (if any)
• Decisions on what to do with the data (migrate, dispose, quarantine or …)
• Protecting sensitive data (PII, financial or even medical)
• Designing a futureproof target architecture in the cloud
• Remapping data from source
• The sheer volume of data can result in longer processing time.

10.
What about business continuity?

11.
Typical project approach for large-scale migrations*
Discover
• Identify stakeholders
and requirements
• Prepare tooling and
access to data sources
• Conduct stakeholder
interviews
• Start scanning data
sources
• ….
Analyze Rationalize Design Deploy
• Implement target
architecture
• Deploy migration
solution
• Perform test
migrations for
technical validity and
to gain performance
metrics
• Plan and estimate the
migration
• …
Migrate
• Perform continuous
migrations
• Adjust and optimize
• Resolve known issues
and investigate new
• Progress reporting
• …
• Analyze scan results
• Review functional and
security requirements
• Determine potential
target locations
• Define business rules
to classify data
• …
• Classify data via
defined business rules
• Identify migration and
relocation scenarios
• Estimate migration
throughput
• Determine ordering
and priority
• …
• Envision future state
• Design target
architecture
• Determine governance
guidelines and security
baseline
• Prepare migration and
onboarding process
• …
* Technical perspective only. Parallel activities commonly include adoption and training

12.
Engage your stakeholders to set and keep
things in motion

13.
To the Cloud
and beyond
Archive,
dispose or …
Application or HMUD?
File types
Ownership
Age
Sensitivity
Rationalizing your data is hard work

14.
Tools you can use to support inventory and rationalization
Tool Source Usage
NTFS Permissions Reporter File share Analyze permission structures
Microsoft Inventory Manager
File share, SharePoint, Box,
OneDrive, Google Drive
Fasttrack tool: Inventory and issue reporting
(use, file type and aging for file share data)
Jam Software Tree Size Pro File share Analyze data (storage) structures and aging
SharePoint Modernization
Scanner
SharePoint
Provides insights on readiness and issues for
modernizing classic sites
SharePoint Migration
Assessment Tool
SharePoint
Identify issues for SharePoint Server 2013
source environments
Veritas Data Insights File share
Inventory and analysis. Can extract
presumable ownership

15.
NTFS Permissions Reporter can help to
determine permission structures

16.
Demo: Microsoft Inventory Manager

17.
Your target architecture should be futureproof
• Leverage the goodness of the Microsoft Cloud:
 Better control of data lifecycle using retention policies and labels
 Increased protection of sensitive data using DLP and sensitivity labels a.o.
 Guided provisioning and flexibility through templating
 Governance for policies & standards, operations and innovation
• Easier to identify ownership (person, team, division, org)
• Flexible site/team structure to accommodate for
changes in the organization (Hub sites FTW)
• …

18.
Mapping source and targets
• Avoid granular permissions, follow the Office
365 Group structure and split into multiple
groups/teams if required
• Use sensitivity labels for item-level protection
• Look for data and permission patterns
• Design a naming convention for sites/teams
and then design it again after validation with
your stakeholders
• Only migrate active data
• Keep it simple.
File share Finance division
Teams
Corporate Finance
Admin
Contracts
Reports
Archive
Incoming

19.
Choosing between migration tools
Microsoft SharePoint Migration tool:
• Basic set of migration features
• Support for one direction: on-premises-cloud
• Basic PowerShell module
• Currently lacking support for mapping, migration of
specific content
• Suitable for basic migrations
• Easily scalable
• Free to use
• Improving with each new update.
ShareGate:
• Lot’s of features for migrations and reporting
• Rich PowerShell module
• On-premises-on-premises, on-premises-cloud,
cloud-on-premises, cloud-cloud
• Map and migrate specific data
• Copy, flatten and merge site objects
• Suitable for complex migrations
• Export data to file system
• Migrate term sets, web parts and other artefacts
• Metadata enrichment during migrations
• Reasonable pricing and excellent support service
• Improving with each new update.

20.
Performance characteristics when using the
SharePoint API for migration (Microsoft)
Type of metadata Examples Maximum
Light ISO files, video files 2 TB/day
Medium List items, Office files (~1.5MB) 1 TB/day
Heavy
List items with custom
columns, small files (~50kb)
250 GB /day
Rules of thumb:
• Large file size migrates faster than smaller ones
• Files migrate faster than objects and list items
• Actual throughput depends on local infrastructure, network speeds etc.

21.
A single ShareGate:Desktop instance typically
migrates 5-15 GB/hour with peaks up to 35 GB
• Sharegate is optimized to run with 4 cores (64 concurrent threads)
• Also consider pre- and postmigration activities:
 SharePoint site/Team provisioning (fair)
 Applying configuration (depends on complexity)
 Assigning Owner and Membership (fast)
 Reporting and logging (fast)
• Network dependencies:
 1 GB via 5/1 Mbps ADSL line takes 16 minutes
 1 GB via 1 GBPS line takes 8 seconds
• Number simultaneous import jobs to each separate Content Database
(contact Microsoft to pre-provision content databases for > 1TB migrations)
• Provisioning sites and Teams in bulk prior to starting data migrations.

22.
Improving migration performance in ShareGate:Desktop
• ShareGate:Desktop 12.0 can now register as
an Azure application
• This allows ShareGate to identify to Office
365 that it is running operations on your
tenant through an access token
• This will reduce throttling on tasks between
Sharegate and Office 365
• Also grants ShareGate rights to use certain
Microsoft functionality, such as the Graph API
• Global Administrator role required for
consent of the permissions.

23.
Azure Active Directory App registration

24.
Modernizing classic SharePoint sites
• Converting classic artefacts to modern experiences
• Why? Duh! Modern is simple, responsive and benefits from ongoing
Microsoft investments
• Some basic principles:
 When moving to the cloud, only migrate data
 When already in the Cloud, modernize what’s classic
• SharePoint Patterns and Practices (PnP) are your best friends:
 Guidance: http://aka.ms/sppnp-modernize
 Toolkit: https://github.com/SharePoint/sp-dev-modernization
• Start with gaining insights on modernization
readiness and issues.

25.
The Modernization Scanner is awesome
• Only for SharePoint Online
• Use via GUI or command line
• Scope to entire tenant, csv or
individual sites
• Configure specific scans or
perform a full scan
• Various authentication options
• Detailed reports in Excel and csv.

26.
Demo: Modernization Scanner

27.
Modernize by connecting to an Office 365 group
• Not supported for classic publishing portals
• Upon “Groupify”:
 A new Office 365 group is created, and connected to
the site A new modern home page is created and set
as home page
 Group owners are now the site collection admins
 Group owners are added to the site's Owners group
 Group members are added to the site's Members
group
• Options to perform “Groupification”:
 Self-service on the site
(“Connect to new Office 365 Group”)
 Via PnP PowerShell: Add-PnPOffice365GroupToSite.

28.
Demo: Modernizing by Groupification

29.
Options for modernizing SharePoint pages
PnP PowerShell conversion:
• Great for centrally managed scenarios
• Superfast and easy to deploy
• Integrates into a migration pipeline
• Mapping files for web parts and layouts
• Built-in logging.
PnP Page Transformation UI solution:
• Great for self-service scenarios
• Fully deploy using a provided script
• Best for non-customized environments
• Feedback option
• Enable or disable on a per site basis.

30.
Demos: SharePoint page conversion

31.
Main script
Create a migration pipeline with all required functions
{f} {f} {f} {f} {f}
csv input
Logging,
reporting
<xml>
<xml>
mapping
file(s)

32.
Designing a migration pipeline
• Design each function with the expected input variables
• Securely cache source and destination credentials
(encrypted xml, Windows Credential Manager)
• Accommodate for nesting of functions
(e.g. copy-data in sub sites)
• Handle errors if e.g. artefacts already exist
• Functions are called through a “main” script and fed by
input files (to-do, mappings, configs).

33.
Demo: Anatomy of a migration pipeline

34.
Complexity increases
when the numbers go up
• IT does not have the full insights on
ownership, and they can’t make decisions
for data at scale. And they shouldn’t
anyway!
• Bigger organizations have multiple
divisions each with different rhythms.
Planning migrations becomes difficult at
scale
• Complexity of merging or splitting data
and other transformations
• Multiple sources: File shares, SharePoint
• Processing time of all repeating tasks are
multiplied by the amount of data: e.g. 1
min. x 20.000 migrations already takes ~
14 days.

35.
For large-scale, even common actions take time
• Creating and continuously updating inventories
• Typical actions like creating mappings or input files
• Logging success and failure
• Scripting solutions for known issues
• Cleaning and customizing
• Liaising with data stewards to resolve data issues
• …

36.
Automate to epic proportions using a Migration Factory
• Build and (incrementally) refresh data source inventories
• Provide self-service to the business to:
 Determine and claim ownership
 Help resolve data related issues
 Enrich inventories with sensitivity info
 Prepare and approve decisions on what to do with the data
 Choose an appropriate migration scenario
 Allow for opt-in on scheduled migration batches
 (Dis)approve migration outcome
 Lock source environment and display banners
• Be flexible in supporting different business requirements*
• Scale up for increasing demand
• Optimize the process of solving problems
• Realtime reporting on progress and issues.

37.
Supporting multiple migration scenarios
• Each migration scenario may contain multiple steps to:
 Provision SharePoint Online sites or Teams (PnP)
 Apply specific configurations (Site Design, PnP)
 Migrate data (ShareGate:Desktop)
 Perform checks and compares
 Notify data stewards to approve migration outcome
 Report output of each step back to the central database
• PowerShell for a high degree of flexibility and extensibility
• Standardized by function approach and shared elements
• Use a repo for source control.

38.
High level design of a Migration Factory
Migration
database
Migration
Administrator
Stakeholders
SharePoint
2013
SharePoint
2010
Filers
Inventory clientInventory client
Migration
portal
C#js
Migration servers
PS
SharePoint Online
Microsoft Teams
Reporting
Migration
AIP-Scanner
Disposition
Inventory client
Decision n
Data
Owner(s)
Data
Steward(s)

39.
Walkthrough: Migration Factory portal

40.
Key takeaways of this session
• Complexity increases with increasing amounts of data
• The business owns the data, support them to rationalize
and facilitate them to migrate their own stuff
• large-scale is all about optimizing the process
• Current technology allow for a high degree of automation
• Keep things simple when the numbers go up. Move first,
adjust later! The nitty gritty stuff will slow you down
• The PnP Community is simply awesome!