Scribe insight 02 need for speed

1. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved.
Need For Speed
Message Queuing vs. Bulk & Upsert Capabilities

2. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 3
Session Abstract
There was a time when including message queuing as part of your
Insight integration design was the recommended design pattern to get
the most “bang for your buck”. But with new bulk and upsert
capabilities available in endpoints and in Insight now you can get better
performance by using those capabilities instead of message queuing.
This session will focus on when you would use bulk capability instead of
message queuing, and we will even teach you a thing or two about
using bulk.

3. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 4
The Pallet
Bulk Load
Multi
Threading
Parallel
Upsert PreSeek
Message
Queue

4. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 5
Dynamics CRM Bulk Load
Scribe Insight
Payload
(200
Rows of
Data)
Adapters
CRM

5. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 6
Bulk Setting
Dynamics CRM
only takes 250
at a time
Best for Insert
operations
SQL???
Challenges
with variable
steps

6. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 7
Scribe Fast Load – Process Threading
Workbench
Payload
(1000
Rows of
Data)
CRM Adapter
Target
CRM API
DLL
Dynamics CRM
TCP
Threads
Process
Threads

7. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 8
Parallel Workbench
0
500
1000
1500
2000
2500
3000
3500
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Worckbench to Error
CPU Error

8. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 9
Optimal Settings
Turn Off Any Plugins / Workflows
Let CRM automatically generate GUID (3% improvement in write)
Move Source to Microsoft Azure VM (15% over on Premise, and Amazon)
Run Workbench in Parallel 7.9
Run Workbench from Command Line
You will break CRM at pome point
Migration of Large Data into CRM

9. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 10
Parallel Operation Insight
Payload
(200
Rows of
Data)
Adapters
CRM
Payload
(200
Rows of
Data)
Adapters
CRM
Payload
(200
Rows of
Data)
Adapters
CRM
64
Times

10. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved.
Upsert
Scribe Insight Provide Keys via Xref or Lookup if Null
then Insert.
Moving to Server Side Operation with Microsoft Fall
Update 2015

11. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved.
Pre Seek - Option
We thought we were helping…….
SQL impact is very high in CRM 2011 and 2013
Turn on if not using target variables

12. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 13
Message Queuing
Transaction
Processing
Reserved
Processors
(7.9)
Message
Priority In the
Queue
Recovery /
Retry
Error
Management

13. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved.
Integration Patterns/Use Cases
One time Migration
Update Million Records a
day
Ongoing Transactions

14. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 15
Massive Data Migration
Stage SQL or Source the SQL directly from Old CRM
CRM On Prem
• Turn off Auditing
• Workflows
• Plugins
• SQL Database Indexing during
Read Microsoft White Paper
Stage Migration weeks ahead Then Build Integration

15. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 16
By The Numbers 100,000 Account Records
Design Rows Per Minute Time Complete
Queue -32 Message Processor
1 Workbench File
10 Workbench File
20 Workbench File
Scribe Online
Entities have different
performance
characteristics
Have a large sample –
10,000 records
minimum
Watch CRM server
CRM Online Limits 2
threads per organization

16. CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 17
Pierre’s Tips and Tricks
Use Staging Tables for Large Migration Jobs.
Resolve lookups references in the source query
Chain The Workbench files
Run multiple Workbench files
17