Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
#CRMUGSummit | #INreno15
#CRMUGSummit
Daniel Cai, KingswaySoft Inc.
SESSION CODE: ADC26
GETTING DATA INTO MICROSOFT DYNAMI...
#CRMUGSummit | #INreno15
INTRODUCTION - YOUR PRESENTER
 Daniel Cai
 Principal Developer @KingswaySoft
 7 Years of devel...
#CRMUGSummit | #INreno15
 Discuss the techniques that you can use to get
data into Microsoft Dynamics CRM faster
 Discus...
#CRMUGSummit | #INreno15
 SSIS Integration Toolkit is used in some occasions in this presentation to
discuss the techniqu...
#CRMUGSummit | #INreno15
 WHY?
 WHERE TO START FROM?
 HOW?
 Final Q&A
 Closing & Surveys
5
AGENDA
WHY?
#CRMUGSummit | #INreno15
 Web Service Interfaces are generally slow
– Dynamics CRM is no exception
 You are often constr...
WHERE TO START FROM?
#CRMUGSummit | #INreno15
 Identify Resources
– Technical Resources
 Get to know your infrastructure
– Networking
– Serve...
#CRMUGSummit | #INreno15
 Optimized Infrastructure
– Networking
– Firewall
– Security Policies
– Network Load Balancer
 ...
#CRMUGSummit | #INreno15
 CRM Data Import Wizard
 Message-driven Integration Solutions
– BizTalk
– Service Bus
– Message...
#CRMUGSummit | #INreno15
 Each tool works somewhat differently
 Establish a good understanding of your tools
– Know the ...
HOW?
#CRMUGSummit | #INreno15
MULTI-THREADING
Make sure to update your .config file in order to
overcome the connection limit o...
#CRMUGSummit | #INreno15
LEVERAGE MULTI-NODE CLUSTER
How to achieve this?
• CRM Server URL option in SSIS Integration Tool...
#CRMUGSummit | #INreno15
BULK API
 Bulk API is more beneficial when
network latency is high
 There is an API throttling ...
#CRMUGSummit | #INreno15
 The less fields, the better performance
 Some fields may require additional services calls usi...
#CRMUGSummit | #INreno15
 Why?
– Smaller SOAP message, which in turn results in less networking time
– Can avoid firing u...
#CRMUGSummit | #INreno15
 CRM Diagnostics Tool
– http://CrmServer/tools/diagnostics/diag.aspx
 Ideally the latency shoul...
#CRMUGSummit | #INreno15
 Which servers
– CRM Server
– Database Server
– Integration Server (or workstation)
 What to wa...
#CRMUGSummit | #INreno15
 Plan CRM database growth
– Have the right server specification to accommodate the growth
– Defi...
#CRMUGSummit | #INreno15
 If your data integration or migration process involves any querying, you should review your que...
#CRMUGSummit | #INreno15
 Reschedule CRM Maintenance Jobs
– By default, the time when CRM maintenance jobs kick off
depen...
#CRMUGSummit | #INreno15
 Manual primary keys could cause performance issue
down the road
– CRM Server generates sequenti...
#CRMUGSummit | #INreno15
 Turn on SQL Server RCSI (Read Committed Snapshot Isolation)
– https://msdn.microsoft.com/en-us/...
#CRMUGSummit | #INreno15
 Help reduce database
deadlocks
 Recommended for CRM
database, but not for data
warehouse
26
SQ...
#CRMUGSummit | #INreno15
 Set CRM database to Simple Recovery mode
– No SQL Server transaction logs
 Disable IIS Logging...
#CRMUGSummit | #INreno15
 White Papers
– Microsoft Dynamics CRM 2015 White Papers & Technical Documentation
• http://blog...
#CRMUGSummit | #INreno15
 Daniel Cai, KingswaySoft
– daniel.cai@kingswaysoft.com
– Twitter: danielcai
– LinkedIn: https:/...
#CRMUGSummit | #INreno15
THANK YOU!
 Please complete & turn in your survey now
 Session code: ADC26
 Visit the online C...
Getting data into microsoft dynamics crm faster
Upcoming SlideShare
Loading in …5
×

Getting data into microsoft dynamics crm faster

2,964 views

Published on

This is the presentation deck file that I gave at CRMUG Summit 2015 on Oct 16, 2015. In this session, I talked about some best practices that you can use to speed up your Dynamics CRM data migration and integration.

Published in: Software
  • For Microsoft Dynamics AX,GP and CRM Online Training, Register at sappractices@gmail.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Getting data into microsoft dynamics crm faster

  1. 1. #CRMUGSummit | #INreno15 #CRMUGSummit Daniel Cai, KingswaySoft Inc. SESSION CODE: ADC26 GETTING DATA INTO MICROSOFT DYNAMICS CRM FASTER
  2. 2. #CRMUGSummit | #INreno15 INTRODUCTION - YOUR PRESENTER  Daniel Cai  Principal Developer @KingswaySoft  7 Years of development experiences with Microsoft Dynamics CRM – Mostly working as a developer  5 times of Microsoft Dynamics CRM MVP since 2011  About KingswaySoft – A leading provider of data integration solutions for Microsoft Dynamics software and beyond • SSIS Integration Toolkit for Microsoft Dynamics CRM • SSIS Integration Toolkit for Microsoft Dynamics AX • SSIS Integration Toolkit for Microsoft Dynamics GP • SSIS Integration Toolkit for Microsoft Dynamics Marketing • SSIS Integration Toolkit for Microsoft Dynamics NAV • SSIS Integration Toolkit for Microsoft Dynamics SL • SSIS Integration Toolkit for Parature • and many more… – Thousands of enterprise clients worldwide depend on our integration solutions to drive their data efficiency 2
  3. 3. #CRMUGSummit | #INreno15  Discuss the techniques that you can use to get data into Microsoft Dynamics CRM faster  Discuss some best practices that you utilize in your CRM data integration and migration projects 3 SESSION OBJECTIVES
  4. 4. #CRMUGSummit | #INreno15  SSIS Integration Toolkit is used in some occasions in this presentation to discuss the techniques – It is a data integration solution developed by KingswaySoft that I work for – More information available at http://www.kingswaysoft.com/products/ssis-integration- toolkit-for-microsoft-dynamics-crm  The techniques should be transferrable to any other third-party or home- grown solutions in a similar fashion, provided an equivalent option is available  I didn’t invent any tips or tricks, neither did I discover all of them by myself, they all come from the community that I have been learning. The credits go to the generous CRM community and some special individuals. 4 DISCLAIMER
  5. 5. #CRMUGSummit | #INreno15  WHY?  WHERE TO START FROM?  HOW?  Final Q&A  Closing & Surveys 5 AGENDA
  6. 6. WHY?
  7. 7. #CRMUGSummit | #INreno15  Web Service Interfaces are generally slow – Dynamics CRM is no exception  You are often constrained by the limited time window – A couple to a few hours time window for your daily integration – Typically a ~50 hours of time window for the initial data migration • Friday 9pm to Monday 6am  Your data volume could be significantly large  You always want to be faster! 7 WHAT ARE THE PROBLEMS
  8. 8. WHERE TO START FROM?
  9. 9. #CRMUGSummit | #INreno15  Identify Resources – Technical Resources  Get to know your infrastructure – Networking – Server Capacities  Define a baseline benchmark – How many records in a second are required in order to meet the time window allowed  Define the infrastructure that can help you achieve the desired benchmark  Decide between Buy vs Write 9 DEFINE YOUR STRATEGIES
  10. 10. #CRMUGSummit | #INreno15  Optimized Infrastructure – Networking – Firewall – Security Policies – Network Load Balancer  Server Capacity (CRM Server, Database Server, Integration Server) – CPU – Memory – I/O 10 HAVING THE RIGHT INFRASTRUCTURE
  11. 11. #CRMUGSummit | #INreno15  CRM Data Import Wizard  Message-driven Integration Solutions – BizTalk – Service Bus – Message Queue  Batch-based ETL Solutions – SSIS – Scribe – Informatica – IBM CastIron 11 CHOOSE THE RIGHT TOOLS
  12. 12. #CRMUGSummit | #INreno15  Each tool works somewhat differently  Establish a good understanding of your tools – Know the capabilities – Know the different options to achieve a particular purpose – Know the techniques that can be used to help achieve better performance 12 TRY TO KNOW YOUR TOOLS
  13. 13. HOW?
  14. 14. #CRMUGSummit | #INreno15 MULTI-THREADING Make sure to update your .config file in order to overcome the connection limit of concurrent service calls <configuration> ... <system.net> <connectionManagement> <add address="*" maxconnection="100"/> </connectionManagement> </system.net> </configuration> TheSSIS way
  15. 15. #CRMUGSummit | #INreno15 LEVERAGE MULTI-NODE CLUSTER How to achieve this? • CRM Server URL option in SSIS Integration Toolkit
  16. 16. #CRMUGSummit | #INreno15 BULK API  Bulk API is more beneficial when network latency is high  There is an API throttling with CRM Online – At most 2 concurrent ExecuteMultiple requests per organization How to achieve this? • Simply enter a Batch Size in SSIS destination component.
  17. 17. #CRMUGSummit | #INreno15  The less fields, the better performance  Some fields may require additional services calls using special SOAP messages when writing to CRM – statecode/statuscode – ownerid – some more • parentsytemuserid and businessunitid (systemuser entity) • businessunitid (team and equipment entities) • Parentbusinessunitid (businessunit entity) – Those special service calls are not necessary if it is an Update and you are using v7.1 or later • For Create or Upsert calls, you still need the additional service calls (as of v7.1).  Any fields registered for CRM plugins/workflows or auditing will have certain performance impact when submitted 17 MINIMIZE THE NUMBER OF FIELDS YOU WORK WITH
  18. 18. #CRMUGSummit | #INreno15  Why? – Smaller SOAP message, which in turn results in less networking time – Can avoid firing unnecessary CRM plugins/workflows or auditing that are registered for the fields  How to achieve this? – Simply turn on “Ignore Unchanged Fields” option in CRM destination component 18 AVOID SUBMITTING UNCHANGED FIELDS
  19. 19. #CRMUGSummit | #INreno15  CRM Diagnostics Tool – http://CrmServer/tools/diagnostics/diag.aspx  Ideally the latency should be as low as possible – Make sure the integration server (or workstation) is close to the CRM Server – A poor infrastructure can result in high network latency, even the servers are close to each other  Bandwidth should be as high as possible 19 MIND YOUR NETWORK LATENCY
  20. 20. #CRMUGSummit | #INreno15  Which servers – CRM Server – Database Server – Integration Server (or workstation)  What to watch – CPU – Memory – I/O – Other CRM performance counters 20 WATCH OUT RESOURCE USAGES ON SERVERS
  21. 21. #CRMUGSummit | #INreno15  Plan CRM database growth – Have the right server specification to accommodate the growth – Define data archiving strategies as required – For initial load, consider sizing the database to the target size after the completion of the load to avoid database growth during the load  Cleanup PrincipalObjectAccessBase table as required (to remove orphan records) – http://support.microsoft.com/kb/2664150 • Consider implementing this as a SQL Server agent job to cleanup weekly or monthly to keep your system healthy  Cleanup AsyncOperationBase table as required (to remove completed workflow logs) – http://support.microsoft.com/kb/968520 for one-time cleanup • Make sure you implement the index first, otherwise it could a long time to complete depending on how many records you have • Consider implementing this as a SQL Server agent job to cleanup weekly or monthly to keep your system healthy – For CRM workflows, consider choosing “Automatically delete completed workflow jobs (to save disk space)” option for automatic cleanup – For asynchronous plugin, consider choosing “Delete AsyncOperation if StatusCode = Successful” option for automatic cleanup 21 MANAGE DATABASE GROWTH
  22. 22. #CRMUGSummit | #INreno15  If your data integration or migration process involves any querying, you should review your query performance  Depending on the situation, if any query involves non-indexed CRM database field(s), you might want to add custom index(es) – CRM On-Prem, add custom indexes to CRM database directly – CRM Online, request adding custom indexes by raising a support ticket. Alternatively, use the following techniques to add custom indexes automatically. • Add the field to CRM quick find view • Create CRM alternate key – will have to be unique  Note that having an excessive number of indexes in one database table (entity) can adversely affect your performance  Where to start from? – Look for long running queries in CRM Server Windows Log • CRM logs any database queries that take longer than 10s – SQL Server “Missing Index” tables  Query Tuning – Show Actual Execution Plan – SQL Server Management Studio (SSMS) – SQL Server Database Engine Tuning Wizard 22 MANAGE DATABASE INDEXES
  23. 23. #CRMUGSummit | #INreno15  Reschedule CRM Maintenance Jobs – By default, the time when CRM maintenance jobs kick off depends on when the server was installed (or the organization being provisioned) – which could be in busy business hours – Ideally your integration job kicks off shortly after the maintenance job finishes  Disable the Reindex All job – What it does? • Reindex • Shrink your CRM database – Create your own CRM database maintenance jobs 23 MANAGE CRM MAINTENANCE JOBS
  24. 24. #CRMUGSummit | #INreno15  Manual primary keys could cause performance issue down the road – CRM Server generates sequential GUID values for best performance – Use manual primary keys only for migration purpose  Further read: – The Dangers of Guid.NewGuid(); • http://blogs.msdn.com/b/crminthefield/archive/2015/01/19/the-dangers-of-guid- newguid.aspx 24 AVOID GENERATING PRIMARY KEY (GUID) YOURSELF
  25. 25. #CRMUGSummit | #INreno15  Turn on SQL Server RCSI (Read Committed Snapshot Isolation) – https://msdn.microsoft.com/en-us/library/tcbchxcb(v=vs.110).aspx ALTER DATABASE CrmOrgName_MSCRM SET ALLOW_SNAPSHOT_ISOLATION ON ALTER DATABASE CrmOrgName_MSCRM SET READ_COMMITTED_SNAPSHOT ON  Add NOLOCK hint to your FetchXML Query <fetch mapping="logical" no-lock="true"> <entity name="account"> <attribute name="name" /> <filter> <condition attribute="creditlimit" operator="gt" value="1000000" /> </filter> </entity> </fetch> 25 DEAL WITH LOCKS
  26. 26. #CRMUGSummit | #INreno15  Help reduce database deadlocks  Recommended for CRM database, but not for data warehouse 26 SQL SERVER – MAX DEGREE OF PARALLELISM
  27. 27. #CRMUGSummit | #INreno15  Set CRM database to Simple Recovery mode – No SQL Server transaction logs  Disable IIS Logging  Disable all CRM workflows / plugins and auditing  Running data integration on CRM server itself – Best possible network latency – You need to alternate the target service URL to point to local server as shown previously 27 EVEN MORE AGGRESSIVE MEASURES – SOME MAY ONLY BE PRACTICAL FOR INITIAL LOAD
  28. 28. #CRMUGSummit | #INreno15  White Papers – Microsoft Dynamics CRM 2015 White Papers & Technical Documentation • http://blogs.msdn.com/b/crminthefield/archive/2015/01/05/microsoft-dynamics-crm-2015-white-papers-amp-technical- documentation.aspx – Optimizing and maintaining the performance of a Microsoft Dynamics CRM 2011 server infrastructure • http://www.microsoft.com/en-ca/download/details.aspx?id=27139  Wikipedia – Dynamics CRM Quick Optimization Guide • http://social.technet.microsoft.com/wiki/contents/articles/13661.dynamics-crm-2011-quick-optimization-guide.aspx  Blog Posts – http://blogs.msdn.com/b/crminthefield/archive/2012/04/26/avoid-performance-issues-by-re-scheduling-crm-2011- maintenance-jobs.aspx  Best Practices – Best practices for developing with Microsoft Dynamics CRM • https://msdn.microsoft.com/en-us/library/gg509027.aspx 28 RESOURCES
  29. 29. #CRMUGSummit | #INreno15  Daniel Cai, KingswaySoft – daniel.cai@kingswaysoft.com – Twitter: danielcai – LinkedIn: https://ca.linkedin.com/in/danielcai – http://www.kingswaysoft.com 29 CONTACT INFO
  30. 30. #CRMUGSummit | #INreno15 THANK YOU!  Please complete & turn in your survey now  Session code: ADC26  Visit the online CRMUG Summit 2015 community to download these slides, ask questions, and connect with participants  Visit the CRMUG Medics in the HUB to get expert advice on your CRM-related problems! 30

×