Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Extending D365 with Azure
Similar to Extending D365 with Azure (20)
Recently uploaded
Recently uploaded (20)
Extending D365 with Azure
- 1. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 112/20/2017 Nelson Johnson Extending CRM with Azure www.broadpoint.net 1 Extending the Dynamics 365 World With Microsoft Azure By Nelson Johnson Solution Architect BroadPoint Technologies
- 2. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 2 Target Audience • Interest in Cloud Architecture • IT Managers • Developers • System Architect • Azure curious • On-prem CRM and considering moving to Dynamics 365 (D365) • Direct integration with SQL • Dealing with D365 Limitations
- 3. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 3 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices
- 4. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 4 Who is Nelson Johnson? • Started programming 1977…in high school…on a mainframe • Became full time software consultant in 1982 • BS in MIS and Decision Science from George Mason University • Worked at DARPA creating web apps in 1996 • Working with CRM since version 3 • Solution Architect for BroadPoint Technologies, LLC • Email: njohnson@broadpoint.net
- 5. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 5 Why would you want to extend D365 to Azure? • Inconsistent workload demands • Consolidate business logic • Scheduled jobs are not easily managed within D365 • SQL stored procedures are not an option • Limitations of FetchXML, Rollup fields • Offload processing • Plugins, workflows, and API in D365 have a timeout limit of 2 minutes • Chain of plugins or workflows are getting too long • Need to create or update a large number of records • Waiting for external services to respond • SQL timeout
- 6. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 6 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices
- 7. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 7 Tools for extending to Azure • Integration • Endpoint Config in D365 • Service Bus / REST API • Scheduled Tasks • Storage • SQL Server Replication • Azure Cosmos DB • Azure Tables • Compute • Azure Functions • Azure Web Jobs • Azure Service Fabric • Azure Worker Role • Logic Apps • Worker Roles (see Best Practices) • Service Plan • Controls CPU, Memory
- 8. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 8 What is Service Bus? • A first-in-first-out message queue • Separates events from actions • Enforces decoupled architecture • Increases options for re-usuable business logic • Lock & complete • High availability • Scalable • Nearly free • Handles failures Source: https://docs.microsoft.com/en-us/azure/service-bus- messaging/service-bus-java-how-to-use-topics-subscriptions
- 9. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 9 How does D365 integrate with Azure? Register Step Create EventServiceBus Entity Endpoint Endpoint ServiceBus ServiceBus Manager Batch Workflow Register Step Create EventServiceBus Entity Endpoint Dues Billing Workflow LOB Service Configure with Plugin Registration Create using Azure UIConfig with CRM Solution Your code Service Fabric WebJobs Functions
- 10. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 10 Service Bus vs Webhooks (D365 CE 9) • Service bus is always up – Web hooks only work when the service is working • SB has low latency • SB can ingest 1000’s of messages per second • 1000 concurrent connections • Publish/subscribe supports a decoupled architecture
- 11. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 11 How do you connect an entity to the Service bus? • Plugin Registration Tool – no code needed! • Endpoint • Step DeveloperSidebar
- 12. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 12 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices Service Fabric, WebJobs, Functions, Logic Apps
- 13. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 13 How low will you go? Source: https://blogs.technet.microsoft.com/kevinremde/2011/04/03/saas-paas-and-iaas-oh-my-cloudy-april-part-3/
- 14. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 14 Azure Compute • Light – Azure Functions, Logic Apps • Mid-size – Azure Webjobs • Heavy – Service Fabric
- 15. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 15 Azure Functions Pros Cons Deployment Consumption plan – ideal for intermittent services Dynamic Scaling More triggers than WebJobs Integration with more services than WebJobs Complements Azure Logic Apps Control – what is that? Up to 10 minute Timeout unless you pay for a service plan Local Dev and Debug Version Control Deploy directly from VS Continuous Integration/deployment
- 16. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 16 Logic Apps Pros Cons Deployment Can schedule jobs Can query and update D365 data Can interface with Service Bus Can connect to over 165 other services Configuration through JSON – no code Price based on # of executions “Server-less” Weekly Updates No code – anything more than simple tasks requires a Function UI to develop is inside Azure – not easy to figure out Documentation is sparse Programing skill level Limited triggers: Min trigger schedule 15 seconds 60 seconds Version control built in to web UI Visual Studio – Publish ARM template deployment
- 17. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 17 Azure App Service WebJobs Pros Cons Deployment PaaS – Managed VM, High availability Multiple apps on single service plan Easier (than SF) for developer to start developing apps Can run continuously or triggered Scales with App Service plan – setup manually You pay for it while it is deployed Timeout limit No remote desktop Local Dev and Debug Version Control Deploy directly from VS Continuous Integration/deployment
- 18. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 18 Azure Service Fabric Pros Cons Deployment Best for Microservices Control over how workloads are run Load-balancing, scaling, failover, replication, stateful VM Scale Sets in clusters Self-hostable Fault Analysis Service Learning curve Code changes will be necessary to forklift code. Re-architect is better approach. Expensive initially because a production system uses 5 VM’s - Value increases as you add more services No remote debugging Local Dev and Debug Deploy directly from VS PowerShell and CLI deployment Deploy many services at one time
- 19. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 19 Wrap up - Which one is best? • Azure Functions • Just getting started and want to experiment • Specialized integration • Extending Azure Logic Apps • Azure Logic Apps • Glue for mashups • Azure WebJobs • Already have a website and want to piggy-back on that service plan • Credit card processing • Azure Service Fabric • Re-architecting a legacy environment • Require high level of control and resilience
- 20. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 20 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Azure Architecture Share our Best Practices Microservice
- 21. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 21 Design Considerations • Must read: Azure Architecture Center(https://docs.microsoft.com/en-us/azure/architecture/) • Application Architecture Guide • Reference Architectures • Cloud Design Patterns (32 of them) • Best Practices • Performance Antipatterns • Design Checklists
- 22. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 22 Microservice Architecture Design Patterns Increase the velocity of application releases by decomposing the application into small autonomous services that can be deployed independently Source: https://azure.microsoft.com/en-us/blog/design-patterns-for-microservices/ D365
- 23. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 23 BASE - Design Patterns for Eventual Consistency • Use Event Sourcing and CQRS Pattern • Data Consistency • Decouple recording the transaction from updating • Reconciliation between services is easier • Retry Pattern • Best for transient faults like timeouts • Service Bus provides this (to a degree) • Idempotent methods • Same result if called multiple times
- 24. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 24 Potential for Inconsistent Data • Two transactions are created to update the same account • They both get the starting balance • They both are adding a different amount to the balance • Last one “wins” Competing Updates Process 1 CRM Process 2 Time Fetch Current Balance Calculate new Balance Update Balance Fetch Current Balance Calculate new Balance Update Balance Balance=100 Balance=110 $10 transaction $20 transaction Balance=120
- 25. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 25 Event Sourcing and CQRS to the rescue! • Two transactions are created to update the same account • They both append their update to a journal • They both send a message to another process to update the balance • The balance is calculated twice, but it is accurate Decoupled Updates Process 1 CRM Process 2 Process 3 Time Write transaction to journal Update Balance Event Update Balance Write transaction to journal Update Balance Event Update Balance Balance=100 Balance=130 $10 transaction $20 transaction Balance=130 Journal=100 Journal=10 Journal=20
- 26. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 26 Plan for Failure! • Compensating transaction Pattern • Un-do in reverse order • Create Exception entity to manage resolution • Provide a view of exceptions in the dashboard • Provide error with steps to resolve it Source: https://docs.microsoft.com/en- us/azure/architecture/patterns/compensating-transaction
- 27. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 27 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Azure Architecture Share our Best Practices
- 28. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 28 Cloud Service Lessons Learned • “Classic” service – not part of ARM deployments • It was an easy starting point • No timeouts or code size limits • You should employ Async Tasks • Doesn’t run under CSP license • Runs all the time like a VM • No options to use KeyVault • No options for dynamic configuration of Web.config/App.config
- 29. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 29 Best Practices • Test Location and latency • Check which datacenter your CRM instances is located https://<instance>.crm.dynamics.com/home/home_debug.aspx Server Name First Two Letters Region BL East US BN North Central US BY West US
- 30. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 30 Best Practices • Time Zone • Integration account in customers timezone • Work with localized datetime fields using UTC • Service Plan • Stress test with the smallest one and work up • Azure Resource Manager • Group by Resource (compute, service plan, storage, service bus) • Tags: Owner, Environment, LoB Function(s), Tier, Project • Test Driven Development • Minimize “chatter” using API Transactions, ExecuteMultiple • Monitor your usage
- 31. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 31 Best Practices • Do not include the Service Endpoints in the D365 Solution file that gets promoted to production • Have your subscribing service validate that the message is coming from the right domain • SB message has the sending domain – compare it to the D365 connection • Use one service bus “namespace” per D365 instance • Configure to use many queues or topics • Use a non-interactive account in D365 • Reduce license cost
- 32. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 32 Use EntityCollection() for Transactions • EntityCollection() • Relationship() • EntityCollection Entities.Add() • <entity>.RelatedEntities.Add() (early bound) • Create(<entity>) DeveloperSidebar
- 33. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 33 Transaction code sample Code Description Guid duesHeaderId = Guid.NewGuid(); Create a new ID so that we can create a reference in each detail record EntityCollection duesHeaderDetailCollection = new EntityCollection(); Create a collection that will hold all the child records Relationship duesHeaderDetailRelationship = new Relationship("bpt_bpt_duesheader_bpt_duesdetail"); Define the relationship that links the entities together duesHeader.bpt_DuesHeaderId = duesHeaderId; Assign the PK of the header record duesHeader.bpt_MemberId = Contact.ToEntityReference(); Assign some values to the header - this is the member Xrm.bpt_duesdetail duesDetail = new Xrm.bpt_duesdetail(); Create an instance of the child entity record duesDetail.bpt_DuesBillId = new CrmEntityReference( Xrm.bpt_DuesHeader.EntityLogicalName, duesHeaderId); Populate the entityreference from the child back to the parent record duesHeaderDetailCollection.Entities.Add(duesDetail); Add the child record to the parent duesHeader.RelatedEntities.Add(duesHeaderDetailRelationship, duesHeaderDetailCollection); Complete the relationship by adding the child collection to the parent entity service.Create(duesHeader); Call the service to create both records at the same time DeveloperSidebar
- 34. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 34 Resources: • Platform feature comparison: https://docs.microsoft.com/en-us/azure/app-service/choose- web-site-cloud-service-vm • Azure Architecture: https://docs.microsoft.com/en-us/azure/architecture/ • Azure Functions Tools for Visual Studio: https://docs.microsoft.com/en-us/azure/azure- functions/functions-develop-vs • Write and test functions locally: https://blogs.msdn.microsoft.com/webdev/2016/12/01/visual-studio-tools-for-azure-functions/ • Azure Functions forum: https://social.msdn.microsoft.com/Forums/azure/en- US/home?forum=AzureFunctions&filter=alltypes&sort=firstpostdesc • Continuous Deployment: https://docs.microsoft.com/en-us/azure/azure-functions/functions- continuous-deployment • Subscription Governance: https://docs.microsoft.com/en-us/azure/azure-resource- manager/resource-manager-subscription-governance
- 35. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 35 Resources (cont): • Design Patterns for Microservices: https://azure.microsoft.com/en-us/blog/design- patterns-for-microservices/ • WebJobs Extensible Triggers: https://azure.microsoft.com/en-us/blog/extensible-triggers- and-binders-with-azure-webjobs-sdk-1-1-0-alpha1/
- 36. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 36www.broadpoint.net Q&A
- 37. 12/20/2017 Nelson Johnson Extending D365 with Azure www.broadpoint.net 37www.broadpoint.net Thank You! Nelson Johnson njohnson@broadpoint.net 301-634-2442
Editor's Notes
- Note: This presentation was developed in October 2017. The services in the cloud are constantly changing.
- There are a lot of good reasons for using Azure, such as reliability. Cost should not be the only consideration. Consider the learning curve to maintain apps in house vs in the cloud, and the cost of keeping updates in lock-step with the platform. Inconsistent workload demands – Web Portal with usage spikes or scheduled business process that does not need to run all the time Good for offload batch processing – Asynchronous process to create 20k records, such as monthly billing Use it to consolidate business logic – such as an integration with Microsoft Graph services, machine learning, or handling complex calculations like linear equations for product mix.
- When you build extensions to D365, you have many options available to run your code in Azure. These are listed in the Compute column. We will discuss the first 5 of these because they provide a diverse set of features that work for many different applications. But the code does not run itself. Somehow the program you have hosted in Azure must determine when to run and which records in CRM they apply to. This is where we introduce the integration options because they control which programs will run in response to user activity in CRM. The most common integration works through defining an endpoint in D365 that connects to a Service Bus. An optional component to consider when extending your D365 to Azure involves using one of several storage options. Lets start with the Integration part first.
- The Service Bus is the most common way to communicate events in D365 out to your Azure programs. Service Bus is a first-in-first-out message queue. It is like a database pipe – messages go in one side and come out the other. When you create a Service Bus Namespace in Azure, it gives you a connection string to connect to it. Your custom code would be the message receiver, and that code must retrieve each message (in effect lock it) run a process based on what is in the message, then mark it as complete. Why would we use it? D365 is super simple to send messages to it, it can queue up 1M messages, and it handles failures for us without writing any logic.
- In general, D365 is easily configured to send messages through a Service Bus to trigger your programs in Azure to start running. When an event happens in D365 that we want to trigger some Azure code to process, we have a workflow that creates a record in our custom Service Bus Entity It is configured to push messages into separate queues or topics depending on what the event was. The event message sits in the Service Bus queue until the Service Bus Manager picks it up and calls your Line of Business (LoB) service. LoB Service connects to CRM and performs work Errors are logged for an operator to fix and restart the process
- Do you need to use the Service Bus? No. You could use Webhooks if you are using D365 CD 9 or later. Here are some reasons you should use the Service Bus: Consider the scenario when you need to update a back-end service. By using a service bus, you can stop and start the back end service without disrupting the other business processes.
- Create a custom entity Register the endpoint Register the step (CRM event message)
- How much control do you want? You might need to give up some control to get scalability, quick deployment, security updates Can you reassign your team to higher value tasks?
- Which service is right for you? There is a tradeoff between what is quick and easy to code and the need for robust enterprise features. I would categorize the services this way:
- “Server-less” just means you don’t own the server. Languages: JavaScript, C#, and F#, and scripting options such as Python, PHP, Bash, Batch, and PowerShell. Triggers on Microsoft Graph Events (Excel Table events, One Drive file events, Outlook), Azure Cosmos DB, Blob storage, Events, HTTP REST or Webhook, Service Bus, and Timer. Integration with: Logic Apps, Storage tables, SQL tables, Twilio, SendGrid, Push Notification Hubs, and Auth Tokens. Complements Azure Logic Apps Run this with service bus topics and subscriptions
- Another “Serverless” option Technology that Flow is built on Example: Poll the database every minute for 10 days + execute 500 workflows = $15/10 days Glue between applications Triggers: recurring schedule, on demand, HTTP, API News features released in frequent updates
- Easier for a developer to get up to speed PCI, SOC, ISO compliant Triggers: Blobs, Service Bus, Queue, Timer, continuous, custom No remote desktop
- Big learning curve. Better choice if you plan to migrate legacy applications away from an enterprise data center Stateful microservice removes the need for queues and caches. Stateful services are high availability and low latency. Auto-scaling built in to clusters. With the Fault Analysis Service, you can induce meaningful faults and run complete test scenarios against your applications
- Reference architectures – Network DMZ, Hybrid, Identity Management, Managed web apps Performance Antipatterns – Busy Database, Chatty I/O, Extraneous Fetching (too much data),
- Shows some example cloud architecture design patterns. There are over 30 to choose from. The Sidecar pattern is for features like logging, and security that may be part of your framework to provide orthogonal services. Anti-corruption layer deals with exposing views of data from the legacy system in a manner consistent with the new system and allows the legacy system to function without modification. The Strangler is used to direct data requests to specific pieces of functionality that is being incrementally moved form the legacy system to a microservice architecture. Ambassador patterns act as proxy to services to allow control over routing, resiliency, security features, and to avoid any host-related access restrictions.
- Your on-premesis systems typically are using a SQL technology that guarantees data consistency, (also called ACID: Atomicity, Consistency, Isolation, Durability) Dynamics 365 lives in the BASE world. Basically Available, Soft state, Eventual consistency means that transactions are distributed and they are eventually updated in the database(s). This means that D365 may not be 100% accurate all the time and you must design processes that will catch inconsistencies and trigger a process to fix it. Event Sourcing is used to keep object state history so that it can be re-evaluated and corrected Command Query Responsibility Separation is used to de-couple the recording of the transaction from an actual update Use Materialized Views design pattern to present the result of the transactions Retry Pattern depends on what the fault is – cant help you if your code is broken.
- Example of what could go wrong if you create a system with ACID in mind, but you are really using a BASE system: You start with a known account balance But you end with an incorrect balance What happened? What was the balance before the transactions? If this is milliseconds apart, it would be very hard to tell what is going on. You could use logging, but then someone has to monitor it, and how do you fix it?
- By keeping the entire history of changes, we no longer really need the Balance (State) of the account We can completely recalculate the balance, or re-play a series of events (think debugging). We can also see the balance at any point in time. Eventually all accesses to an item will return the last updated value. By pushing updates into a transaction log, process does not have to wait for CRUD operations The cost: storing more data, overall system is more complex because it is many smaller parts, overhead to complete the task is more
- For each transaction, keep track of steps taken If the process is not completed, have a process that will walk back through and undo what was done You are effectively creating what you might have done If you have used transactions in SQL Server
- None of these reasons should stop you from using it, unless you are selling D365 licenses under a CSP model Microsoft has not been putting any effort into this area – seems like it is going away Our customers only get about 5% of CPU during peak periods, but pay all month for it
- Location, location, location – test the latency because it can really cost you
- My profile was for East coast time, but customer was in Houston and they saw my changes as having the wrong time. Keeping track of your services using ARM is really important to containing costs - Make a governance policy on who owns what using tags! TDD costs a bit more up front, but really pays for itself after sprint 1
- If you include the Service Endpoints in the solution file that you promote from dev to production, you will be connecting the production instance back to the development code The message on the service bus has the domain of the system that created the message. Compare it to the connection string domain to make sure they are the same. You can differentiate which messages belong to which programs by using queues and topics – don’t use multiple service buses In D365, check the “non-interactive” box on the user account you use for integration.
- These are the classes you need to combine to into creating a transaction The benefit is to reduce the amount of chatter with the API, and it is an all-or-nothing, which saves you from having to clean up partial transactions
- Some key elements are color coded to make it easier to follow where they are initialized and used. This sample is a membership dues Header / Detail A key point – the duesHeaderId is manually assigned in order to build the link between the parent and child record Typically, you would have more than one detail (child) record.