When to use What? A look at choosing Integration Technology

  • 977 views
Uploaded on

There’s never been more options -- or more confusion -- about which technologies to use when connecting systems? When is BizTalk Server the right choice? Should I still use WCF? Is the Windows Azure …

There’s never been more options -- or more confusion -- about which technologies to use when connecting systems? When is BizTalk Server the right choice? Should I still use WCF? Is the Windows Azure Service Bus ready for production scenarios? In this session, we’ll first do an overview of each core technology in the integration portfolio. Then we’ll review a simple decision framework for down-selecting your choices. Finally, we’ll have an interactive discussion (and demonstration) of real-life scenarios and which technology is the right fit.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
977
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
26
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Who am i
  • Where are we / problem
  • Previous book (cover)
    Technology count: WCF, WF, AppFabric, BizTalk, ESB, SSIS, SSSB, Sync Framework, Azure SB Relay, StreamInsight
    Technology count TODAY: WCF, WF, AppFabric, BizTalk, ESB, SSIS, SSSB, Sync Framework, Azure SB relay, Azure SB Brokered, Azure VMs, Azure Notification Hubs, BizTalk Services, StreamInsight
    LET’S TALK THROUGH SOME TRENDS THAT EMERGED SINCE THIS BOOK
  • Source: http://www.flickr.com/photos/davenielsen/5718247676/sizes/l/

    Cloud computing
    Change to the traditional computing paradigm
    Host matters less; elastic, on-demand
    Accepted part of an organization’s app portfolio
    New types of applications and scenarios
    Widely distributed
    Mix of hosting providers
    Different integration scenarios
    Cloud to cloud, ground to cloud, cloud to ground
  • Source: http://www.flickr.com/photos/mattandbeccasteudelwedding/9067696311/sizes/l/

    Micro Services
    Something championed by ThoughtWorks a while back, and now something Netflix (and others) do a lot of
    Mini applications/services
    No heavy, complex services
    Very specific services that are easy to consume and weave together
    Think of cloud providers who have distinct services for provisioning services, autoscale, etc
  • Source: http://www.flickr.com/photos/24752578@N02/5589897831/sizes/l/

    Lightweight Integration
    It’s not just about a powerful, centralized integration server
    That’s of course always been the case, but new attention paid to simplified messaging
    Something between a basic queue and a full blown ESB
    New tools, new way to connect lightweight services together
  • Source: http://www.flickr.com/photos/anekoho/9638930676/sizes/h/

    NoSQL endpoints
    New types of endpoints, often outside your direct control
    Relational databases are less important now than ever
    Still a place for them, but major companies are finding they don’t need them
    We’ve replaced our SQL backend with one
    To the integrator, this adds new scenarios
  • Source: http://www.flickr.com/photos/peternijenhuis/6959653638/sizes/h/

    Automation
    DevOps and the cultural movement to unprecedented synergy between Dev and Ops teams
    Automated builds, continuous integration, continuous deployment
    Possible to get rid of snowflake servers, and move to immutable servers
  • Source: http://www.flickr.com/photos/suckamc/5907306188/sizes/l/

    We want to be at a place where we have the simplest toolset possible, but one that meets our needs
    This continued expansion of available integration services is logical, when considering these trends
    There are more and more tools for specific scenarios (micro-integration services), and fewer monolithic tools that serve EVERY purpose
  • How do we get there?
    Walk through a decision framework
    Talk through the available technologies
    Apply these principles
  • What’s a decision framework?
    Way to analyze a situation and identify the right fit
    There cannot be a massive decision matrix that drops the correct solution in your lap
    Instead, there are multiple dimensions to consider, and those may be unique to each organization and situation
    Steps
    First, let’s discuss the sources that describe our situation
    Then, we’ll talk about the 4 dimensions to discuss
  • Input into a decision …
    Functional requirements
    What is the application supposed to do?
    Business requirements
  • Input into a decision …
    Non functional requirements
    Describe what the system itself needs to do to support the functional requirements
    security, availability, maintainability, compliance, exception handling, etc
    You can argue if these are actually functional requirements, and in some cases they are
  • Input into a decision …
    Derived requirements
    Ones that aren’t explicit stated, but can be derived from what’s needed
    Based on a functional requirement to provide a certain type of data, I might derive a requirement to persist data in a warehouse for long range analytics
  • Input into a decision …
    Organization direction/strategy
    This is the least transferrable!
    What are the company goals, investments?
    These are often fixed, and non-negotiable


    NOW, LET’S TALK ABOUT THE DECISION CRITERIA ITSELF
  • Solution design – design patterns required
    Data volumes, data set size, guaranteed delivery, data sources/destinations, latency, sync/async, data processing (workflow, quality operations), SSO, failure handling
    MODERN UPDATES … (reliability of endpoint, location of endpoint, constraints imposed by source/destination, circuit breakers)
  • Solution development – what it takes to build the solution
    Developer availability, learning curve, developer setup, IDE and tool integration, support system, source control integration, automated builds
    MODERN UPDATES … (works in cloud, web-based IDEs, continuous integration/delivery support)

  • Solution operations – what it takes to maintain the solution
    Event logging, tuning ability, handle bursts and failures, backup/restore routines, support tools, versioning capabilities, scriptability
    MODERN UPDATES … (immutable runtimes, rapid releases/rollbacks, works in distributed fashion)

  • Organizational considerations – enterprise standards and org direction
    Long term solution, vendor support, uses existing software investment, limited budget impact, build vs buy compatibility, risk tolerance, speed to market
    MODERN UPDATES … (OSS, commodity skill sets, easy/no upgrade)

  • Product: BizTalk Server
    Tag Line: Comprehensive application integration
    Most Recent Release: 4/13
    Capabilities: Durable messaging, orchestration, technology adapters, trading partner management, BAM, BRE, etc
    Core Use Cases: Async messaging-based integration between applications; stateful, long running workflow processes that coordinate interactions between systems
    Buy-Hold-Sell: Buy
  • Product: ESB Toolkit
    Tag Line: “Dynamic messaging”
    Most Recent Release: 4/13 with BizTalk 2013
    Capabilities: Dynamic routing, services for mapping and pipelines
    Core Use Cases: Diverse message paths; message flow modeling, supremely loose coupling, centralized exception management
    Buy-Hold-Sell: Soft Buy
  • Product: Windows Server AppFabric
    Tag Line: Advanced service hosting
    Most Recent Release: 3/12
    Capabilities: Caching, advanced WCF and WF hosting; troubleshooting; persistent workflows
    Core Use Cases: WCF service management, hosting long running WF services
    Buy-Hold-Sell: Hold
  • Product: Service Bus for Windows
    Tag Line: On-premises durable messaging
    Most Recent Release: 10/13
    Capabilities: Running messaging aspects of Azure Service Bus on-premises; topics and queues; manage with new Windows Azure Pack on Windows
    Core Use Cases: Start work on premises and move to cloud; basic pub-sub messaging;
    Buy-Hold-Sell: Hold
  • Product: Workflow Manager
    Tag Line: On-premises workflow host
    Most Recent Release: 3/13 (CU)
    Capabilities: multi-tenancy, tracking and monitoring, instance management, versioning support
    Core Use Cases: Host and manage WF apps;
    Buy-Hold-Sell: Hold

    http://msdn.microsoft.com/library/jj193528
  • Product: WCF
    Tag Line: Sophisticated service framework
    Most Recent Release:
    Capabilities: Contract first service design for REST and SOAP services; configuration-centric service model that is extreme (overly?) extensible
    Core Use Cases: Building SOAP based web services; Sync or async
    Buy-Hold-Sell: Soft Buy
  • Product: ASP.NET Web API
    Tag Line: Modern web services
    Most Recent Release: 1/14
    Capabilities: REST/HTTP service design; extensible model; Odata support
    Core Use Cases: web service design
    Buy-Hold-Sell: Buy
  • Product: Windows Workflow
    Tag Line: Lightweight, durable processes
    Most Recent Release:
    Capabilities: declarative models that describe a process; built-in and custom activities; durable workflow; service endpoints
    Core Use Cases: long running workflows; exposing business processes as services
    Buy-Hold-Sell: Soft Buy
  • Product: SSIS (also Master Data Services?)
    Tag Line: Bulk data transfer
    Most Recent Release: SQL 2014 (no big changes)
    Capabilities: Data integration and data transformation; lots of data sources (files, FTP, HTTP, SMTP, RDBMS); bulk movement
    Core Use Cases:
    Buy-Hold-Sell: Buy
  • Product: StreamInsight
    Tag Line: Event stream processing
    Most Recent Release: 6/12
    Capabilities: high volume pattern matching against event streams; adapter model; Iobservable, Ienumerable support; multiple hosting options; LINQ queries
    Core Use Cases: situational awareness from data streams
    Buy-Hold-Sell: Soft Hold
  • Product: Windows Azure Virtual Machines – BizTalk Image
    Tag Line: Full BizTalk in the Cloud
    Most Recent Release: June 2013
    Capabilities: Run full BizTalk Server in the cloud; no HA with SQL clustering
    Core Use Cases: Dev/test scenarios
    Buy-Hold-Sell: Hold
  • Product: Windows Azure Service Bus Relays
    Tag Line: Services that span boundaries
    Most Recent Release:
    Capabilities: Traverse networks and securely expose sync and async WCF services
    Core Use Cases: partner integration, mobile device integration, remote worker tools, cloud to cloud integration
    Buy-Hold-Sell: Strong Buy
  • Product: Service Bus Brokered Messaging
    Tag Line: Durable messaging in the cloud
    Most Recent Release:
    Capabilities: Durable messaging through queues (single reader), and topics (broadcast with subscriptions)
    Core Use Cases: partner integration, mobile integration, cloud to cloud integration
    Buy-Hold-Sell: Buy
  • Product: Service Bus Notification Hubs
    Tag Line: Scalable message distribution
    Most Recent Release:
    Capabilities: Broadcast messages to mobile devices and Windows 8;
    Core Use Cases:
    Buy-Hold-Sell: Buy
  • Product: Windows Azure BizTalk Services
    Tag Line: Lightweight cloud message broker
    Most Recent Release:
    Capabilities: Link applications via cloud-hosted broker; source/destination adapters; strong data mapper
    Core Use Cases: connecting partners, cloud apps
    Buy-Hold-Sell: Strong Hold
  • MANY other products that work well in a Windows and cloud world
    Brokers like Apache Kafka
    Queue technologies like RabbitMQ, or Amazon SQS
    Cloud-based integration tools like SnapLogic and Informatica Cloud
    SaaS-integrated services for message publication and workflow
    E.g. Dynamics CRM Online or Salesforce.com
  • Options …
    Use messaging and use BizTalk to send messages to vendor endpoints
    PRO: Good tracking, chance for enrichment before sending out; guaranteed delivery
    CON: hard to quickly change endpoints; puts responsibility on internal team to manage/scale;
    Use messaging and use BizTalk to send to Topics, local DB, and notification Hubs
    Use messaging and send loans directly to SB topics (and internal DB) from WF
    PRO
    CON
    Send loans to SB topic and one listener is internal DB
    PRO: Simple, offload responsibility for delivery
    CON: No coordination; chance of race condition; no notification for vendors of new messages
    Send loans to SB topic, one listener is internal DB, and also use Notification Hubs for alerting
    Provide REST-ful query API (exposed via SB) on loan DB and use notification hubs to send messages to providers to query and provide their best offer
    PRO: Simple; no replication; easy to onboard new vendors; offload availability to 3rd party; durable;
    CON: Not event based; need to chain together way to notify vendors; no orchestration or tracking; limited visibility into vendor processing
  • Let’s consider the option that leverages the Service Bus for all this
    Send a message to Notification Hubs so very easy to broadcast info and push alerts
    Use Topics for scalable, durable messaging
    Subscribe ourselves to update internal systems

Transcript

  • 1. brought to you by When to Use What? A Look at Choosing Integration Technology RICHARD SEROTER @rseroter
  • 2. Richard Seroter Director of Product Mgmt at CenturyLink Cloud Microsoft MVP for Integration Technical Trainer at Pluralsight Cloud Editor for InfoQ.com 3-time Book Author
  • 3. WHERE ARE WE?
  • 4. Integration Technologies Covered: 10 Integration Technologies TODAY: 14+
  • 5. TRENDS
  • 6. CLOUD COMPUTING
  • 7. MICRO SERVICES
  • 8. LIGHTWEIGHT INTEGRATION
  • 9. NOSQL ENDPOINTS
  • 10. AUTOMATION
  • 11. DESIRED END STATE?
  • 12. MAINTAINABLE SIMPLICITY
  • 13. GETTING THERE
  • 14. DECISION FRAMEWORK
  • 15. FUNCTIONAL REQUIREMENTS NON FUNCTIONAL REQUIREMENTS DERIVED REQUIREMENTS ORGANIZATIONAL STRATEGY
  • 16. FUNCTIONAL REQUIREMENTS NON FUNCTIONAL REQUIREMENTS DERIVED REQUIREMENTS ORGANIZATIONAL STRATEGY
  • 17. FUNCTIONAL REQUIREMENTS NON FUNCTIONAL REQUIREMENTS DERIVED REQUIREMENTS ORGANIZATIONAL STRATEGY
  • 18. FUNCTIONAL REQUIREMENTS NON FUNCTIONAL REQUIREMENTS DERIVED REQUIREMENTS ORGANIZATIONAL STRATEGY
  • 19. Data volumes Guaranteed delivery Data sources / destinations Latency Authentication / authorization Failure handling Endpoint location, reliability, constraints Circuit breakers Design DevOperations Org Strategy
  • 20. Design Dev Operations Org Strategy Developer availability Learning curve Developer setup IDE and tool integration Support system Automated build support Deploy to cloud Web-based dev tools Continuous integration / delivery
  • 21. Design Dev Operations Org Strategy Event logging Performance tuning knobs Burst and failure handling Backup / restore tools Scriptability Immutable servers Rapid release / rollback Distributed deployments
  • 22. Design DevOperations Org Strategy Long term solution fit Vendor support available Uses existing (software) investments Budget impact Build vs. buy philosophy Risk tolerance Speed to market OSS components Commodity skill sets Simple upgrades
  • 23. PRODUCT EVALUATION
  • 24. BIZTALK SERVER “Comprehensive application integration” BUYHOLDSELL
  • 25. ESBTOOLKIT “Dynamic BizTalk messaging” BUYHOLDSELL
  • 26. WINDOWS SERVER APPFABRIC “Advanced service hosting” BUYHOLDSELL
  • 27. SERVICE BUS FOR WINDOWS “On-premises durable messaging” BUYHOLDSELL
  • 28. “On-premises, multi- tenant Workflow host” BUYHOLDSELL WORKFLOW MANAGER
  • 29. WCF “Sophisticated service framework” BUYHOLDSELL
  • 30. ASP.NET WEB API “Modern web services” BUYHOLDSELL
  • 31. WINDOWS WORKFLOW “Lightweight long- running processes” BUYHOLDSELL
  • 32. SSIS “Bulk transfer between data sources” BUYHOLDSELL
  • 33. STREAM INSIGHT “Complex event stream processing” BUYHOLDSELL
  • 34. WINDOWS AZURE VM “Full BizTalk Server in the cloud” BUYHOLDSELL
  • 35. SERVICE BUS RELAY “Services that span network boundaries” BUYHOLDSELL
  • 36. SERVICE BUS BROKERED MESSAGING “Durable messaging in the cloud” BUYHOLDSELL
  • 37. “Scalable mobile message distribution” BUYHOLDSELL SERVICE BUS NOTIFICATION HUBS
  • 38. BIZTALK SERVICES “Lightweight cloud message broker” BUYHOLDSELL
  • 39. APACHE KAFKA AMAZON SQS RABBIT MQ MULE SNAPLOGIC INFORMATICA
  • 40. EXAMPLE SCENARIO
  • 41. QUICKQUALIFY “Get multiple loan quotes within minutes!” BUSINESS NEED COMPANY OVERVIEW • Accept loan parameters from web customers • Store loan request in on-premises system • Reliably provide loan request to volatile list of external providers • Track responses and show the customer choices • High uptime, low latency integration • Strong development team • Over-extended Ops team • Decentralized org, local autonomy • Microsoft-friendly organization
  • 42. QUICKQUALIFY Customer website Loan DB ESB Services ETL CRM System Loan VendorLoan Vendor Loan Vendor CLOUD
  • 43. DEMONSTRATION
  • 44. CONCLUSION