1. Introduction to Service Broker inSQL Server 2005/2008 Lior King Database & Infrastructure Group Manager SRL Group

2. 2 Agenda Messaging, SOA, SODA. What is Service Broker (SB) and is it good for. Microsoft’s alternatives to Service Broker. Messaging Basics: Conversation, Dialogs, Conversation Groups, Service SB building blocks: Message types, Contracts, Queues, Service programs. Messaging flow. Handling errors and poison messages. Internal & External activation Using conversations group lock. Routing and security

3. 3 Why Messaging? Asynchronous message processing Decoupling long duration business processes from the client app. For example: Sending a letter or an email. Getting scalability. Benefits of deferred message processing: The receiver controls when messages are processed. Load balancing occurs during peak times. Fault tolerance on the receiving side. Vital for Distributed Systems. Software components are running on different servers on the network Enabling scalability Handling security issues: Authentication, Encryption.

4. 4 What is SOA? SOA = Service Oriented Architecture. Explicit Service boundaries Exposing the available operations to other clients. Autonomous services. Minimum dependency on other services – loosely coupled. Explicit data contracts A contract defines the contents of a message sent in both directions. Interoperability Ability to work with any platform, including non-windows. Service broker is exposed to other clients through XML web services. Even Java clients can interact with Service broker.

5. 5 What is SODA? SODA – Service Oriented DATABASE Architecture Natively implementing messages (data) inside the database. To implement SODA, SQL Server offers: Integration into .NET (SQLCLR) Query Notifications Service Broker XML support Web Services support Business functionality - with SQLCLR stored procedures Message bus – with Service Broker Publishing – with SQL Server Web Service support.

6. What Is Service Broker? 6

7. 7 What is Service Broker (SB) A platform for building asynchronous, scalable database applications. Three features that distinguish SSB are reliable, ordered, and asynchronous delivery of messaging. Supports queuing out of the box which includes features like locking, ordering, and multithreading. We can use SB with a single instance or scale out across multiple instances

8. 8 Messaging complications solved by Service Broker (1) Performance With service broker, the queue is a database object and thus distributed transaction and operations are usually not needed. Queue Reader Management With multiple queues – like a supermarket. What happens when a new line in open? Queue balancing has a cost! The Service broker way – like an airport check in. Using one queue with multiple queue readers.

9. 9 Messaging complications solved by Service Broker (2) Transaction Management Determine how queue readers are coordinated against one queue. Example: An order header record and some order items records. Service broker uses "Conversation Group Locking" which ensures that the same queue reader processes messages from the same conservation group. Message Sequencing When there are several messages, each of which depends on a previous message. SB includes a sequence number with every sent message. In SQL 2008, SB includes built in support for conversation priorities.

10. 10 Messaging complications solved by Service Broker (3) Message Correlation. When a client sends a few messages and gets back a few responses. How can a client know which response is for which request? SB associates a "Conversation ID" to each message which the client can use to correlate responses to requests. Maintenance With MSMQ, messages are stored in the file system and data in the database. A "sync" backup will require stopping the MSMQ while performing the backup!!! With SB, when performing a database backup, you backup also the queues with the messages.Everything is IN SYNC.

11. What Service Broker is NOT? Not a general purpose messaging system – it only works for SQL Server. Supports only transactional messaging – if transactional messaging is not required this is unnecessary overhead. Not just a messaging system, we can use the queuing features on a single SQL Server instance or multiple instances. 11

12. 12 MS Messaging alternatives (1) MSMQ – Microsoft Message Queuing. Since the first version of Windows NT Comes with Windows for free. Not bound to a specific DB Message size – up to 4MB For distributed transaction – MSMQ needs touse DTC. Message ordering is NOT guaranteed. Message correlation – not supported. Queue readers are implemented manually. Synching and locking between queue readers - are implemented manually. Problem with backup consistency.

13. 13 MS Messaging alternatives (2) Queued Components Part of Component Object Model (COM+) infrastructure. Can enqueue a user request to a COM+ app and execute it asynchronously. Internally- it uses MSMQ to enqueue the message. On the server side, a component named "listener" is used to dequeue the message and calls the specified COM+ object. Used in projects that already uses the COM+ infrastructure.

14. 14 MS Messaging alternatives (3) BizTalk is a Microsoft product for Business Process Management (BPM). Enables to automate, orchestrate and optimize business processes. Includes tools for design, develop, deploy, and manage those processes. BizTalk can also manipulate the contents of messages over multiple transports, map message formats, manage workflows etc. Uses its own messaging for Enterprise Application Integration (EAI). High licensing cost (especially where scale out is needed). If your app just requires reliable messaging from one SQL Server instance to another - SSB is probably a better choice.

15. 15 MS Messaging alternatives (4) XML Web Services Based upon open standards such as SOAP and WSDL. Since .NET 1.0 - full support for creation of web services. Easy to design, implement, publish and reuse. No native queuing.

16. 16 MS Messaging alternatives (5) WCF = Windows Communication Foundation Since .NET 3.0. Based on standards. Provides the same API across different communication technologies. The WCF API enables to write apps in a communication-independent way. No native queuing. We can combine SB and WCF in applications.

17. 17 Service Broker Basics

18. 18 Conversation A conversation is a reliable, ordered exchange of messages between two SB services. Dialog – 2 way conversation between exactly two SB services Service 1 = Initiator service Service 2 = Target service Monologue – A one way conversation between a single publisher service and several subscriber services. It's a version of "publish-subscribe" paradigm. Only Dialogs are supported in SQL Server 2005/8.

19. 19 Dialog Guaranteed delivery – even if the receiver is currently offline. Long lived – From a few seconds to several years. Exactly once – No need to resend messages. In order delivery – SB ensures the messages are received in the same order. Persistence – Survives restarts of the DB server. Dialog lifetime: Each participant can end the conversation The Initiator can optionally specify a maximum lifetime for the dialog expiration). Both participants will track the lifetime. Conversations never live beyond their maximum lifetime (if specified).

20. 20 Conversation Groups Identifies one or more related conversations. Allow the SB to coordinate conversations involved in a specific task. A conversation can belong to one group only (one-to-many). Conversation Group Lock - When a message is initiated, SB locks the conversation group. Only one session at a time can receive message of the conversation group. Example: Books ordering on the internet: Credit card validation. Inventory service Accounting service Shipping service.

21. 21 Conversation Groups

22. 22 Message sequencing Each message in the conversation gets a sequence number starting at #1. The target tries to dequeue message #1, then #2 and so on. When a message is lost during the sending process, the target waits until the message is successfully re-sent. Reliable Delivery across the network Resending and acknowledgement is built in SB.

23. 23 Service A SB Service is a named endpoint to which messages from other services are sent. A SB service is a SQL Server object The interface is defined through the messages. Services contain the logic (code) and the state (data) Services communicate through conversions. Services are mapped to queues. Messages send to a service are stored in its associated queue. Each SB service consist of 4 DB objects: Message types Contracts Queue Service program The first three are DB objects. The Service program can be implemented: Internally – as a stored procedure. Externally – as a separate application.

24. 24

25. 25 Services deployment On the same SQL Server DB On the same SQL instance but on separate DBs. On different SQL instances and on separate DBs – on a different DB Servers. The services location is transparent during the development.

26. 26 ואחרי כל ההקדמה התיאורית הגיע זמן לקצת "תכל'ס"

27. 27 Message Types Defines the type of data that a message contains. Must be created in each DB that participates in a conversation Can perform the following validations: Optional XML validation against XML schema definition (XSD). Well formed XML No validation (for example: when there is binary data) Empty (the message body MUST be empty). NOTE: XML validations can impact the overall performance (due to parsing). Recommended for developing and testing only.

28. 28 Message Types Beware of message type conflicts. It’s conventional to prefix each message type and contract with your own internet domain. Use the DDL command CREATE MESSAGE TYPE. Example: CREATE MESSAGE TYPE [http://mydomain/SBMessages/RequestMessage] Validation=NONE Use sys.service_message_types catalog view to get information about the defined message types. To change a message type, use the command ALTER MESSAGE TYPE. A [DEFAULT] message type exists in every database.

29. 29 Contracts An agreement between two SB services. Defines the message types a SB service uses. Service Broker ensures that only message types that are defined in the contract are processed. Determines which message type is use by the initiator and which by the target (or used by both): SENT BY INITIATOR SENT BY TARGET SENT BY ANY

30. 30 Contracts Use the DDL command: CREATE CONTRACT. Example: CREATE CONTRACT [http://mydomain/SBMessages/DemoContract] ([http://mydomain/SBMessages/RequestMessage] SENT BY INITIATOR, [http://mydomain/SBMessages/RequestMessage] SENT BY TARGET) Use sys.service_contracts and sys.service_contract_message_usages to get the contracts and the message types associated to each contract. To change a contract – use DROP CONTRACT and CREATE CONTRACT again (ALTER will not work). A [DEFAULT] contract exists in every database.

31. 31 Queues A queue – a storage provider for messages that are sent/received. Must be defined for the initiator and for the target services. When a message is sent to a target, it is enqueued in the target's queue. The target service dequeues the message and processes it. When the processing is done – the transaction is committed and the message is removed from the queue. The queue is implemented by hidden tables. Queues can be only queried for monitoring.

32. 32 Queues Use the DDL command: CREATE QUEUE. A simple example: CREATE QUEUE InitiatorQueue WITH STATUS= ON CREATE QUEUE TargetQueue WITH STATUS= ON Use the sys.service_queues catalog view to get information about the queues defined.

33. 33 Service Programs Service programs process incoming messages from the queue. SB can activate a service program automatically when a message arrives. Service programs activation can also be scheduled. The Response to the initiator is a part of the message's conversation.

34. 34 Service Programs Use the DDL command CREATE SERVICE. Example: CREATE SERVICE InitiatorService ON QUEUE InitiatorQueue ( [http://mydomain/SBMessages/DemoContract]) CREATE SERVICE TargetService ON QUEUE TargetQueue ( [http://mydomain/SBMessages/DemoContract]) Use the sys.services catalog view to get information about the services defined. Use the ALTER SERVICE to change the service definition. Example: ALTER SERVICE TargetService ON QUEUE MyNewTagetQueue ( ADD CONTRACT [MyNewContract] DROP CONTRACT [http://mydomain/SBMessages/DemoContract])

35. 35 Sending & Receiving messages Sending Message In order to start the messaging, you need to: Create a new conversation between 2 SB services with BEGIN DIALOG command. Send a message of the required message type with SEND ON CONVERSATION command. Receiving a Message Use the RECEIVE command. Using RECEIVE is like doing a “SELECT FROM QUEUE” command. Changes the @@ROWCOUNT system variable. Can be combined with WAITFOR… TIMEOUT command for a limiting the waiting time.

36. 36 Ending a conversation Use the END CONVERSATION command. SB sends a message of the “built-in” message type [http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog] You must explicitly end SB conversations on both service sides. If you forget to end a conversation on one side, the conversation will remain open. Can use WITH CLEAN clause to remove all messages and information without notifying the other side. This is useful if the remote service has been suddenly removed.

37. 37 Poison messages A poison message is a message that will always cause the receiver to fail with its processing. Example: A primary key violation. A deadlock – is NOT a poison message. A retry might eventually succeed. SB disables a queue that experiences 5 rollbacks in a row. It can be resolves manually or automatically with an application that subscribes to the poison message notification, using CREATE EVENT NOTIFICATION. Re-enable the queue with: ALTER QUEUE <queuename> WITH STATUS = ON

38. 38 Error Messaging (1) Use END CONVERSATION error message to send an error and end the conversation at the same time. Example: END CONVERSATION @ch WITH ERROR = 1234 DESCRIPTION = 'My error msg' The actual XML sent is: <Error xmlns="http://schemas.microsoft.com/SQL/ServiceBroker/Error"> <Code>1234</Code> <Description>My error message</Description> </Error>

39. 39 Error Messaging (2) The service that receives the error can check for an error: IF (@messagetypename = 'http://schemas.microsoft.com/SQL/ServiceBroker/Error') BEGIN SET @errorcode = (SELECT @messagebody.value( N'declare namespace brokerns= "http://schemas.microsoft.com/SQL/ServiceBroker/Error"; (/brokerns:Error/brokerns:Code)[1]', 'int')); SET @errormessage = (SELECT @messagebody.value( N'declare namespace brokerns= "http://schemas.microsoft.com/SQL/ServiceBroker/Error"; (/brokerns:Error/brokerns:Description)[1]', 'nvarchar(3000)')); END

40. 40 DEMO

41. 41 Activation Polling the queue to see if messages had arrived wastes a lot of resources when few messages are arriving on the queue. Activation allows you to start a service program as soon as a new message arrives in the service queue: Internal activation: For activating a stored procedure. External activation: For activating an external application. Parallel activation: For starting more than one stored procedure simultaneously. Troubleshooting activation. Activation is performed using “queue monitors” which determine how and when to activate the service.

42. 42 Internal activation - steps Configure a service queue according to your requirements Create (or alter) a queue that enables activation. Example: CREATE QUEUE [TargetQueue] WITH ACTIVATION ( STATUS=ON, PROCEDURE_NAME = [YourServiceProcName], MAX_QUEUE_READERS=1, EXECUTE AS SELF) MAX_QUEUE_READERS – maximum instances of the stored proc that the queue starts at the same time. This is actually the level of parallelism by which messages in the queue are processed. EXECUTE AS – The DB account under which the activated stored proc runs. Using EXECUTE AS SELF means that the stored proc executes as the current user.

43. 43 Activating a service on a different DB The activation procedure needs to call a stored procedure on another database. The activation procedure might not have sufficient privileges to call another database. To fix this, you must sign the activated stored procedure with a certificate and map a user with the needed permission to this certificate. Another option is to set the database as trustworthy (not recommended for production environment): ALTER DATABASE <db> SET TRUSTWORTHY ON

44. 44 Processing many queues with a single procedure Create the necessary queues with the associated services. Set up internal activation on each queue and point to the same procedure. Write a procedure that determines which queue you’ll use to get messages from. To determine from which queue has activated the procedure: SELECT @queue_name = sq.[name] FROM sys.service_queues sq INNER JOIN sys.dm_broker_activated_tasks bat ON bat.spid=@@spid and sq.object_id = bat.queue_id

45. 45 DEMO

46. 46 External Activation External activation activates all kinds of windows applications (console apps, windows services, win forms etc). Internal activation is not recommended for long-duration work inside a stored procedure. External activation uses SQL Server event notification mechanism. Use CREATE EVENT NOTIFICATION command.

47. Why use event notification Needed for external service that should be activated by more than one queue 47

48. 48 External Activation – Steps example CREATE QUEUE ExternalActivatorQueue CREATE SERVICE ExternalActivatorService ON ExternalActivatorQueue ( [http://schemas.microsoft.com/SQL/Notifications /PostEventNotification] ) CREATE EVENT NOTIFICATION EventNotifTargetQueue ON QUEUE TargetQueue FOR QUEUE_ACTIVATION TO SERVICE 'ExternalActivatorService', 'current database'

49. 49 Using managed code Use the ServiceBrokerInterface solution that can be found in the samples directory of SQL Server 2008 Contains classes such as Conversation, Message, Service. You can build a service class derived from the class “Service”. c#

50. 50 DEMO

51. 51 Conversation Groups Ensures ordered message processing and supports synchronization support for multiple queue readers. Multiple queue readers can gain SCALABILITY. Ensures that only one task can read messages from a particular conversation at the same time. Every conversation belongs to a conversation group. A conversation group can contain one or more conversations. Use the WITH RELATED_CONVERSATION clause when creating a new dialog You might need to create an application state table for more complex business logic and monitoring.

52. 52 Using Conversation Groups Use the WITH RELATED_CONVERSATION clause when creating a new dialog. BEGIN DIALOG @ch1 FROM SERVICE [InitiatorService] TO SERVICE 'TargetService1' WITH ENCRYPTION = OFF; BEGIN DIALOG @ch2 FROM SERVICE [InitiatorService] TO SERVICE 'TargetService2' WITH RELATED_CONVERSATION = @ch1, ENCRYPTION = OFF;

53. Distributed Architecture You’ll need to set up the following: Remote Service Bindings Routes Service Broker Endpoints 53

54. Remote Service Binding A remote service bindingestablishes a relationship between: Local database user The certificate for the user The name of a remote service. A remote service binding specifies the identity to be used to authenticate to the remote service. Use T-SQL command: CREATE REMOTE SERVICE BINDING 54

55. 55 Routes SQL Server objects that specify on which IP a SB service is located. By default, each database contains a route that specifies that messages for any service which does not have an explicit route are delivered within the SQL Server instance. Enables flexible deployment. Development can be on the same DB and on the same instance. On production, can be on different DB servers.

56. Routes 56

57. Endpoints A SB endpoint configures SQL Server to send and receive Service Broker messages over the network. Endpoints are specified using service names. Listens on a specific port. Options for transport security and message forwarding. 57

58. 58 Security SB allows services to communicated even when the instances are on different machines, have no trusted relationship and on different networks. 2 levels of security: Transport security Dialog security

59. Transport security Windows-based authentication - using NTLM or Kerberos. Can be used only on servers of the same domain or server on different domains that are trusted. Certificate-based authentication – using certificates. Usually used with services that reside on different physical networks on distrusted domains. Dialog security Service Broker supports "forwarders". A forwarder is a SQL Server instance that accepts service broker messages and forwards them to the next hop on the route to the target service. Using a forwarder requires the decryption of the message and encrypting it again. Causes slower performance. 59

60. Dialog security Secures each individual dialog between two dialog endpoints, regardless of how many networks the messages traverse as the travel between the conversation endpoints. Dialog security establishes authenticated and encrypted communication between two services. Dialog security provides encryption, remote authentication, and remote authorization for a specific conversation. 60

61. 61 Summary Service Broker is a good choice for transactional, data processing based messaging solution. Service broker database objects such as: Message types, Contracts, Queues and Services can be easily built using T-SQL. SB can Handling errors and poison messages. SB can maintain a logical connection between messages using conversation groups. SB supports Internal activation as well as external activation. SB can support distributed application architecture using routes. SB can secure the messages payload with the excellent encryption algorithms SQL Server provides.

62. 62 Thank you for listening Lior King Database & Infrastructure Group Manager SRL Group Download this presentation from my Blog: KingLior.WordPress.Com My Email: LiorK@SRL.co.il