MySpace SQL Server Service Broker

5,807 views

Published on

Service Dispatcher: The MySpace Implementation of Service Broker

Christa Stelzmuller, Chief Data Architect at MySpace, speaking to the San Francisco SQL Server User Group in October 2009. http://bit.ly/cTgAH

Christa explains the architecture and code used to power Service Dispatcher, the broker based component MySpace built to enhance the broker feature set. This presentation covers:

•MySpace use cases leading to Service Broker as the choice for ETL and distributed transaction management.

•How MySpace handles data integrity in a distributed environment.

•Technical details of MySpace's Service Dispatcher component.

•How you can simplify your Service Broker applications.

Published in: Technology
1 Comment
5 Likes
Statistics
Notes
No Downloads
Views
Total views
5,807
On SlideShare
0
From Embeds
0
Number of Embeds
937
Actions
Shares
0
Downloads
100
Comments
1
Likes
5
Embeds 0
No embeds

No notes for slide

MySpace SQL Server Service Broker

  1. 1. Service Dispatcher: The MySpace Implementation of Service Broker Speaker: Christa Stelzmuller MySpace Chief Data Architect San Francisco SQL Server User Group September 14, 2009 Mark Ginnebaugh, User Group Leader www.bayareasql.org
  2. 2. Christa Stelzmuller Chief Data Architect at MySpace since Oct 2006 Formerly at Yahoo! Engineering Manager Data Architect for the Yahoo! Music Team Specializes in very large databases with high volumes of transactions Tonight’s Topic: Service Dispatcher: The MySpace Implementation of Service Broker
  3. 3. What is Service Broker? Message Queue Asynchronous Unicast Intra- and Inter-database, instance, and server Integrated into the database engine Instance level DML extensions
  4. 4. Why Service Broker? Flexible Atomic transactions can be split across synchronous and asynchronous work items Work items can occur in parallel Predictable Asynchronous work items are processed at a constant rate leveling peaks and valleys Transactional In-Database Common administration
  5. 5. Challenges with Service Broker Unicast Communication is from a single sender to a single target Excessive message transmission Routing Requires a physical route between each service Unmanageable number of routes Non-Intuitive Four object types must be implemented for each service, and up to seven understood High learning curve for average database developer
  6. 6. Introducing Service Dispatcher
  7. 7. Core Features Simulated Multicast Communication is from a single sender to a specified set of targets Wildcard targets support simulated broadcast Route Management Routes to all dispatcher client enabled databases are managed centrally Abstraction API for “SEND” Friendly database names are used instead of service broker instances
  8. 8. Advanced Features Dialog Management Technique of dialog reuse abstracted Pooling based on spid, service, and group Route Receipts Used in simulated broadcast scenarios to return information on all servers from which acknowledgement should be expected Extensible Publish/Subscribe (target agnostic messages) Notifications
  9. 9. Major Use Cases Broker Gateway Asynchronous message delivery APIs Load balanced farm Abstracted data destination Example Usage Activities Stream
  10. 10. Major Use Cases Backend Initiated Processing Foreign keys and relationships spread across many databases Coordinated messaging Example Usage Profile management Content take-downs
  11. 11. Major Use Cases List Broadcast Local data used to broadcast updates to lists Two views of the relationship represented Example Usage Photo Tags Friend Categories
  12. 12. Developer Highlights Message Types No need for explicit service broker message types If needed, application message types are supported as part of the message body Standard set of message types, system and dispatcher Contracts No need for explicit service broker contracts Each application service must contain a reference to at least one dispatcher contract
  13. 13. Developer Highlights Conversation Group Locking Alternative to GET CONVERSATION GROUP Dialog consolidation for performance Requires explicit conversation group lock release Conversation Ending Dispatcher specific state tables Error handling
  14. 14. Administrator Highlights Build Central catalog Common client code deployed Automated services Dispatcher route creation Client route creation Database Configuration GET CONVERSATION GROUP Maximum valid groups Dialog pool size Dialog timeout settings
  15. 15. Administrator Highlights Health & Monitoring Reports Scheduled Ad-Hoc Notifications Automated queue reactivation Quarantine Ping
  16. 16. Unsupported Broker Features Message Ordering Farm of servers, random access Dialog death & resend Abstracted conversation management Fire & Forget Conversation management is tightly controlled by dispatcher, and logic to properly handle conversation ends and errors is built in. Begin/End Conversation Applications cannot control dialog state, this is entirely managed by dispatcher
  17. 17. Upcoming Features Dialog Emulation Introducing the concept of circuits Message sequence number Message ordering becomes possible Applications can control circuit state Abstracted activation logic RECEIVE logic abstracted Internal broker message type handling
  18. 18. Service Broker Stats Average Average Peak Max Peak Farm Total MySpace DB Sends/Sec/Server Sends/Sec/Server Sends/Sec/Server Sends/Sec Dispatcher 45 229 816 9,164 Profile 90 258 925 130,290 Mail 40 2,397 8,546 1,210,485 Shared 264 540 - - Features 21 36 37 720 Average Average Peak Max Peak Farm Total MySpace DB Rcvs/Sec/Server Rcvs/Sec/Server Rcvs/Sec/Server Receives/Sec Dispatcher 10 33 48 1,328 Profile 60 303 432 153,015 Mail 38 3,186 4,980 1,608,930 Shared 218 351 - - Features 18 92 119 1,840
  19. 19. When not to use Dispatcher Local asynchronous messages One-to-one messaging When an unsupported feature is required
  20. 20. Alternatives to Dispatcher Abstracted SEND logic (same as dispatcher without the router component) Dynamic route management (Microsoft’s solution
  21. 21. More Information Case Study http://www.microsoft.com/casestudies/Case_Study_Detail. aspx?CaseStudyID=4000004532 Codeplex Soon to come!
  22. 22. Credits Creator & Original Concept, Mikhail Berlyant Principal Dispatcher Architect, Mark Lovretovich Contributors, Kevin Stephenson & Justin Engelman
  23. 23. To learn more or inquire about speaking opportunities, please contact: Mark Ginnebaugh, User Group Leader mark@designmind.com

×