MySpace SQL Server Service Broker

Service Dispatcher: The MySpace Implementation of Service Broker Speaker: Christa Stelzmuller MySpace Chief Data Architect San Francisco SQL Server User Group September 14, 2009 Mark Ginnebaugh, User Group Leader www.bayareasql.org

Christa Stelzmuller Chief Data Architect at MySpace since Oct 2006 Formerly at Yahoo! Engineering Manager Data Architect for the Yahoo! Music Team Specializes in very large databases with high volumes of transactions Tonight’s Topic: Service Dispatcher: The MySpace Implementation of Service Broker

What is Service Broker? Message Queue Asynchronous Unicast Intra- and Inter-database, instance, and server Integrated into the database engine Instance level DML extensions

Why Service Broker? Flexible Atomic transactions can be split across synchronous and asynchronous work items Work items can occur in parallel Predictable Asynchronous work items are processed at a constant rate leveling peaks and valleys Transactional In-Database Common administration

Challenges with Service Broker Unicast Communication is from a single sender to a single target Excessive message transmission Routing Requires a physical route between each service Unmanageable number of routes Non-Intuitive Four object types must be implemented for each service, and up to seven understood High learning curve for average database developer

Introducing Service Dispatcher

Core Features Simulated Multicast Communication is from a single sender to a specified set of targets Wildcard targets support simulated broadcast Route Management Routes to all dispatcher client enabled databases are managed centrally Abstraction API for “SEND” Friendly database names are used instead of service broker instances

Advanced Features Dialog Management Technique of dialog reuse abstracted Pooling based on spid, service, and group Route Receipts Used in simulated broadcast scenarios to return information on all servers from which acknowledgement should be expected Extensible Publish/Subscribe (target agnostic messages) Notifications

Major Use Cases Broker Gateway Asynchronous message delivery APIs Load balanced farm Abstracted data destination Example Usage Activities Stream

Major Use Cases Backend Initiated Processing Foreign keys and relationships spread across many databases Coordinated messaging Example Usage Profile management Content take-downs

Major Use Cases List Broadcast Local data used to broadcast updates to lists Two views of the relationship represented Example Usage Photo Tags Friend Categories

Developer Highlights Message Types No need for explicit service broker message types If needed, application message types are supported as part of the message body Standard set of message types, system and dispatcher Contracts No need for explicit service broker contracts Each application service must contain a reference to at least one dispatcher contract

Developer Highlights Conversation Group Locking Alternative to GET CONVERSATION GROUP Dialog consolidation for performance Requires explicit conversation group lock release Conversation Ending Dispatcher specific state tables Error handling

Administrator Highlights Build Central catalog Common client code deployed Automated services Dispatcher route creation Client route creation Database Configuration GET CONVERSATION GROUP Maximum valid groups Dialog pool size Dialog timeout settings

Administrator Highlights Health & Monitoring Reports Scheduled Ad-Hoc Notifications Automated queue reactivation Quarantine Ping

Unsupported Broker Features Message Ordering Farm of servers, random access Dialog death & resend Abstracted conversation management Fire & Forget Conversation management is tightly controlled by dispatcher, and logic to properly handle conversation ends and errors is built in. Begin/End Conversation Applications cannot control dialog state, this is entirely managed by dispatcher

Upcoming Features Dialog Emulation Introducing the concept of circuits Message sequence number Message ordering becomes possible Applications can control circuit state Abstracted activation logic RECEIVE logic abstracted Internal broker message type handling

Service Broker Stats Average Average Peak Max Peak Farm Total MySpace DB Sends/Sec/Server Sends/Sec/Server Sends/Sec/Server Sends/Sec Dispatcher 45 229 816 9,164 Profile 90 258 925 130,290 Mail 40 2,397 8,546 1,210,485 Shared 264 540 - - Features 21 36 37 720 Average Average Peak Max Peak Farm Total MySpace DB Rcvs/Sec/Server Rcvs/Sec/Server Rcvs/Sec/Server Receives/Sec Dispatcher 10 33 48 1,328 Profile 60 303 432 153,015 Mail 38 3,186 4,980 1,608,930 Shared 218 351 - - Features 18 92 119 1,840

When not to use Dispatcher Local asynchronous messages One-to-one messaging When an unsupported feature is required

Alternatives to Dispatcher Abstracted SEND logic (same as dispatcher without the router component) Dynamic route management (Microsoft’s solution

More Information Case Study http://www.microsoft.com/casestudies/Case_Study_Detail. aspx?CaseStudyID=4000004532 Codeplex Soon to come!

Credits Creator & Original Concept, Mikhail Berlyant Principal Dispatcher Architect, Mark Lovretovich Contributors, Kevin Stephenson & Justin Engelman

To learn more or inquire about speaking opportunities, please contact: Mark Ginnebaugh, User Group Leader mark@designmind.com

http://www.designmind.com	824
http://markginnebaugh.com	62
http://www.slideshare.net	12
http://webcache.googleusercontent.com	2
http://sps2010.designmind.com	2
http://static.slidesharecdn.com	1

MySpace SQL Server Service Broker

by Mark Ginnebaugh on Oct 16, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

6 Embeds 903

Statistics

MySpace SQL Server Service Broker — Presentation Transcript