Building reliable systems from unreliable components

Arnon Rotem-Gal-OzBuilding reliable systemsfrom unreliable components

Arnon Rotem-Gal-Oz

What’s in a 9

0.99 reliabilityWe have a nice little legacy business component

And we move it to SOA0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99

Failsafe hardware Status Technologies FT Server

Or try to detect failure , handle it andminimize its effect on the business service© Rosendahl

BEFORE WE BEGIN

SOA Adheres to Policy governed by Binds to End Point Exposes Serves Service Understands Contracts implements ServiceConsumer describes Key Component Sends/Receives Messages Sends/Receives Relation

Distributed Pipes and Layered Client Agents Filters System Server Stateless Comm. SOA is derived from SOA other styles

SOA vs. RESTDistributed Pipes and Layered Client Replicated Uniform Virtual Agents Filters System Server Repository Interface Machine Stateless Code On Cacheable Comm. Demand SOA REST

System Mobile Integration 3G Video DedicatedMonitoring MMS 3rd parties Calls Client Usage Datmart Applications Ad Management Targeted Campaign Billing Acquisition Interactions branding Advertizing Mgmt.Reporting ResourcesData mining Reference& Statistics Interactions Links Data Link Managment Reports Publishing Data Web Interaction tools Interfaces Front-end Designer integration

Load balancer Smart phones Web Web MMS 3G 3G Server Server Gateway Gateway Gateway (IIS/Apache (IIS/Apache Camera Phones DMZ Firewall Load balancer App App App App App App App Server Server Server Server Server Server Server Operational FirewallBackend BI & Sync. Paper NMS Links Reporting Registeration Server Editor Web Firewall Server (IIS/Apache DB Datamart DB Advertizing clients Links usage References Admin Console DMZ

CHALLENGE – SERVICEAVAILABILITY

What’s the effect of a failure - Server E1 line = 30 concurrent video Call Flow calls Service

Service Instance Service Business logicEnd point reaction Distribute request Dispatcher Edge Service Instance

What’s the effect of a failure - Channel Call Flow Call Flow Channel Call Flow Call Flow Call Flow Call FlowChannel Channel Channel Flow Channel Call Channel Call Flow Channel Call Flow Call Flow E1 line = 30 Channel Call Flow concurrent video Call Flow Channel Channel Channel calls Channel Call Flow Call Flow Channel FlowCall Channel Call FlowChannel Call FlowChannel Channel

Service Instance with NLBReal IP : 1.1.1.4 Real IP : 1.1.1.3 Real IP : 1.1.1.2 Windows Host Cluster Host Cluster Host Service Service Edge Instance Instance Windows Kernel Windows Kernel Windows Kernel TCP/IP TCP/IP TCP/IP NLB Driver NLB Driver NIC Driver NIC Driver NIC Driver NIC NIC NIC Virtual IP : 1.1.1.1

Virtual Endpoint

Request/Reply EndPoint 2. 1. Request Synchronous processing 3. ReplyService Consumer Service

Things look Cool & Simple ™3G SessionCall negations Render Image resutls extraction Translation identification to links

3G GW IVP (RV) (RV) SIP Listner 3G VAS (Cestel) RTP Image Extractor WS 3G Builder (Cestel) Alg. EngineWebRenderer Resource Manager Dispatcher WebConnector Turn out Complicated & Ugly ™

Parallel Pipelines Key SOA Component Pattern Component Queue Relation Concern/attribute pipeline pipeline EdgeRequest EndPoint Request 1 EndPoint EndPoint Request 2 Perform PerformReaction Task Task pipeline EndPoint Perform Task Service

Inversion of Communications

Consumer view var sendMmsEvent = new SendMmsEvent() { FromNumber = simpleMessageDetails.DialedNumber, Subject = mmsContents.Subject, ToNumber = simpleMessageDetails.Sender, ImageExtension = mmsContents.ImageExtension, ImageAsByteArray = mmsContents.Image, TextAsByteArray = mmsContents.Text }; eventBroker.RaiseEvent(sendMmsEvent); http://www.flickr.com/photos/crimson_wolf/2851737125/sizes/l/

Service view[ServiceContract]public interface ImPostOffice : ImContract, IHandleSendCoupon,IHandleSendSms,IHandleStatus,IHandleAdminStatus,IHandleWapLink,IHandleSendMms{}[ServiceContract]public interface IHandleSendMms{ [OperationContract] int SendMms(SendMmsEvent eventOccured);}[DataContract] public class SendMmsEvent : ImEvent { /// <summary> /// end users number. should be in international format: +[country-code]number. Example: +491737692260 /// </summary> [DataMember] public string ToNumber { get; set; } /// <summary> /// services number, usually a short-code. Example: 84343 /// </summary> [DataMember] public string FromNumber { get; set; }

Edge translates external structures to internal onespublic int SendMms(SendMmsEvent eventOccured) { var eventContext = eventOccured.ToString(); if (log.IsDebugEnabled) log.Debug("inside SendMms, event context = [" + eventContext + "]"); var fromNumber = eventOccured.FromNumber; var sender = mmsSenderFactory.Get(fromNumber); if (null == sender) { if (log.IsWarnEnabled) log.Warn("cannot get mms sender derived from " + (fromNumber ?? "null") + ""); return 0; } IMmsSubmitResponse response; try { var mmsMessageDetails = new MmsMessageDetails(eventOccured.ToNumber, eventOccured.TextAsByteArray, eventOccured.ImageAsByteArray, eventOccured.ImageExtension, eventOccured.Subject); response = sender.Submit(mmsMessageDetails); } catch (Exception ex) { log.Error("cannot send mms message, context = [" + eventContext + "]", ex); return 0; } if (log.IsInfoEnabled) { var responseMessage = (null == response) ? "null" : response.ToString(); log.Info("sent mms with event context = [" + eventContext + "], response = [" +responseMessage + "]"); }

Sagas tie instances together for conversations

Call Recovery Image ExtractorSIP Identification

Alternative : Orchestration Service Service reaction request Manage Protocol Process Schedule Coordinatorrouterequest monitor Workflow instance Offline designer Host Workflow Engine Workflows Auxiliary toolsOrchestration platform

Be Wary of Nano-Services

CHALLENGE - MANAGEMENT

Blogjecting Watchdog Edge Service EndPoint Monitor Watchdog Watchdog Monitor AgentRequest Edge EndPoint Heal Reports Monitor Report Monitor Log Monitor

Blogjects concept is about collaborating objects

WatchDogService A Service B WDWatcher

WatchDog Call Recovery Image Extractor SIP Identification

Service Monitor Collect Metrics collection Status Fault Monitoring Reporting & Security Dashboarding Edge/Service monitoring Status Policy governance Notify Commands Control In Monitor Act Edge/ServiceService Monitor

RESTful resource management root 1 2 3 Abcde/ Sessions/ Efgh/ / Resources/ Dispatchers/ Xyz/

http://devrig:52141/RM/Sessions/abc/• ATOMPUB – Session details • URI (ID) • State (start/end/status etc.) • Resources – Knows status – URI for the Resource representation on the RM – URI for the Resource itself

Keep the BIThttp://www.flickr.com/photos/37643027@N00/2050024263/sizes/o/

WatchDog Call Recovery Image Extractor X SIP Identification WatchDog 3G Call Liveliness Monitor

CHALLENGE – MULTI-TENNANCY(LIES, I TELL YOU, ALL LIES)

Same event different subscribers Player (interaction Call Flow Renderer) Play Movie Event Bridge to 3rd Call Flow Party

Routing[ServiceContract][Participate("3G")]public interface ImPlayer : ImContract, IHandleCallStarted, IHandleCIHandlePlayMovie,IHandleCallAborted{}[ServiceContract][Participate("3GPartner")]public interface ImXsightsGateWay : ImContract, IHandleCallAborted, IHandlePlayMovie, IHandleReadyForSearch, IHandleSearchStarted, IHandleJoinThirdParty{}

Raise a saga initiating eventInitiator A Participant A Capacity : 2 Participant BInitiator B Capacity : 1 Now what ?!

Reservation Pattern

Service Instance #1 Control Control Edge Edge Service Instance #2Business Event Control Business Event Logic Broker Edge Broker Logic Resource Event Business Resource Allocator Service Host Logic Broker Allocator Service Host Resource Host Service Service Host Allocator Service Host Service Host ControlReservation Edge Event Business Broker Logic Resource Service Host Allocator Service Host

Good old2PC to the rescue

Takeaways• Things break• Decouple• Fail Fast• Monitor & Detect• Compensate• Throw state unto others

Arnon Rotem-Gal-Oz @arnonrgo http://arnon.mehttp://www.cloudvalue.com

Building reliable systems from unreliable components

by Arnon Rotem-Gal-Oz on Mar 13, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 1

Statistics

Building reliable systems from unreliable components — Presentation Transcript