Your SlideShare is downloading. ×
0
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
1 of 26 Scaling and Fault Tolerance for Distributed Messages ...
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

1 of 26 Scaling and Fault Tolerance for Distributed Messages ...

517

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
517
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Clients with poor and broken connections do not get missed events
  • We would like to have an open architecture We would like to enable late and broken clients to receive the parts of the stream that they have missed
  • Access Grid (AG) Uses MBONE tools (VIC – for video, RAT – for audio) Voyager Open source archiving tool used to record audio/video streams in MBONE sessions. Records and plays back AV streams in AccessGrid sessions. Stores them to the local storage.  Clients connect to the server using RTSP and receives the streams. InSORS Unicast support for clients that do not have multicast support. IG Recorder Similar to Voyager, it records audio/video streams as well as other data streams (i.e. powerpoint slides) in AG sessions. Virtual Rooms Videoconferencing System (VRVS) The closest system to GlobalMMCS ***** If needed, use the following Reflectors play a big role in VRVS design. A reflector is a host that connects each client to a Virtual Room by a permanent IP tunnel. The reflectors have many IP tunnels among themselves. The clients multiplex these tunnels. The reflectors and their links form a set of virtual sub-networks through which audio, video or data flows. The use of the reflector technology assures the quality needed for videoconferences transmission. Reflectors are the ones responsible for transmitting the streams. The basic architecture of VRVS is shown in figure 4.
  • RTSP – Real Time Streaming Protocol VCR-like control protocol over media. Clients can perform operations such as pause, fast forward, reverse, and absolute positioning etc. No limitation on the transport protocol; multicast UDP, unicast UDP or TCP can be used. An RTSP server can use any type of packet format (RTP, RDT) for sending media data to an RTSP client. SETUP : Causes the server to allocate resources for a stream and start an RTSP session. PLAY and RECORD : Starts data transmission on a stream allocated via SETUP. PAUSE : Temporarily halts a stream without freeing server resources. TEARDOWN : Frees resources associated with the stream. The RTSP session ceases to exist on the server.
  • RTCP is used for statistical purposes.
  • QT -> Quicktime format
  • GlobalMMCS The current prototype of GlobalMMCS includes: A XGSP media server provides services for bridging multicast and unicast, video-switching, video-mixing and audio-mixing to H.323, SIP as well as AG endpoints. H.323, SIP gateways and Real Servers for A/V clients XGSP A/V Session Server manages real-time A/V sessions, receives messages from gateways and the web server, and performs appropriate actions on the media server. The web server provides an easy-to-use web interface for users to join multimedia sessions and for administrators to perform administrative tasks.
  • We use NB to transport real-time audio/video data
  • Scalable replay service How to discover the right replica for replay Effect of network threshold on the number of clients supported by a single replay service How many requests a replay service can support Effect of using different types of databases on this number; file system, mySQL, Oracle How caching will improve the service
  • Do we need extensions to RTSP? Are there any methods that need to be added to RTSP protocol?
  • We expect to build the following components to implement the prototype system. NB Time Service NTP is used to synchronize timekeeping among a set of distributed time servers and clients. It defines the architectures, algorithms, entities and protocols used by NTP. NB Buffering Service It releases events based on 3 parameters. If any of these criteria is satisfied it releases the events. Buffer size in KB Number of events in the buffer The duration of release Delay introduced by the buffer service can vary based on the above parameter values. High Resolution Clock Gives 2-3 usec resolution Provides sleep(long duration) API to enable threads sleep under 10 msec. Calls to sleep function with an input < 10 msec does not work in Java.
  • NB Replay Service Should provide API to support RTSP semantics. Data is archived as NB Events regardless of the type of the data. Current NB replay service does not have RTSP semantics. RTSP Server / Proxy Ability to manage NB archive and replay services Ability to dynamically locate replay and archiving services maintaining load balance among replicas. Ability to switch between replicas to provide robust streaming to clients in case of node failures
  • RTP packets or NB events
  • RTP packets or NB events
  • If 2 fails, 3 streams to the topic
  • RTP packets or NB events
  • An fault tolerant architecture for collaboration systems Switching between replay services during failures A persistent collaboration environment backed up with archiving and replay services Support for heterogeneous clients (desktop and mobile devices) Support for RTSP clients on cellular phones.
  • Provide distributed archiving/replay system
  • Transcript

    • 1. Scaling and Fault Tolerance for Distributed Messages in a Service and Streaming Architecture Thesis Proposal Hasan Bulut [email_address]
    • 2. Outline <ul><li>Motivation </li></ul><ul><li>Goals of the Architecture & Example Applications </li></ul><ul><li>Literature Survey </li></ul><ul><li>Research Issues and Tasks </li></ul><ul><li>Milestones </li></ul><ul><li>Typical Scenarios </li></ul><ul><li>Tests </li></ul><ul><li>Contributions </li></ul><ul><li>Summary </li></ul>
    • 3. Motivation <ul><li>Collaboration systems enable people to collaborate with each other. However, there are various open research issues in these systems. Some of them are: </li></ul><ul><li>A more fault tolerant system </li></ul><ul><ul><li>A distributed and replicated archiving system </li></ul></ul><ul><li>An architecture or framework to cope with network failures </li></ul><ul><ul><li>A mechanism to recover from failures while session is recorded </li></ul></ul><ul><li>Playback is available only after the session is over </li></ul><ul><ul><li>Playback mechanism for live sessions </li></ul></ul>
    • 4. Motivation <ul><li>An architecture or framework to recover late or broken clients </li></ul><ul><ul><li>Late clients will miss parts of the session that have already passed </li></ul></ul><ul><li>Extending services to unicast clients </li></ul><ul><ul><li>What happens if multicast feature is disabled on the network? </li></ul></ul><ul><li>Support for heterogeneous clients </li></ul><ul><ul><li>Support for videoconferencing (i.e. H.323 clients) and streaming clients (i.e. RealOne player) </li></ul></ul><ul><ul><li>Support for desktops and mobile devices such as cellular phones. </li></ul></ul>
    • 5. Goals of the Architecture <ul><li>A service oriented architecture </li></ul><ul><ul><li>Provide RTSP (Real Time Streaming Protocol) semantics </li></ul></ul><ul><ul><li>Compatible with Web Services standards and technologies </li></ul></ul><ul><li>Persistent and fault tolerant architecture </li></ul><ul><ul><li>A distributed and replicated archiving system in a messaging system environment </li></ul></ul><ul><ul><li>Dynamic replay service. Ability to switch among distributed replay services in case node failures </li></ul></ul><ul><li>Scalable architecture </li></ul><ul><ul><li>Allow a large number of clients to connect to the system. </li></ul></ul><ul><ul><li>Allow heterogeneous (different types of) clients to connect to the system </li></ul></ul>
    • 6. Goals of the Architecture <ul><li>Provide a flexible and extendable framework for new services </li></ul><ul><ul><li>Allow instant replay of streams. With this feature, it would be possible to annotate streams </li></ul></ul><ul><li>Improve Quality of Service (QoS) </li></ul><ul><ul><li>Time ordering of events </li></ul></ul><ul><ul><li>Maintaining the time spacing between consecutive events </li></ul></ul><ul><li>Enable late and broken clients to receive the past events (streams) </li></ul><ul><li>A generic architecture that can work with any collaboration tool, such as audio/video, whiteboard, text chat etc. </li></ul>
    • 7. Example Applications <ul><li>Consider a late client joining live audio/video session. This client has three options: </li></ul><ul><ul><li>Does not care about the missed stream. </li></ul></ul><ul><ul><li>Plays the missed stream in a faster mode until he/she catches up with the live stream. </li></ul></ul><ul><ul><li>Plays the stream from the beginning and follows the live session from behind. </li></ul></ul><ul><li>The stream is not necessarily a video stream. It can be events from a shared displays/applications such as whiteboards or from other collaboration tools. </li></ul><ul><li>Client can play a 2-hour long archived stream in 30 min (scaling 2-hour stream to 30-min stream). </li></ul>
    • 8. Literature Survey <ul><li>Collaboration systems </li></ul><ul><ul><li>Access Grid, InSORS, VRVS, Web based collaboration tools (WebEx, Centra) </li></ul></ul><ul><li>Archiving and replay services used in collaboration systems </li></ul><ul><ul><li>Voyager, IG Recorder </li></ul></ul><ul><li>Streaming media standards </li></ul><ul><ul><li>SMIL, RTSP (RFC 2326), RTP/RDT, data types such as H.261, H.263, MPEG-4, RealMedia </li></ul></ul><ul><li>XGSP – XML Based General Session Protocol; GlobalMMCS </li></ul><ul><li>NaradaBrokering - Distributed messaging infrastructure </li></ul>
    • 9. Collaboration Systems <ul><li>Access Grid (AG) </li></ul><ul><ul><li>Uses Internet2 multicast for audio/video transmission. </li></ul></ul><ul><ul><li>Voyager: Open source archiving tool used to record audio/video streams in MBONE sessions. </li></ul></ul><ul><li>InSORS: Can be viewed as a commercial version of AG. </li></ul><ul><ul><li>IG Recorder </li></ul></ul><ul><ul><ul><li>Similar to Voyager, it records audio/video streams as well as other data streams (i.e. powerpoint slides) in AG sessions. </li></ul></ul></ul><ul><li>VRVS </li></ul><ul><ul><li>Provides some kind of integration of different A/V endpoints. </li></ul></ul><ul><ul><li>No information about archiving system. </li></ul></ul><ul><li>WebEx / Centra : Web based collaboration systems. </li></ul><ul><ul><li>Recording and playback is done in a traditional way; session is recorded in a local storage. </li></ul></ul>
    • 10. Streaming Media Standards <ul><li>RTSP – Real Time Streaming Protocol </li></ul><ul><ul><li>NOT a transport protocol. </li></ul></ul><ul><ul><li>VCR-like control protocol over media. </li></ul></ul><ul><ul><li>Stateful server-client communication. </li></ul></ul>RTSP States Init Ready SETUP TEARDOWN PLAY / RECORD PAUSE TEARDOWN Playing / Recording
    • 11. Streaming Media Standards <ul><li>SMIL - Synchronized Multimedia Integration Language </li></ul><ul><ul><li>“ An XML-based language that allows authors to write interactive multimedia presentations” </li></ul></ul><ul><ul><li>Multiple streams can be presented in a synchronized timeline. </li></ul></ul><ul><li>Real Time Transport Protocol – RTP </li></ul><ul><ul><li>Usually used in conjunction with RTCP. </li></ul></ul><ul><ul><li>RTSP server can deliver media data using RTP </li></ul></ul><ul><li>RealNetworks’ Data Transport – RDT </li></ul><ul><ul><li>RealNetworks’ proprietary standard to deliver media. </li></ul></ul><ul><ul><li>Can be used over UDP or TCP </li></ul></ul><ul><li>Data types </li></ul><ul><ul><li>H.261, H263, JPEG , etc. (mostly used in VC systems) </li></ul></ul><ul><ul><li>RealMedia, MPEG, etc. (mostly used in RTSP streaming clients) </li></ul></ul>
    • 12. Streaming Servers <ul><li>Streaming servers are implementation of RTSP. Support for RTSP may vary. </li></ul><ul><li>Helix Streaming Server </li></ul><ul><ul><li>Streaming server from RealNetworks </li></ul></ul><ul><ul><li>Open source version has limited capability. Formats: RealMedia, mp3 </li></ul></ul><ul><ul><li>Commercial version provides live archiving to the local storage (as media files). Formats: RealMedia, mp3, mpeg-4, QT and WM </li></ul></ul><ul><li>Darwin Streaming Servers </li></ul><ul><ul><li>Open source streaming server from Apple. </li></ul></ul><ul><ul><li>Supports QT format. </li></ul></ul><ul><ul><li>Archives the session to the local storage (as media files) </li></ul></ul>
    • 13. XML Based General Session Protocol (XGSP) <ul><li>XGSP is a conference control framework. </li></ul><ul><li>The goal of XGSP is to integrate heterogeneous systems into one collaboration system. </li></ul><ul><li>Includes three components; user session management, application session management and floor control. </li></ul><ul><li>SIP is a non-XML text-based signaling protocol for Internet conferencing, telephony and instant messaging </li></ul><ul><li>GlobalMMCS : A prototype system to verify and refine XGSP conference control framework. </li></ul><ul><ul><li>A XGSP media server </li></ul></ul><ul><ul><li>H.323, SIP gateways and Real Servers for A/V clients </li></ul></ul><ul><ul><li>XGSP A/V Session Server </li></ul></ul><ul><ul><li>The web server </li></ul></ul>
    • 14. NaradaBrokering (NB) <ul><li>Virtualizes communication transport and endpoints </li></ul><ul><ul><li>UDP, TCP, Multicast, SSL ….. </li></ul></ul><ul><li>Based on a distributed network of cooperating broker nodes. (brokers support software overlay network) </li></ul><ul><li>Efficiently routes (content or endpoint-based) information from producers to consumers of content. </li></ul><ul><ul><li>Subscriptions can be based on SQL, Regular expressions and XPath queries. </li></ul></ul><ul><li>Been deployed and tested in the context of multimedia conferencing and Grid applications. </li></ul><ul><li>Introduces delays of order one to two milliseconds at each broker </li></ul>
    • 15. Research Issues <ul><li>We need to research capabilities/services that need to exist in a messaging system to achieve a higher quality of service (qos) of archiving and replay service </li></ul><ul><ul><li>Effect of </li></ul></ul><ul><ul><ul><li>Timestamping events using NTP on achieving synchronization among streams </li></ul></ul></ul><ul><ul><ul><li>Time ordering of events using buffering service and </li></ul></ul></ul><ul><ul><ul><li>Time spaced release of events using time differential service on stream quality. </li></ul></ul></ul><ul><li>A metadata management service for archiving and replay </li></ul><ul><ul><li>How to build a session catalog to describe information regarding the streams in the session </li></ul></ul><ul><ul><li>How to manage messaging system topics for RTSP sessions </li></ul></ul><ul><ul><li>How to expose this service as a web service </li></ul></ul>
    • 16. Research Issues <ul><li>Improving fault tolerance of the system </li></ul><ul><ul><li>Redundancy in archiving/replay services </li></ul></ul><ul><ul><li>How to provide continuity of the stream in case of a replay service node crash </li></ul></ul><ul><ul><li>How the replay service can leverage fault tolerance </li></ul></ul><ul><li>Scalable replay service </li></ul><ul><ul><li>How many requests a replay service can support </li></ul></ul><ul><ul><li>Load balancing among replay services </li></ul></ul><ul><ul><li>Effect of network threshold </li></ul></ul><ul><ul><li>Supporting different type of clients with different capabilities </li></ul></ul><ul><li>Other research issues </li></ul><ul><ul><li>Systematic applications of major and minor event concepts in event driven systems </li></ul></ul><ul><ul><li>How to expose RTSP semantics as a web service </li></ul></ul><ul><ul><li>Synchronization of replaying multiple streams </li></ul></ul>
    • 17. Research Tasks <ul><li>RTSP semantics support in XGSP (service oriented architecture) </li></ul><ul><ul><li>How RTSP clients can join to XGSP sessions </li></ul></ul><ul><ul><ul><li>A RTSP to XGSP signaling gateway </li></ul></ul></ul><ul><ul><li>How XGSP will support RTSP clients </li></ul></ul><ul><li>RTSP semantics support in NB (messaging system) </li></ul><ul><ul><li>How to support active replay (play, pause, rewind, forward, absolute positioning, etc) for both live and archived streams </li></ul></ul><ul><ul><li>Instant replay </li></ul></ul><ul><ul><ul><li>How to support and provide seeking capability in live streams </li></ul></ul></ul><ul><ul><ul><li>Current RTSP servers do not support rewind in live streams </li></ul></ul></ul><ul><ul><li>Changes to NB archive and replay service to support RTSP semantics </li></ul></ul><ul><ul><li>Do we need extensions to RTSP? </li></ul></ul>
    • 18. Milestones I <ul><li>NB Time Service </li></ul><ul><ul><li>An implementation of Network Time Protocol (RFC 1305) </li></ul></ul><ul><ul><li>Entities generating events in the system should utilize Time Service to timestamp the events. </li></ul></ul><ul><li>NB Buffering Service </li></ul><ul><ul><li>The goal is to time-order events. </li></ul></ul><ul><ul><li>Delay introduced by the buffer service can vary based on the above parameter values. </li></ul></ul><ul><li>Time Differential Service </li></ul><ul><ul><li>Releases events preserving the time spacing between events. </li></ul></ul><ul><li>Streaming Gateway </li></ul><ul><ul><li>Transcodes audio/video streams into RealMedia format. </li></ul></ul><ul><ul><li>Targets both desktop PCs and cellular phones </li></ul></ul><ul><ul><li>Stream conversion is a CPU intensive application </li></ul></ul>
    • 19. Milestones II <ul><li>NB Replay Service </li></ul><ul><ul><li>Should provide API to support RTSP semantics. </li></ul></ul><ul><li>RTSP Media/Topic Manager </li></ul><ul><ul><li>Binding RTSP sessions with related NB topics. </li></ul></ul><ul><li>XGSP Archive Manager </li></ul><ul><ul><li>Provides RTSP RECORD semantics to start archiving of topics. </li></ul></ul><ul><li>Session Metadata Service </li></ul><ul><ul><li>Metadata service for archiving system. </li></ul></ul><ul><li>RTSP Server / Proxy </li></ul><ul><ul><li>Ability to dynamically locate replay and archiving services. </li></ul></ul><ul><ul><li>Ability to switch between replicas. </li></ul></ul><ul><li>We will apply those to e-sports project </li></ul>
    • 20. Typical Scenario for Live Streaming and Recording NB RTSP Client RTSP Server/Proxy Producer (XGSP Client (MBONE tools, ...) ,…) X 2 NB Stable Storage Replay/ Archiving Service NB Stable Storage Replay/Archiving Service Two way NB link One way NB link that carries stream Local Storage access Communication channel Topic X 1 4 3 5 1 : XGSP client sends and receives RTP packets 2, 3 : Archiving service subscribes to the topic and records the sessions on different storages. 4 : RTSP client communicates with RTSP server/proxy and establishes a RTSP session. 5 : RTSP client receives the stream from the topic.
    • 21. Typical Scenario for Live Streaming and Recording (with stream conversion) NB RTSP Client RTSP Server/Proxy Producer (XGSP Client (MBONE tools, ...) ,…) Streaming Gateway X 4 NB Stable Storage Replay/ Archiving Service NB Stable Storage Replay/Archiving Service Two way NB link One way NB link that carries stream Local storage access Communication channel Topic X 2 3 1 6 5 7 1 : XGSP client sends and receives RTP packets 2 : Streaming Gateway (SG) subscribes to the stream topic and receives the stream 3 : SG publishes the stream over NB link 4, 5 : Archiving service subscribes to the topic and records the sessions on different storages. 6 : RTSP client communicates with RTSP server/proxy and establishes a RTSP session. 7 :RTSP client receives the stream from the topic. 4
    • 22. Typical Scenario for Playback NB Replay/Archiving Service RTSP Client RTSP Server/Proxy X 1 2 3 One way NB link One way NB link that carries stream Communication channel Topic X 4 NB Stable Storage NB Stable Storage Replay/Archiving Service 1 : RTSP client communicates with RTSP server/proxy and establishes a RTSP session. 2 : Stream is published by replay service 3 : Alternate stream to 2 4 : RTSP client receives the stream from the topic.
    • 23. Typical Scenario for Instant Replay <ul><ul><li>NB </li></ul></ul>RTSP Client RTSP Server/Proxy Producer (XGSP Client (MBONE tools, ...) ,…) X NB Stable Storage Replay/ Archiving Service Two way NB link One way NB link that carries stream Local Storage access Communication channel Topic X 1 3 2 4 X 6 7 1 : XGSP client sends and receives RTP packets 2 : Archiving service subscribes to the topic and records the sessions. 3 : RTSP client communicates with RTSP server/proxy and establishes a RTSP session. 4 : RTSP client receives the stream from the topic. 5 : RTSP client communicates with RTSP server/proxy for instant replay. 6 : Replay service publishes the archived stream to a topic 7 : RTSP client receives the archived stream. ,5
    • 24. Tests <ul><li>NB Time Service tests on several machines. </li></ul><ul><li>Time differential service performance test. </li></ul><ul><li>Measuring number of clients that can be supported by a single replay service and storage. </li></ul><ul><li>Measuring client scalability </li></ul><ul><li>Measuring latency of recovery from failures </li></ul><ul><ul><li>How long will it take to dynamically switch between replay services during a node failure (node that provides the replay service)? </li></ul></ul><ul><ul><li>How long will it take for an archiving node to recover the missed events? </li></ul></ul>
    • 25. Contribution of this Thesis <ul><li>Combines the benefits of RTSP with distributed messaging system and a service oriented architecture for archiving and replay in a geographically distributed large network </li></ul><ul><li>A fault tolerant architecture for collaboration systems </li></ul><ul><li>Enables late, broken clients to receive missed streams </li></ul><ul><li>An architecture for instant replay of live streams </li></ul><ul><li>A scalable replay architecture benefits from the advantages of service oriented architecture and messaging systems </li></ul><ul><li>Support for heterogeneous clients </li></ul>
    • 26. Summary <ul><li>This thesis addresses the following open research issues in collaboration systems </li></ul><ul><ul><li>A framework for fault tolerance: </li></ul></ul><ul><ul><ul><li>Support for late or broken clients in live sessions. </li></ul></ul></ul><ul><ul><ul><li>Distributed archiving/replay system </li></ul></ul></ul><ul><ul><li>Support for different clients : Research extension of architectures to support different clients with different capabilities, i.e. cellular phone clients. </li></ul></ul><ul><ul><li>Client scalability: Research extension of architectures to support as many clients as possible. Centralized servers support a limited number of clients </li></ul></ul><ul><ul><li>An instant replay mechanism for live streams. </li></ul></ul>

    ×