Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. AUDIO AND VIDEO STREAMING WHAT IS STREAMING Streaming corresponds to playing audio and video files from an Internet server. This opposes to downloading the corresponding files. Pierre Nugues As transmission over the Internet is not synchronous, streaming uses a buffer to store a small part of data. This buffer dampens irregularities in Internet transmission. Streaming imposes constraints on the network speed: Download speed on the average should be at least as fast as playback speed. Streaming, May 2005 –1– Streaming, May 2005 –2– APPLICATIONS OF STREAMING PROBLEMS WITH TCP/UDP Applications are numerous. In addition to data: The original TCP/UDP protocols are based on packet transmission • Internet telephony and video conferences and have no quality of service. • Radios and Digital TV: ordinary broadcast but through Internet, Multimedia transmission has to tackle: no frontier, no distance • Delay. Must be less than 300 ms. (Perception threshold: 150 ms) • Audio and video server: on demand movies and concerts. • Jitter. Packets may use different transmission paths that results into • Games and virtual reality time expansion and compression • Interaction • Loss. Routers may drop packets when the network load is too high Triple play: data (IP), audio (VoIP), video (TVoIP). Audio and video transmission needs buffering and synchronization, It is made possible because of the growing availability of ADSL and possibly error correction, for instance by repeating data fast Internet UDP is just a layer to address ports. It is compatible with the Companies in Sweden: NetInsight, etc. requirements TCP design makes it less adapted to real-time, fast communications Streaming, May 2005 –3– Streaming, May 2005 –4–
  2. 2. PACKET TRANSMISSION REAL TIME TRANSPORT PROTOCOL The Real Time Transport Protocol (RTP) • Identifies the content • Adds time stamps • Adds sequence numbers RTP does not guarantee a real-time delivery. RTP needs an application layer to • Re-order packets • Attenuate jitter • Compensate packet loss Streaming, May 2005 –5– Streaming, May 2005 –6– ARCHITECTURE REAL TIME TRANSPORT PROTOCOL (RTP) RTP is on top of UDP: Ethernet IP UDP RTP Media content: MPEG, AIFF, and son on The simplified RTP header structure is: Payload Sequence Timestamp Sync. Source ID Other type Number (SSRC) fields RTP RFCs are available here: and Other reference: Streaming, May 2005 –7– Streaming, May 2005 –8–
  3. 3. RTP HEADER TIMESTAMP ACCORDING TO THE RFC • Payload type (7 bits) corresponds to the packet content: PCM = 0, “The timestamp reflects the sampling instant of the first octet in the RTP data packet. DVI4=5, MPEG-2 = 33 ( The sampling instant must be derived from a clock that increments monotonically and linearly in time to allow synchronization and jitter calculations (see Section 6.3.1). The parameters) resolution of the clock must be sufficient for the desired synchronization accuracy and • Sequence number (16 bits) is incremented each time a packet is sent for measuring packet arrival jitter (one tick per video frame is typically not sufficient). (Nothing guarantees the arrival order with UDP) The clock frequency is dependent on the format of data carried as payload and is specified statically in the profile or payload format specification that defines the format, • The timestamp (32 bits) corresponds to the sampling instant of the or may be specified dynamically for payload formats defined through non-RTP means. first octet in the RTP data packet. (Clock of the sending machine) If RTP packets are generated periodically, the nominal sampling instant as determined from the sampling clock is to be used, not a reading of the system clock. As an • SSRC (32 bits) is the source of the stream. (A sending machine can example, for fixed-rate audio the timestamp clock would likely increment by one have multiple sessions.) for each sampling period. If an audio application reads blocks covering 160 sampling periods from the input device, the timestamp would be increased by 160 for each such block, regardless of whether the block is transmitted in a packet or dropped as silent.” 8 kHz! 125 !s, 160 samples ! 20 ms Streaming, May 2005 –9– Streaming, May 2005 – 10 – REAL-TIME CONTROL PROTOCOL (RTCP) ENCODING FORMATS Part of the RTP protocol and defined in the same RFC. Telephone: 8 kHz, 1 octet (256 values), 64 kbit/s It sends periodically control packets to all participants in the session CD: 44.1 kHz, 16 bits, stereo, 1.4 mbits/s and uses a different port MP3, compressed, 96, 128, 160 kbit/s It provides feedback on the quality of the data distribution from the G.732, (Internet telephony in H.323) 5.3 kbit/s or 6.4 kbit/s sender and the receiver: Statistics on packets sent, received, lost, jitter MPEG-2 used in DVD, 3-6 mbit/s Should limit to 5% of the bandwidth Two important concepts in encoding methods: Commands Description • Bit rate can be constant (CBR) or variable (VBR) SR Sender report • From an original format, encoding compress data with or without RR Receiver report loss. Lossy compression generally results in better rates but lower SDES Source description quality: sometimes not perceptible. Loss rate can be a parameter of BYE Quit the method. Streaming, May 2005 – 11 – Streaming, May 2005 – 12 –
  4. 4. PULSE CODE MODULATION JPEG Compression standard for still pictures: Sampling Digitization • Maps RGB images onto YUV coordinates (luminance and chrominance) • Applies a Discrete Cosine Transform (DCT) • Quantizes, which results in a compression with an adjustable loss 100 0 80 120 90 0 40 40 10 30 90 140 170 180 170 140 100 60 0 70 • Run-length encoding Digitization can be linear or logarithmic: "-law, A-law Streaming, May 2005 – 13 – Streaming, May 2005 – 14 – MPEG CODECS Initially sequences of frames using JPEG (I Frames) Codec encode and decode original data streams. 25 or 30 frames/s Depending on the media you are sending, you must have the Uses temporal redundancies between images: differences between corresponding codec. frames (P and B Frames) Formats supported by the RTP implementation of the Java Media MPEG 2 has possible resolutions: 720 ! 480, 720 ! 576, 1920 ! 1080. Framework: Multimedia streams contain audio and video data that are media/jmf/2.1.1/formats.html#RTPFormats synchronized in MPEG Codecs can be found from many sources as: ( Streaming, May 2005 – 15 – Streaming, May 2005 – 16 –
  5. 5. JAVA MEDIA FRAMEWORK JMF COMPONENTS According to Sun, the Java Media Framework (JMF) is an “API for The main components are: incorporating time-based media into Java applications and applets”. • The Clock interface keeps track of time in a media stream It also features RTP capabilities • Managers: Manager, PackageManager, etc. construct Players, JMF makes it possible to build media applications assembling modules. Processors, DataSources, etc Just follow the examples • JMF programs report their current state using MediaEvents. Available as an optional package from Sun Many subclasses include ControllerEvent and RTPEvent • DataSources manage media transfer. It contains the location of Native implementations for Windows, Linux, and Solaris the media (URL), protocol, and software. Java implementation for the others, including MacOS Source available under Sun’s license (SCSL) Apparently, no active development from Sun. Latest release: 2003, an MP3 update Streaming, May 2005 – 17 – Streaming, May 2005 – 18 – JMF PLAYERS PLAYER STATES A player reads a DataSource and renders it (From Sun’s documentation) (From Sun’s documentation) Streaming, May 2005 – 19 – Streaming, May 2005 – 20 –
  6. 6. A FIRST EXAMPLE: A PLAYER return; if ((visual = mplayer.getVisualComponent()) != null) { available from Dimension size = visual.getPreferredSize(); videoWidth = size.width; media/jmf/2.1.1/solutions/index.html videoHeight = size.height; if ((url = new URL(mediaFile)) == null) { getContentPane().add("Center", visual); … } else player = Manager.createPlayer(url); videoWidth = 320; … if ((control = mplayer.getControlPanelComponent()) != null) mplayer = player; { mplayer.addControllerListener((ControllerListener) this); controlHeight = control.getPreferredSize().height; mplayer.realize(); getContentPane().add("South", control); … } public void controllerUpdate(ControllerEvent ce) { setSize(videoWidth + insetWidth, if (ce instanceof RealizeCompleteEvent) { videoHeight + controlHeight + insetHeight); mplayer.prefetch(); validate(); } else if (ce instanceof PrefetchCompleteEvent) { mplayer.start(); if (visual != null) Streaming, May 2005 – 21 – Streaming, May 2005 – 22 – } else if (ce instanceof EndOfMediaEvent) { PROCESSORS mplayer.setMediaTime(new Time(0)); mplayer.start(); A processor is a specialized player that can output data to a } DataSource and carry out some processing } Streaming, May 2005 – 23 – Streaming, May 2005 – 24 –
  7. 7. PROCESSOR STATES SENDING AUDIO available from media/jmf/2.1.1/solutions/index.html private String createProcessor() { if (locator == null) return "Locator is null"; DataSource ds; DataSource clone; try { ds = Manager.createDataSource(locator); } catch (Exception e) { (From Sun’s documentation) return "Couldn't create DataSource"; Configure connects to the DataSource and demultiplexes the stream } // Try to create a processor to handle the input media locator try { Streaming, May 2005 – 25 – Streaming, May 2005 – 26 – processor = Manager.createProcessor(ds); if (tracks == null || tracks.length < 1) } catch (NoProcessorException npe) { return "Couldn't find tracks in processor"; return "Couldn't create processor"; boolean programmed = false; } catch (IOException ioe) { AudioFormat afmt; return "IOException creating processor"; // Search through the tracks for a Audio track } for (int i = 0; i < tracks.length; i++) { Format format = tracks[i].getFormat(); // Wait for it to configure if ( tracks[i].isEnabled() && boolean result = waitForState(processor, format instanceof AudioFormat && Processor.Configured); !programmed) { if (result == false) afmt = (AudioFormat)tracks[i].getFormat(); return "Couldn't configure processor"; AudioFormat ulawFormat = new AudioFormat(AudioFormat.DVI_RTP); // Get the tracks from the processor // afmt.getSampleRate(), TrackControl [] tracks = processor.getTrackControls(); // afmt.getSampleSizeInBits(), // afmt.getChannels()); // Do we have at least one track? // 8000,4,1); Streaming, May 2005 – 27 – Streaming, May 2005 – 28 –
  8. 8. tracks[i].setFormat(ulawFormat); // Realize the processor. This will internally create a flow System.err.println("Audio transmitted as:"); // graph and attempt to create an output datasource for System.err.println(" " + ulawFormat); ULAW/RTP // Assume succesful // Audio frames. programmed = true; result = waitForState(processor, Controller.Realized); } else if (result == false) tracks[i].setEnabled(false); return "Couldn't realize processor"; } if (!programmed) // Get the output data source of the processor return "Couldn't find Audio track"; dataOutput = processor.getDataOutput(); return null; // Set the output content descriptor to RAW_RTP } ContentDescriptor cd = new ContentDescriptor(ContentDescriptor.RAW_RTP); processor.setContentDescriptor(cd); Streaming, May 2005 – 29 – Streaming, May 2005 – 30 – SENDING AUDIO (II) while (p.getState() < state && !failed) { synchronized (getStateLock()) { private synchronized boolean waitForState(Processor p, int state) try { { getStateLock().wait(); p.addControllerListener(new StateListener()); } catch (InterruptedException ie) { failed = false; return false; } // Call the required method on the processor } if (state == Processor.Configured) { } p.configure(); if (failed) } else if (state == Processor.Realized) { return false; p.realize(); else } return true; } // Wait until we get an event that confirms the // success of the method, or a failure event. // See StateListener inner class Streaming, May 2005 – 31 – Streaming, May 2005 – 32 –
  9. 9. SENDING AUDIO (III) // for the processor to start sending data. So we need to start the // Creates an RTP transmit data sink. This is the easiest way to // output data source of the processor. We also need to start create the // an RTP transmitter. The other way is to use the // processor itself, which is done after this method returns. RTPSessionManager API. try { // Using an RTP session manager gives you more control if you wish rtptransmitter = Manager.createDataSink(dataOutput, to outputLocator); // fine tune your transmission and set other parameters.; private String createTransmitter() { rtptransmitter.start(); // Create a media locator for the RTP data sink. dataOutput.start(); // For example: } catch (MediaException me) { // rtp:// return "Couldn't create RTP data sink"; String rtpURL = "rtp://" + ipAddress + ":" + port + "/audio"; } catch (IOException ioe) { MediaLocator outputLocator = new MediaLocator(rtpURL); return "Couldn't create RTP data sink"; } // Create a data sink, open it and start transmission. It will return null; wait } Streaming, May 2005 – 33 – Streaming, May 2005 – 34 – SENDING VIDEO (int)(size.width / 8) * 8); int h = (size.height % 8 == 0 ? size.height : available from (int)(size.height / 8) * 8); VideoFormat jpegFormat = new VideoFormat(VideoFormat.JPEG_RTP, media/jmf/2.1.1/solutions/index.html new Dimension(w, h), for (int i = 0; i < tracks.length; i++) { Format.NOT_SPECIFIED, Format format = tracks[i].getFormat(); Format.byteArray, if ( tracks[i].isEnabled() && frameRate); format instanceof VideoFormat && tracks[i].setFormat(jpegFormat); !programmed) { System.err.println("Video transmitted as:"); System.err.println(" " + jpegFormat); // Found a video track. Try to program it to output // Assume succesful JPEG/RTP programmed = true; // Make sure the sizes are multiple of 8's. } else Dimension size = ((VideoFormat)format).getSize(); tracks[i].setEnabled(false); float frameRate = ((VideoFormat)format).getFrameRate(); } int w = (size.width % 8 == 0 ? size.width : Streaming, May 2005 – 35 – Streaming, May 2005 – 36 –
  10. 10. RECEIVING RESOURCES FOR JMF available from Documentation: media/jmf/2.1.1/solutions/index.html media/jmf/reference/docs/index.html It opens RTP sessions for each track Program examples available from Sun: It receives events from the RTP streams It creates one player per stream media/jmf/2.1.1/solutions/index.html Active discussion list: Codecs available from SourceForge: Streaming, May 2005 – 37 – Streaming, May 2005 – 38 – REAL-TIME STREAMING PROTOCOL (RTSP) SESSION INITIATION PROTOCOL (SIP) There is no procedure to connect to a remote host in UDP or RTP. RTSP is a protocol to control streaming media. It acts as a sort of This is handled by SIP. Defined by IETF in RFC: remote control. Commands Description SIP enables to set up a call, negotiate the parameters, manage and SETUP Causes the server to allocate resources for a stream and start an close the session. RTSP session Borrows many ideas from HTTP and uses UDP or TCP. PLAY Tells the server to start sending data Once the session is established, the media transmission can use RTP RECORD Records data or something else. PAUSE Temporarily halts a stream without freeing server resources TEARDOWN Frees resources associated with the stream. The RTSP session ceases to exist on the server RTSP servers typically use RTP for the audio/video transmission. Streaming, May 2005 – 39 – Streaming, May 2005 – 40 –
  11. 11. A SIP EXAMPLE (SIMPLIFIED FROM THE RFC) A CONTENT EXAMPLE Alice sends an INVITE request addressed to Bob’s SIP URI. The session description protocol, SDP, specifies details of the INVITE"> SIP/2.0 connection using name-value pairs. Via: SIP/2.0/UDP;branch=z9hG4bK776asdhds ( Max-Forwards: 70 v(ersion)=0 To: Bob <>">>; o(wner)=bell 53655765 2353687637 IN IP4 From: Alice c(onnection)=IN IP4 <>;tag=1928301774">>;tag=1928301774 Call-ID: m(edia)=audio 3456 RTP/AVP 0 3 4 5 CSeq: 314159 INVITE Contact: <>">>; Content-Type: application/sdp Content-Length: 142 Streaming, May 2005 – 41 – Streaming, May 2005 – 42 – A SIP EXAMPLE (SIMPLIFIED FROM THE RFC) SIP METHODS Alice Proxy Proxy Bob Methods Descriptions INVITE F1 -----> INVITE Invites a session INVITE F2 -----> ACK Acknowledges <--- 100 TRYING F3 INVITE F4 -----> OPTIONS Server capabilities <--- 100 TRYING F5 <--180 RINGING F6 BYE Closes a session <--180 RINGING F7 <-----200 OK F9 CANCEL Cancels a pending request <--180 RINGING F8 <-----200 OK F10 REGISTER <-----200 OK F11 ACK F12 ----------> Media <---------- BYE F13 OK F14 200 ----------> Streaming, May 2005 – 43 – Streaming, May 2005 – 44 –
  12. 12. SIP REGISTRAR SIP REGISTRAR (II) When the SIP client starts, it registers its location. It enables proxies to find people in different places using multiple C->S: REGISTER SIP/2.0 devices. Via: SIP/2.0/UDP REGISTER SIP/2.0 From: Via: SIP/2.0/UDP To: From: Call-ID: To: CSeq: 1 REGISTER Call-ID: Contact: CSeq: 1 REGISTER Contact: <;transport=udp> Expires: 7200 Streaming, May 2005 – 45 – Streaming, May 2005 – 46 – H.323 H.323 is a competitor to SIP. It has been promoted by the ITU – the telephone companies Complete and in the beginning more complex then SIP Good integration with telephone systems Streaming, May 2005 – 47 –