This document summarizes recent advances in channel-adaptive video streaming. It reviews adaptive media playout at the client to reduce latency, rate-distortion optimized packet scheduling to determine the best packet to send, and channel-adaptive packet dependency control to improve error robustness and reduce latency. It also discusses challenges for wireless video streaming and different wireless streaming architectures.
Requiring only half the bitrate of its predecessor, the new standard – HEVC or H.265 – will significantly reduce the need for bandwidth and expensive, limited spectrum. HEVC (H.265) will enable the launch of new video services and in particular ultra HD television (UHDTV).
State-of-the-art video compression techniques – HEVC/H.265 – can reduce the size of raw video by a factor of about 100 without any noticeable reduction in visual quality. With estimates indicating that compressed real-time video accounts for more than 50 percent of current network traffic, and this figure is set to rise to 90 percent within a few years, HEVC/H.265 will be a welcome relief for network operators.
New services, devices and changing viewing patterns are among the factors contributing to the growth in video traffic as people watch more and more traditional TV and video-streaming services on their mobile devices.
Ericsson has been heavily involved in the standardization of HEVC since it began in 2010, and this Ericsson Review article highlights some of the contributions that have led to the compression efficiency offered by HEVC.
.
Motion Vector Recovery for Real-time H.264 Video StreamsIDES Editor
Among the various network protocols that can be
used to stream the video data, RTP over UDP is the best to do
with real time streaming in H.264 based video streams. Videos
transmitted over a communication channel are highly prone
to errors; it can become critical when UDP is used. In such
cases real time error concealment becomes an important
aspect. A subclass of the error concealment is the motion
vector recovery which is used to conceal errors at the decoder
side. Lagrange Interpolation is the fastest and a popular
technique for the motion vector recovery. This paper proposes
a new system architecture which enables the RTP-UDP based
real time video streaming as well as the Lagrange
interpolation based real time motion vector recovery in H.264
coded video streams. A completely open source H.264 video
codec called FFmpeg is chosen to implement the proposed
system. Proposed implementation was tested against the
different standard benchmark video sequences and the
quality of the recovered videos was measured at the decoder
side using various quality measurement metrics.
Experimental results show that the real time motion vector
recovery does not introduce any noticeable difference or
latency during display of the recovered video.
Webcast: Reduce latency, improve analytics and maximize asset utilization in ...Emulex Corporation
Join Emulex and Myricom experts to learn how to maximize performance in HFT, network security, network analytics and video content delivery environments with Emulex Network Xceleration (NX) solutions.
This webcast will discuss ways of reducing latency, increasing asset utilization and improving network analytics in high performance networks.
Subjective quality evaluation of the upcoming HEVC video compression standard Touradj Ebrahimi
Slides of my presentation at SPIE Optics+Photonics 2012 Applications of Digital Image Processing XXXV, San Diego, August 12-16, 2012
Paper available at: http://infoscience.epfl.ch/record/180494
Value Networks and Business Models of Information-centric NetworkingTapio Levä
Presentation given in the "Future Media Distribution using Information Centric Networks" event organized by the SAIL and EFRAIM projects on February 13, 2013 in Stockholm, Sweden.
The presentation is based on the results of the SAIL (EC-FP7) project.
Analyzing Video Streaming Quality by Using Various Error Correction Methods o...IJERA Editor
Transmission video over ad hoc networks has become one of the most important and interesting subjects of study for researchers and programmers because of the strong relationship between video applications and frequent users of various mobile devices, such as laptops, PDAs, and mobile phones in all aspects of life. However, many challenges, such as packet loss, congestion (i.e., impairments at the network layer), multipath fading (i.e., impairments at the physical layer) [1], and link failure, exist in transferring video over ad hoc networks; these challenges negatively affect the quality of the perceived video [2].This study has investigated video transfer over ad hoc networks. The main challenges of transferring video over ad hoc networks as well as types of errors that may occur during video transmission, various types of video mechanisms, error correction methods, and different Quality of Service (QoS) parameters that affect the quality of the received video are also investigated.
Multicasting Of Adaptively-Encoded MPEG4 Over Qos-Cognizant IP NetworksEditor IJMTER
we propose a novel architectural planning for multicasting of adaptively-encoded
layered MPEG4 over a QoS-aware IP network. We re-quire a QoS-aware IP network in this case to
(1) Support priority dropping of packets in time of congestion. (2) Provide congestion notification to
the multicast sender. For the first requirement, we use RED's extension for service differentiation. It
recognizes the priority of packets when they need to be dropped and drops lower priority packets
first. We couple RED with our proposal for the second requirement which is the adoption of
Backward Explicit Congestion Notification (BECN) for use with IP multicast. BECN will provide
early congestion notification at the IP layer level to the video sender. BECN detects upcoming
congestion based on size of the RED queue in the routers. The MPEG4 adaptive-encoder can change
the sending rate and also can divide the video packets into lower priority packets and high priority
packets. Based on BECN messages from the routers, a simple flow controller at the sender sets the
rate for the adaptive MPEG4 encoder and also sets the ratio between the high priority and low
priority packets within the video stream. We use a TES model for generating the MPEG4 traffic that
is based on real video traces. Simulation results show that combining priority dropping, MPEG4
adaptive encoding, and multicast BECN: (1) Improves bandwidth utilization (2) Reduces time to
react to congestion and hence improves the received video quality (3) Maintains graceful degradation
in quality with congestion and provides minimum quality even if congestion persists.
IJERA (International journal of Engineering Research and Applications) is International online, ... peer reviewed journal. For more detail or submit your article, please visit www.ijera.com
Requiring only half the bitrate of its predecessor, the new standard – HEVC or H.265 – will significantly reduce the need for bandwidth and expensive, limited spectrum. HEVC (H.265) will enable the launch of new video services and in particular ultra HD television (UHDTV).
State-of-the-art video compression techniques – HEVC/H.265 – can reduce the size of raw video by a factor of about 100 without any noticeable reduction in visual quality. With estimates indicating that compressed real-time video accounts for more than 50 percent of current network traffic, and this figure is set to rise to 90 percent within a few years, HEVC/H.265 will be a welcome relief for network operators.
New services, devices and changing viewing patterns are among the factors contributing to the growth in video traffic as people watch more and more traditional TV and video-streaming services on their mobile devices.
Ericsson has been heavily involved in the standardization of HEVC since it began in 2010, and this Ericsson Review article highlights some of the contributions that have led to the compression efficiency offered by HEVC.
.
Motion Vector Recovery for Real-time H.264 Video StreamsIDES Editor
Among the various network protocols that can be
used to stream the video data, RTP over UDP is the best to do
with real time streaming in H.264 based video streams. Videos
transmitted over a communication channel are highly prone
to errors; it can become critical when UDP is used. In such
cases real time error concealment becomes an important
aspect. A subclass of the error concealment is the motion
vector recovery which is used to conceal errors at the decoder
side. Lagrange Interpolation is the fastest and a popular
technique for the motion vector recovery. This paper proposes
a new system architecture which enables the RTP-UDP based
real time video streaming as well as the Lagrange
interpolation based real time motion vector recovery in H.264
coded video streams. A completely open source H.264 video
codec called FFmpeg is chosen to implement the proposed
system. Proposed implementation was tested against the
different standard benchmark video sequences and the
quality of the recovered videos was measured at the decoder
side using various quality measurement metrics.
Experimental results show that the real time motion vector
recovery does not introduce any noticeable difference or
latency during display of the recovered video.
Webcast: Reduce latency, improve analytics and maximize asset utilization in ...Emulex Corporation
Join Emulex and Myricom experts to learn how to maximize performance in HFT, network security, network analytics and video content delivery environments with Emulex Network Xceleration (NX) solutions.
This webcast will discuss ways of reducing latency, increasing asset utilization and improving network analytics in high performance networks.
Subjective quality evaluation of the upcoming HEVC video compression standard Touradj Ebrahimi
Slides of my presentation at SPIE Optics+Photonics 2012 Applications of Digital Image Processing XXXV, San Diego, August 12-16, 2012
Paper available at: http://infoscience.epfl.ch/record/180494
Value Networks and Business Models of Information-centric NetworkingTapio Levä
Presentation given in the "Future Media Distribution using Information Centric Networks" event organized by the SAIL and EFRAIM projects on February 13, 2013 in Stockholm, Sweden.
The presentation is based on the results of the SAIL (EC-FP7) project.
Analyzing Video Streaming Quality by Using Various Error Correction Methods o...IJERA Editor
Transmission video over ad hoc networks has become one of the most important and interesting subjects of study for researchers and programmers because of the strong relationship between video applications and frequent users of various mobile devices, such as laptops, PDAs, and mobile phones in all aspects of life. However, many challenges, such as packet loss, congestion (i.e., impairments at the network layer), multipath fading (i.e., impairments at the physical layer) [1], and link failure, exist in transferring video over ad hoc networks; these challenges negatively affect the quality of the perceived video [2].This study has investigated video transfer over ad hoc networks. The main challenges of transferring video over ad hoc networks as well as types of errors that may occur during video transmission, various types of video mechanisms, error correction methods, and different Quality of Service (QoS) parameters that affect the quality of the received video are also investigated.
Multicasting Of Adaptively-Encoded MPEG4 Over Qos-Cognizant IP NetworksEditor IJMTER
we propose a novel architectural planning for multicasting of adaptively-encoded
layered MPEG4 over a QoS-aware IP network. We re-quire a QoS-aware IP network in this case to
(1) Support priority dropping of packets in time of congestion. (2) Provide congestion notification to
the multicast sender. For the first requirement, we use RED's extension for service differentiation. It
recognizes the priority of packets when they need to be dropped and drops lower priority packets
first. We couple RED with our proposal for the second requirement which is the adoption of
Backward Explicit Congestion Notification (BECN) for use with IP multicast. BECN will provide
early congestion notification at the IP layer level to the video sender. BECN detects upcoming
congestion based on size of the RED queue in the routers. The MPEG4 adaptive-encoder can change
the sending rate and also can divide the video packets into lower priority packets and high priority
packets. Based on BECN messages from the routers, a simple flow controller at the sender sets the
rate for the adaptive MPEG4 encoder and also sets the ratio between the high priority and low
priority packets within the video stream. We use a TES model for generating the MPEG4 traffic that
is based on real video traces. Simulation results show that combining priority dropping, MPEG4
adaptive encoding, and multicast BECN: (1) Improves bandwidth utilization (2) Reduces time to
react to congestion and hence improves the received video quality (3) Maintains graceful degradation
in quality with congestion and provides minimum quality even if congestion persists.
IJERA (International journal of Engineering Research and Applications) is International online, ... peer reviewed journal. For more detail or submit your article, please visit www.ijera.com
RESOURCE ALLOCATION ALGORITHMS FOR QOS OPTIMIZATION IN MOBILE WIMAX NETWORKSijwmn
WiMAX is based on the standard IEEE 802.16e-2009 for wireless access in Metropolitan Area Networks. It
is one of the solutions for 4G IP based wireless technology. WiMAX utilizes Orthogonal Frequency
Division Multiple Access which also supports Multicast and Broadcast Service with appropriate
Modulation and Coding Scheme. Presently, Scheduling and Resource allocation algorithm in Opportunistic
Layered Multicasting provides multicasting of layered video over mobile WiMAX to ensure better QoS.
Initially, the knowledge based allocation of subcarriers is used for scheduling. In addition, to reduce the
burst overhead, delay and jitter, SWIM (Swapping Min-Max) algorithm is utilized. Another promising
technology that can greatly improve the system performance by exploring the broadcasting nature of
wireless channels and the cooperation among multiple users is the Cooperative Multicast Scheduling
(CMS) technique. The simulation results show, Swapping Min-Max performs better with lesser number of
bursts, Zero jitter and with optimal throughput. The results with Cooperative Multicast Scheduling show
the enhanced throughput for each member in the Multicasting Scenario.
Video transmission over wireless networks is considered the most interesting application in our daily life nowadays. As
mobile data rates continue to increase and more people rely on wireless transmission, the amount of video transmitted over at least one
wireless hop will likely continue to increase. This kind of application needs large bandwidth, efficient routing protocols, and content
delivery methods to provide smooth video playback to the receivers. Current generation wireless networks are likely to operate on
internet technology combined with various access technologies. Achieving effective bandwidth aggregation in wireless environments
raises several challenges related to deployment, link heterogeneity, Network congestion, network fluctuation, and energy consumption.
In this work, an overview of technical challenges of over wireless networks is presented. A survey of wireless networks in recent video
transmission schemes is introduced. Demonstration results of few scenarios are showed.
Video streaming over Ad hoc on-demand distance vector routing protocoljournalBEEI
Video streaming is content sent in compressed form over the netwoks and viwed the users progressively. The transmission of video with the end goal that it can be prepared as consistent and nonstop stream. The point is that to give client support to client at anyplace and at whatever time. Mobile Ad hoc Networks (MANETs) are considered an attractive nertwork for information transmission in many applications where the customer programme can begin showing the information before the whole record has been transmitted.
Ad hoc On-demand Distance Vector (AODV) protocol is considered as one of the most important routing protocols in MANET. However, routing protocols assume a crucial part in transmission of information over the network. This paper investigates the performance of AODV Routing Protocol under video traffic over PHY IEEE 802.11g. The protocol model was developed in OPNET. Different outcomes from simulation based models are analyzed and appropriate reasons are also discussed. A different scenarios of video streaming were used. The metric in terms of throughput, end to end delay, packet delivery ratio and routing overhead were measured.
A comparision with GRP and GRP are also reported.
An SDN Based Approach To Measuring And Optimizing ABR Video Quality Of Experi...Cisco Service Provider
Reprinted with permission of NCTA, from the 2014 Cable Connection Spring Technical Forum Conference Proceedings. For more information on Cisco video solutions, visit: http://www.cisco.com/c/en/us/products/video/index.html
Advancements in Complementary Metal Oxide Semiconductor (CMOS) technology have enabled Wireless Sensor Networks (WSN) to gather, process and transport multimedia (MM) data as well and not just limited to handling ordinary scalar data anymore. This new generation of WSN type is called Wireless Multimedia Sensor Networks (WMSNs). Better and yet relatively cheaper sensors – sensors that are able to sense both scalar data and multimedia data with more advanced functionalities such as being able to handle rather intense computations easily - have sprung up. In this paper, the applications, architectures, challenges and issues faced in the design of WMSNs are explored. Security and privacy issues, over all requirements, proposed and implemented solutions so far, some of the successful achievements and other related works in the field are also highlighted. Open research areas are pointed out and a few solution suggestions to the still persistent problems are made, which, to the best of my knowledge, so far haven’t been explored yet.
The main problem is to avoid the complexity of retrieving the video content without streaming problem in multi network clients. The proposed work is to improve Collaboration among streaming contents on server resources in order to improve the network performance. Implementing network collaboration on a content delivery scenario, with a strong reduction of data transferred via servers. Audio and video files are transmitted in blocks to clients through the peer using the Network Coding Equivalent Content Distribution scheme. The objective of the system is to tolerate out-of-order arrival of blocks in the stream and is resilient to transmission losses of an arbitrary number of intermediate blocks, without affecting the verifiability of remaining blocks in the stream. To formulate the joint rate control and packet scheduling problem as an integer program where the objective is to minimize a cost function of the expected video distortion. Suggestions of cost functions are proposed in order to provide service differentiation and address fairness among users.
Multimedia Video transmission is over Wireless Local Area Networks is expected to be an important component of many
emerging multimedia applications. However, Wireless networks will always be bandwidth limited compared to fixed networks due to
background noise, limited frequency spectrum, and varying degrees of network coverage and signal strength One of the critical issues
for multimedia applications is to ensure that the Quality of Service (QoS) requirement to be maintained at an acceptable level. Modern
mobile devices are equipped with multiple network interfaces, including 3G/LTE WiFi. Bandwidth aggregation over LTE and WiFi
links offers an attractive opportunity of supporting bandwidth-intensive services, such as high-quality video streaming, on mobile
devices. Achieving effective bandwidth aggregation in wireless environments raises several challenges related to deployment, link
heterogeneity, Network congestion, network fluctuation, and energy consumption. In this work, an overview of schemes for video
transmission over wireless networks is presented where an acceptable quality of service (QoS) for video applications required realtime
video transmission is achieved
Stabilization of Variable Bit Rate Video Streams Using Linear Lyapunov Functi...ijait
Streaming videos over wireless networks suffers from low video quality due to network ability limitations. The quality of the channel and the characteristics of source play the major role in transmitting video stream over mobile environments. On failure of wireless video transmission, retransmission method was employed to improve the reliability of wireless link. However, retransmission of video leads to significant impact on energy consumption and bounded average waiting time is also increased. The longer waiting time on retransmission results in buffer starvation. Therefore, it is desirable to reduce the variable bit rate of transmitted video signal and increase the stability level when buffer starvation is occurs. In order to overcome such limitation, a technique named Response based Stabilization Analysis (RSA) using Distributed Optimality Bit Rate Allocation (RSA-DOBRA) is proposed in this paper. Initially, video stream is segregated into frames of different classes (i.e., size). Each frame is transmitted based on the variable bit rate response using Optimal Quantization process. Secondly, Linear Lyapunov Functions is employed with RSA to prove the stability of different bit rates on wireless video streaming. The application of Linear Lyapunov Function maintains the stability level of bit rate on different class of frame transmission on wireless link. Finally, Distributed Optimality Bit Rate Allocation uses the time slicing procedure to reduce the bounded average waiting time. RSA performs the time slicing based on multiplexed wireless video transmission on variable bit rate to avoid buffer starvation. RSA at the final stage reduces the energy consumption by improving the reliability of wireless link. Experiment is conducted on factors such as buffered starvation rate, waiting time on video frame transmission, and energy consumption rate.
Wimax Emulator to Enhance Media and Video Qualityijceronline
International Journal of Computational Engineering Research (IJCER) is dedicated to protecting personal information and will make every reasonable effort to handle collected information appropriately. All information collected, as well as related requests, will be handled as carefully and efficiently as possible in accordance with IJCER standards for integrity and objectivity.
Qo s management for mobile satellite communicationeSAT Journals
Abstract In this paper, a cross-layer architecture (QoSatAr) is developed to provide end-to-end quality of service (QoS) guarantees for Internet protocol (IP) traffic over the Digital Video Broadcasting-Second generation (DVB-S2) satellite systems. The architecture design is based on a cross-layer optimization between the physical layer and the network layer to provide QoS provisioning based on the bandwidth availability present in the DVB-S2 satellite channel. One of the most important aspects of the architecture design is that QoSatAr is able to guarantee the QoS requirements for specific traffic flows considering a single parameter: the bandwidth availability which is set at the physical layer (considering adaptive code and modulation adaptation) and sent to the network layer by means of a cross-layer optimization. The architecture has been evaluated using the NS-2 simulator. Keywords: QoSatAr, DVB-S2, ACM, RQM, DiffServ
IJRET : International Journal of Research in Engineering and Technology is an international peer reviewed, online journal published by eSAT Publishing House for the enhancement of research in various disciplines of Engineering and Technology. The aim and scope of the journal is to provide an academic medium and an important reference for the advancement and dissemination of research results that support high-level learning, teaching and research in the fields of Engineering and Technology. We bring together Scientists, Academician, Field Engineers, Scholars and Students of related fields of Engineering and Technology
Similar to ADVANCES IN CHANNEL-ADAPTIVE VIDEO STREAMING (20)
1. Proc. IEEE Intern. Conf. Image Processing
ICIP 2002, Rochester, NY, Sept. 2002.
ADVANCES IN CHANNEL-ADAPTIVE VIDEO STREAMING
Bernd Girod, Mark Kalman, Yi J. Liang, Rui Zhang
Information Systems Laboratory
Department of Electrical Engineering, Stanford University
Stanford, CA 94305
Invited Paper
Abstract and concealment techniques to mitigate the effects of lost packets
[3]. Unless forced by firewalls, streaming media systems do not
We review recent advances in channel-adaptive video streaming. rely on TCP for media transport but implement their own appli-
Adaptive media playout at the client can be used to reduce re- cation level transport mechanisms to provide the best end-to-end
ceiver buffering and therefore average latency, and provide a lim- delivery while adapting to the changing network conditions. Com-
ited rate scalability. Rate-distortion optimized packet scheduling mon issues include retransmission and buffering of packets [4],
determines the best packet to send given the distortion reduction generating parity check packets [5], TCP-friendly rate control [6],
associated with sending that packet, interpacket dependencies, and and receiver-driven adaptation for multicasting [7]. New network
the success of past transmissions. Channel-adaptive packet depen- architectures, such as DiffServ [8] and the path diversity trans-
dency control can greatly improve the error-robustness of stream- mission in packet networks [9], also fall into this category. The
ing video and reduce or eliminate the need for packet retransmis- media server can help implementing intelligent transport mecha-
sions. Finally we consider three architectures for wireless video nisms, by sending out the right packets at the right time, but the
streaming and discuss the utility of the discussed techniques for amount of computation that it can perform for each media stream
each architecture. is very limited due to the large number of streams to be served
simultaneously. Most of the burden for efficient and robust trans-
mission is therefore on the encoder application that, however, faces
1. INTRODUCTION
the added complication that it cannot adapt to the varying chan-
nel conditions but rather has to rely on the media server for this
Since the introduction of the first commercial products in 1995, In-
task. Representations that allow easy rate scalability are very im-
ternet video streaming has experienced phenomenal growth. Over
portant to adapt to varying network throughput without requiring
a million hours of streaming media contents are being produced
computation at the media server. Multiple redundant representa-
every month and served from hundreds of thousands of streaming
tions are an easy way to achieve this task, and they are widely used
media servers. Second only to the number-one Web browser, the
in commercial systems [4]. To dynamically assemble compressed
leading streaming media player has more than 250 million regis-
bit-streams without drift problems, S-frames [10] and, recently,
tered users, with more than 200,000 new installations every day.
SP-frames [11] have been proposed. Embedded scalable video
This is happening despite the notorious difficulties of transmitting
representations such as FGS [12] would be more elegant for rate
data packets with a deadline over the Internet, due to variability
adaptation, but they are still considerably less efficient, particularly
in throughput, delay and loss. It is not surprising that these chal-
at low bit-rates. Embedded scalable representations are a special
lenges, in conjunction with the commercial promise of the tech-
case of multiple description coding of video that can be combined
nology, has attracted considerable research efforts, particularly
advantageously with packet path diversity [9] [13]. Finally, the
directed towards efficient, robust and scalable video coding and
source coder can trade-off some compression efficiency for higher
transmission [1] [2].
error resilience [14]. For live encoding of streaming video, feed-
A streaming video systems has four major components: 1. back information can be employed to adapt error resiliency, yield-
The encoder application (often called the “producer” in commer- ing the notion of channel-adaptive source coding. Such schemes
cial systems) that compresses video and audio signals and uploads have been shown to possess superior performance [15]. For pre-
them to the media server. 2. The media server that stores the com- compressed video stored on a media server, these channel-adaptive
pressed media streams and transmits them on demand, often serv- source coding techniques can be effected through assembling se-
ing hundreds of streams simultaneously. 3. The transport mecha- quences of appropriately precomputed packets on the fly.
nism that delivers media packets from the server to the client for
the best possible user experience, while sharing network resources In our opinion, the most interesting recent advances in video
fairly with other users. 4. The client application that decompresses streaming technology are those that consider several system com-
and renders the video and audio packets and implements the inter- ponent jointly and react to the packet loss and delay, thus perform-
active user controls. For the best end-to-end performance, these ing channel-adaptive streaming. In this paper, we review some
components have to be designed and optimized in concert. recent advances in channel-adaptive streaming. As an example of
The streaming video client typically employs error detection a new receiver-based technique, we discuss adaptive media play-
2. out in Section 2, to reduce delay introduced by the client buffer among packets in a way that minimizes a Lagrangian cost function
and provide rate scalability in a small range. We then review rate- of rate and distortion. For example, consider a scenario in which
distortion optimized packet scheduling as the most important re- uniformly sized frames of media are placed in individual packets,
cent advance in transport mechanisms in Section 3. An example and one packet is transmitted per discrete transmission interval. A
of a channel-adaptive encoder-server technique we discuss is the rate-distortion optimized streaming system decides which packet
new idea of packet dependency control to achieve very low latency to transmit at each opportunity based on the packets’ deadlines,
in Section 4. All of these techniques are applicable for wireline as their transmission histories, the channel statistics, feedback infor-
well as wireless network. Architectures and the specific challenges mation, the packets’ interdependencies, and the reduction in dis-
arising for wireless video streaming are discussed in the conclud- tortion yielded by each packet if it is successfully received and
ing Section 5. decoded.
The framework put forth in [19] is flexible. Using the frame-
2. ADAPTIVE MEDIA PLAYOUT work, optimized packet schedules can be computed at the sender
or receiver. The authors have also presented simplified methods to
Adaptive media playout (AMP) is a new technique that allows a compute approximately optimized policies that require low com-
streaming media client, without the involvement of the server, to putational complexity. Furthermore, the framework, as shown in
control the rate at which data is consumed by the playout process. [20], appears to be robust against simplifications to the algorithm
For video, the client simply adjusts the duration that each frame and approximations of information characterizing the value of in-
is shown. For audio, the client performs signal processing in con- dividual packets with respect to reconstruction distortion. Low
junction with time scaling to preserve the pitch of the signal. In- complexity is important for server-based implementation, while
formal subjective tests have shown that slowing the playout rate of robustness is important for receiver-based implementations, where
video and audio up to 25% is often un-noticeable, and that time- the receiver makes decisions. We have recently extended Chou’s
scale modification is preferable subjectively to halting playout or framework for adaptive media playout, such that each packet is op-
errors due to missing data [16] [17]. timally scheduled, along with a recommended individual playout
One application of AMP is the reduction of latency for stream- deadline. For that, the distortion measure is extended by a term
ing media systems that rely on buffering at the client to protect that penalizes time-scale modification and delay [18].
against the random packet losses and delays. Most noticeable to
the user is the pre-roll delay, i.e., the time it takes for the buffer 4. CHANNEL-ADAPTIVE PACKET DEPENDENCY
to fill with data and for playout to begin after the user makes a CONTROL
request. However, in streaming of live events or in two-way com-
munication, latency is noticeable throughout the entire session. While for voice transmission over the Internet latencies below 100
With AMP, latencies can be reduced for a given level of protection ms are achievable, video streaming typically exhibits much higher
against channel impairments. For instance, pre-roll delays can be latencies, even if advanced techniques like adaptive media play-
reduced by allowing playout to begin with fewer frames of media out and R-D optimized packet scheduling are used. This is the
stored in the buffer. Using slowed playout to reduce the initial con- result of dependency among packets due to interframe prediction.
sumption rate, the amount of data in the buffer can be grown until If a packet containing, say, one frame is lost, the decoding of all
sufficient packets are buffered and playout can continue normally. subsequent frames depending on the lost frame will be affected.
For two-way communication or for live streams, AMP can be used Hence, in commercial systems, time for several retransmission at-
to allow smaller mean buffering delays for a given level of protec- tempts is provided to essentially guarantee the error-free reception
tion against channel impairments. The application was explored of each frame, at the cost of higher latency.
for the case of two-way voice communication in [16]. It is easily Packet dependency control has been recognized as a power-
extended to streaming video. In [18] it is shown that this simple ful tool to increase error-robustness. Earlier work on this topic
playout control policy can result in latency reductions of 30% for includes long-term memory prediction for macroblocks for in-
a given level of protection against underflow. creased error-resilience [21], the reference picture selection (RPS)
AMP can also be used for outright rate-scalability in a limited mode in H.263+ [22] and the emerging H.26L standard [23], and
range, allowing clients to access streams which are encoded at a the video redundancy coding (VRC) technique [24]. Those encod-
higher source rate than their connections would ordinarily allow ing schemes can be applied over multiple transmission channels
[18]. for path diversity to increase the error-resilience [9] [25], similar
to what has been demonstrated for real-time voice communication
3. R-D OPTIMIZED PACKET SCHEDULING [26].
In our recent work [27], in order to increase error-resilience
The second advance that we are reviewing in this paper is a trans- and eliminate the need for retransmission, multiple representations
port technique. Because playout buffers are finite, and because of certain frames are pre-stored at the streaming server such that
there are constraints on allowable instantaneous transmission rates, a representation can be chosen that only uses previous frames as
retransmission attempts for lost packets divert transmission oppor- reference that may be received with very high probability. We con-
tunities from subsequent packets and reduce the amount of time sider the dependency across packets and dynamically control this
that subsequent packets have to successfully cross the channel. A dependency in adapting to the varying channel conditions. With
streaming media system must make decisions, therefore, that gov- increased error-resilience, the need for retransmission is elimi-
ern how it will allocate transmission resources among packets. nated. Buffering is needed only to absorb the packet delay jit-
Recent work of Chou et al. provides a flexible framework to ter, so that the buffering time can be reduced to a few hundred
allow the rate-distortion optimized control of packet transmission milliseconds. Due to the trade-off between error-resilience and
[19] [20]. The system can allocate time and bandwidth resources coding efficiency, we apply optimal picture type selection (OPTS)
3. within a rate-distortion (RD) framework, considering video con- are required that can distinguish loss due to congestion and a dete-
tent, channel loss probability and channel feedback (e.g. ACK, riorating wireless channel. Some promising research is underway,
NACK, or time-out). This applies to both pre-encoding the video but the proposed techniques are still limited [28]. ARQ in the radio
offline and assembling the bitstreams during streaming. In coding link can avoid wireless losses altogether, but reduce throughputs
each frame, several trials are made, including using the I-frame as and increases delay. For streaming applications where delay is not
well as Inter-coded frames using different reference frames in the critical, radio link ARQ is superior.
long-term memory. The associated rate and expected distortion Fig. 1 (b) shows an architecture with a proxy server separat-
are obtained to calculate the cost for a particular trial through a ing the wireless and wireline portion of the network. Instead of
Lagrangian formulation. The distortions are obtained through an connecting to the back-end streaming media server directly, the
accurate binary tree modeling considering channel loss rate and client connects to the proxy server, which in turn connects to the
error propagation. The optimal picture type is selected such that streaming media server. The proxy is responsible for pre-fetching
the minimal RD cost is achieved. Even without retransmission, and buffering packets, such that they are available when the mo-
good quality is still maintained for typical video sequences sent bile client needs them. Channel-adaptive streaming techniques can
over lossy channels [27]. Thus the excellent robustness achiev- now be applied to each of the two connections separately. The
able through packet-dependency control can be used to reduce or proxy server might also implement simple transcoding to reduce
even entirely eliminate retransmission, leading to latencies similar the bit-rate or increase error resilience for low-delay applications.
to those for Internet voice transmission. Fig. 1 (c) shows an architecture where a gateway between the
wireline and wireless part of the network marks the territory of the
Internet. For the wireless link, an integrated wireless media proto-
5. CHALLENGES OF WIRELESS VIDEO STREAMING
col, tailored to the needs of wireless audio and video transmission,
In our previous discussion, we have not differentiated between is used. This integrated wireless media protocol could even be a
video streaming for the wireline and the wireless Internet. Increas- circuit-switched multimedia protocol stack, such as H.324M [29].
ingly, the Internet is accessed from wireless, often mobile termi- Channel-adaptive streaming techniques would be used between
nals, either through wireless LAN, such as IEEE 802.11, or 2.5G the gateway and the streaming media server, while packet-oriented
or 3G cellular networks. It is expected that in 2004, the number of streaming media techniques, such as dynamic packet scheduling,
mobile Internet terminal will exceed the number of fixed terminals might not be applicable to the wireless link. With H.324M, error-
for the first time. Wireless video streaming suffers from the same resilience of the video stream is important, as is rate scalability or
fundamental challenges due to congestion and the resulting best- rate control to accommodate variable effective throughput even on
effort service. Packets still experience variable delay, loss, and a nominally fixed-rate link. The 3G-PP consortium has evolved the
throughput, and channel-adaptive techniques as discussed above ITU-T recommendation H.324M into 3G-324M, which also sup-
are important to mitigate these problems. ports MPEG-4 video, in addition to H.263v2, for conversational
services.
The mobile radio channel, however, introduces specific addi-
The streaming architecture in Fig. 1 (c) is actually being im-
tional constraints, and many of the resulting challenges still hold
plemented by some companies, but it appears to be a short-term
interesting research problems. Fading and shadowing in the mo-
solution. The segregation of the world into wireline and wireless
bile radio channel leads to additional packet losses, and hence
terminals is a far too serious drawback. Establishing and tearing
TCP-style flow control often results in very poor channel uti-
down a circuit for each video stream is cumbersome and wasteful,
lization. Frame sizes of wireless data services are usually much
particularly considering that a packet-switched always-on connec-
smaller than the large IP packets preferable for video streaming,
tion will soon be widely available in 2.5G and 3G systems. The
hence fragmentation is necessary. Since the loss of any one frag-
open architecture of IP-solutions as shown in Fig. 1 (a) and (b) will
ment knocks out an entire IP packet, this effectively amplifies the
undoubtedly prevail.
loss rate of the wireless link. An obvious remedy is to use ARQ
for the radio link, trading off throughput and delay for reliability
of the wireless link. Most, but not all mobile data services operate 6. REFERENCES
in this way.
Other objections against using IP for streaming over mobile [1] M. R. Civanlar, A. Luthra, S. Wenger, and W. Zhu (eds.),
radio links is the RTP/UDP/IP encapsulation overhead that can use Special Issue on Streaming Video, IEEE Trans. CSVT,
up a significant portion of the throughput of the expensive wireless vol. 11, no. 3, Mar. 2001.
link. Moreover, mobility management in IP is lacking, and mobile [2] C. W. Chen, P. Cosman, N. Kingsbury, J. Liang, and
IP protocols that employ further encapsulation might be even more J. W. Modestino (eds.), Special Issue on Error Resilient Im-
wasteful. Header compression, however, can very efficiently over- age and Video Transmission, IEEE Journal on Selected Area
come this problem and will be widely deployed in future radio in Communications, vol. 18, no. 6, June 2001.
systems. [3] Y. Wang, and Q. Zhu, “Error control and concealment for
Three alternative architectures for wireless video streaming video communication: a review,” Proceedings of the IEEE,
are shown in Fig. 1. The end-to-end architecture in Fig. 1. (a) pre- vol. 86:5, p. 974-97, May 1998.
serves the Internet paradigm of stateless routing with connection- [4] G. J. Conklin, G. S. Greenbaum, K. O. Lillevold, A. F. Lipp-
oriented services implemented in the terminals. Channel-adaptive man, and Y. A. Reznik, “Video coding for streaming media
streaming methods, as discussed above, would be implemented in delivery on the Internet,” IEEE Trans. CSVT, vol. 11, no. 3,
the client and the server only. We need to distinguish systems with pp. 269-81, Mar. 2001.
ARQ on the radio link and lossy system. In order to solve the [5] W. Tan, and A. Zakhor, “Video multicast using layered FEC
problem of sharing bandwidth fairly both in the wireline and the and scalable compression,” IEEE Trans. CSVT, vol. 11, no. 3,
lossy wireless links, reliable loss differentiation algorithms (LDA) pp. 373-87, Mar. 2001.
4. session control: RTSP over TCP or UDP
application:
media
application [13] Y. Wang, M. Orchard, V. Vaishampayan, and A. R. Reib-
server
control: RTCP over TCP man, “Multiple description coding using pairwise correlating
transforms,” to appear in IEEE Trans. Image Processing.
transmission: RTP over UDP [14] R. Zhang, S. L. Regunathan, and K. Rose, “Video coding
TCP or UDP
with optimal inter/intra-mode switching for packet loss re-
transport layer TCP / UDP
silience,” IEEE Journal on Selected Areas in Communica-
network layer IP IP IP tions, vol. 18, no. 6, pp. 966-76, June 2000.
Router
wireless [15] B. Girod, and N. F¨ rber, “Wireless video,” in A. Reibman,
a
link layer data link layer
wired bit-pipes
service M.-T. Sun (eds.), Compressed Video over Networks, Marcel
Server Client Dekker, 2000.
(a)
[16] Y. J. Liang, N. F¨ rber, and B. Girod, “Adaptive playout
a
application:
RTSP over TCP or UDP RTSP over TCP or UDP
application scheduling using time-scale modification in packet voice
media
server RTCP over TCP RTCP over TCP
communication,” Proc. ICASSP ’01, Salt Lake City, May
Proxy
2001.
RTP over UDP Server RTP over UDP [17] E. G. Steinbach, N. F¨ rber, and B. Girod, “Adaptive play-
a
out for low Latency video streaming,” Proc. International
transport layer TCP or UDP TCP or UDP TCP / UDP
Conference on Image Processing (ICIP-01), Thessaloniki,
network layer IP IP IP Greece, Oct. 2001.
wireless [18] M. Kalman, E. Steinbach, and B. Girod, “R-D Optimized
link layer data link layer
wired bit-pipes
service
Media Streaming Enhanced With Adaptive Media Playout,”
Server Client in Proc. Int’l Conf. Multimedia and Exhibition, Lausanne,
(b) Switzerland, Aug. 2002.
RTSP over TCP or UDP
[19] P. A. Chou and Z Miao, “Rate-distortion optimized
application: Integrated
media streaming streaming of packetized media,” IEEE Transac-
server RTCP over TCP media
application tions on Multimedia, February 2001. Submitted.
Gateway
RTP over UDP http://research.microsoft.com/ pachou
Integrated
wireless media
[20] P. A. Chou and A. Sehgal, “Rate-distortion optimized
transport layer TCP or UDP protocol
(e.g. 3G-324M)
receiver-driven streaming over best-effort networks,” Packet
IP
Video Workshop, Pittsburg, PA, April 2002.
network layer
[21] T. Wiegand, N. F¨ rber, and B. Girod, “Error-resilient video
a
wireless
link layer
wired bit-pipes data link layer transmission using long-term memory motion-compensated
service
Server Client
prediction,” Journal on Selected Areas in Communications,
(c) vol. 18, no. 6, pp. 1050–1062, June 2000.
[22] ITU-T Recommendation H.263 Version 2 (H.263+), Video
coding for low bitrate communication, Jan. 1998.
[23] ITU-T Video Coding Expert Group, H.26L Test Model
Figure 1: Wireless streaming architectures: (a) end-to-end, (b) Long Term Number 8, July 2001, online available at
proxy server, (c) gateway with integrated media protocol. ftp://standard.pictel.com/video-site/h26L/tml8.doc.
[24] S. Wenger, G. D. Knorr, J. Ott, and F. Kossentini, “Error
resilience support in h.263+,”IEEE Journal on Circuits and
[6] W. Tan, and A. Zakhor, “Real-time Internet video using er-
Systems for Video Technology, vol. 8, no. 7, pp. 867–877,
ror resilient scalable compression and TCP-friendly transport
Nov. 1998.
protocol,” IEEE Trans. Multimedia, vol. 1, no. 2, pp. 172-86, [25] S. Lin, S. Mao, Y. Wang, and S. Panwar, “A reference pic-
June 1999. ture selection scheme for video transmission over ad-hoc net-
[7] S. McCanne, M. Vetterli, and V. Jacobson, “Low-complexity works using multiple paths,”Proc. of the IEEE International
video coding for receiver-driven layered multicast,” IEEE Conference on Multimedia and Expo (ICME), Aug. 2001.
Journal on Selected Areas in Communications, vol. 15, no. 6, [26] Y. J. Liang, E. G. Steinbach, and B. Girod, “Real-time voice
pp. 983-1001, Aug. 1997. communication over the Internet using packet path diversity,”
[8] J. Shin, J. Kim, and C.-C. J. Kuo, “Quality-of-service map- Proceedings ACM Multimedia 2001, Oct. 2001, pp. 431–
ping mechanism for packet video in differentiated services 440, Ottawa, Canada.
network,” IEEE Transactions on Multimedia, vol.3, no.2, [27] Y. J. Liang and B. Girod, “Rate-distortion optimized low-
pp.219-31, June 2001. latency video streaming using channel-adaptive bitstream
[9] J. Apostolopoulos, T. Wong, W. Tan, and S. Wee, “On mul- assembly,” accepted by IEEE International Conference on
tiple description streaming with content delivery networks,” Multimedia and Expo, Aug. 2002.
IEEE Infocom, July 2002. [28] S. Cen, P. C. Cosman, and G. M. Voelker, “End-to-end dif-
[10] N. F¨ rber, and B. Girod, “Robust H.263 Compatible Video
a ferentiation of congestion and wireless losses,” SPIE Multi-
Transmission for Mobile Access to Video Servers,” Proceed- media Computing and Networking (MMCN2002), San Jose,
ings ICIP 97, vol. 2, pp. 73-76, Santa Barbara, Oct. 1997. CA, Jan 18-25, 2002.
[11] M. Karczewicz, and R. Kurceren, “A proposal for SP- [29] N. F¨ rber, B. Girod, and J. Villasenor, “Extensions of
a
frames,” Proposal to H.26L, Jan. 01. the ITU-T Recommendation H.324 for error resilient video
[12] M. van der Schaar, and H. Radha, “A hybrid temporal-SNR transmission,” IEEE Communications Magazine, vol. 36, no.
fine-granular scalability for Internet video,” IEEE Trans. 6, pp. 120-128, June 1998.
CSVT, vol. 11, no. 3, pp. 318-31, Mar. 2001.