Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Virtualized Screen: A Third Element for Cloud - Mobile Convergence


Published on

In recent years, we have witnessed the strong growth of cloud computing, which refers to both applications delivered as services over the Internet and the infrastructure (including both hardware and software systems) in the data centers that supports those services.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Virtualized Screen: A Third Element for Cloud - Mobile Convergence

  1. 1. Wenjun ZengMultimedia at Work University of Missouri-Columbia Virtualized Screen: A Third Element for Cloud—Mobile Convergence Yan Lu,Shipeng Li, and I n recent years, we have witnessed the strong growth of cloud computing, which refers to screen to refer to the display images. There are a number of advantages to putting screen ren- Huifeng Shen both applications delivered as services over dering in the cloud. Microsoft the Internet and the infrastructure (including First, because screen rendering is closely Research Asia both hardware and software systems) in the coupled with program execution and data stor- data centers that supports those services.1 In age, putting screen rendering in the cloud general, cloud computing provides a central- where data storage and program execution are ized platform where programs can be executed centralized actually simplifies the cloud com- and data can be stored. Cloud services built on puting architecture. Second, screen rendering, the platform can offload many computing tasks especially graphics-rich screen rendering, is normally solely done on the client devices and not a trivial task. It often requires a powerful can be accessed instantly from anywhere at any CPU and GPU on the client devices. Moving time. However, until recently, the place where screen rendering to the cloud will greatly re- screen images that interface with users are ren- duce the hardware requirement for client de- dered has been mostly done locally on client vices and thus make low-cost devices possible. devices. Third, even if the client devices are equipped Our argument is that the screen rendering with powerful CPU and GPU processors, off- can also be moved to the cloud and the ren- loading all or part of screen-rending task to dered screen can be delivered as part of the the cloud will spare the client processing cloud services. In general, the screen represents power to handle more effectively higher prior- the whole or part of the display images. In a ity tasks (such as the local UI) and rich interac- broad sense, it also represents a collection of tions (such as touch and gesture recognition) data involved in user interfaces such as display that require fast responses locally. Last, images, audio data, mouse, keyboard, pen and screen rendering in the cloud also brings a touch inputs, and other multimodality inputs new avenue for optimizing the overall comput- and outputs. In this article, we use the term ing experience. Screen virtualization or screen rendering in the cloud doesn’t always mean putting the en- Editor’s Note tire screen-rendering task in the cloud. Depend- Mobile and cloud computing have emerged as the new computing ing on the actual situations—such as localplatforms and are converging into a powerful cloud—mobile comput- processing power, bandwidth and delay of theing platform. This article envisions a virtualized screen as a new dimen- network, data dependency and data traffic,sion in such a platform to further optimize the overall computing and display resolution—screen rendering canexperience for users. In a virtualized screen, screen rendering is done be partially done in the cloud and partiallyin the cloud, and delivered as images to the client for interactive dis- done at the clients (that is, scalable screen virtu-play. This enables thin-client mobile devices to enjoy many computa- alization); collaboratively, a complete renderedtionally intensive and graphically rich services. Technical challenges screen is presented to the user. This is very sim-are discussed and addressed. Two novel cloud-mobile applications, ilar to traditional cloud computing in whichCloud Browser and Cloud Phone, are presented to demonstrate the we have to decide where we should put theadvantages of such a virtualized screen. program execution and data storage, either —Wenjun Zeng remotely in the cloud or locally on the device, to obtain an optimized computing experience. 4 1070-986X/11/$26.00 c 2011 IEEE Published by the IEEE Computer Society
  2. 2. Screen Cloud Client Figure 1. The virtualization Virtual display/UI Display/UI conceptual diagram of the cloud—client computing Screen Screen architecture. Cloud Local computing computing Program Program (CPU/GPU) Data Data Data Local farm storageThis flexibility added by screen virtualization computing architecture, the data and the pro-can enable us to provide an even further opti- gram can be stored, loaded, and run remotelymized computing experience by balancing be- and/or locally. To take advantage of thetween local client devices and remote clouds cloud, computationally intensive tasks are usu-in terms of data storage, program execution, ally undertaken in the cloud to generate someand now, screen rendering. intermediate results (for example, HTML However, rendering a screen in the cloud data), which are then delivered to the clientsalso introduces obstacles for the client devices for the creation of a locally processed displayto access the virtual screen, if it needs to main- screen. In other words, local screen renderingtain high-fidelity display images and responsive is separated from the data storage and programuser interactions. Fortunately, we have already execution, and is connected to them throughdeveloped a number of advanced multimedia the Internet.and networking technologies to address these Up until now, the virtualization of theissues. Ultimately, we would like to define a screen in the cloud has been an underdevel-common cloud API for cloud computing with oped area in cloud computing. An even lessscalable screen virtualization, with which the addressed area is how to leverage the virtualdevelopers never have to care where the data screen in the cloud and combine it withstorage, program execution, and screen render- local rendering capabilities to give the sameing actually occur because the cloud services for or even better user experiences across differ-the API will adaptively and optimally distribute ent devices, regardless of their computing ca-the storage, execution, and rending among the pability, rendering capability, bandwidth,cloud and the clients. Similar to the GUI, which and screen resolution. Figure 1 depicts thedrove the wide grassroots adoption of personal proposed conceptual architecture for cloud—computers, the cloud API, which bridges the client computing, in which the virtual screencloud and a wide variety of clients, will intro- is rendered in the cloud. Behaving like theduce new computing experiences both locally data and the program, the screen can be pro-and remotely. It will drive the evolution of cessed adaptively and collaboratively betweencloud—mobile computing into a revolution. the cloud and the clients, determined by the client and cloud capabilities. As we men- tioned earlier, the cloud—client architecture April—June 2011Screen in the cloud with screen adaptation could provide the fol- Rapid development of the Internet has pro- lowing benefits:vided opportunities for using remote comput-ing and storage resources hosted by powerful, Š an optimized interactive experience to endparallel, distributed machines in a public or users across different screens, such as laptop,private data center. In a typical cloud-client television, and phone; 5
  3. 3. Multimedia at Work System architecture User-input Decrypt Receive Thin-client, remote-computing systems are transform expected to provide high-fidelity displays and Screen Screen Screen responsive interactions to end users as if they capture encoder sender were using local machines. However, the com- plicated graphical interfaces and multimedia applications usually present technical chal- Previous screen frame Network lenges to thin-client developers for achieving in codec buffer efficient transmissions with relatively low- bandwidth links. Figure 2 depicts the proposed thin-client, remote-computing system, which Screen Screen Screen decouples the application logic (remote) and presenter decoder receiver the user interface (local) for clients to use re- mote servers deployed as virtual machines in User-input Encrypt Send the cloud. The servers and the clients commu- transform nicate with each other over a network through an interactive screen-remoting mechanism. The clients send user inputs to the remote serv-Figure 2. The Š a simplified programing model for develop- ers, and the servers return screen updates to theinteractive screen- ers, as if they were building local applica- clients as a response.remoting system. tions without struggling on various types The screen-update model determines whether of real-time data communications; and the screen can be efficiently compressed and transmitted to the client. Most of the existing Š a quick model for software providers to de- thin-client systems, such as Virtual Network ploy software services in the cloud, through Computing (VNC)2 and Remote Desktop Pro- a mechanism of virtual machines plus screen tocol (RDP),3 represent screen updates with remoting—for example, this model could graphics primitives of arbitrarily-sized regions. help to quickly deploy trial software without Such a mechanism allows the server to simply piracy concerns. forward the graphics primitives to be updated into the compressors and discard other stable Is it feasible to deliver real-time screen as ser- regions directly. At the client side, the screen vices through the Internet to achieve near-local presenter renders the received graphics primi- interactive experiences? The screen as display tives and overlays rectangular areas of pixels images indeed contains a large volume of in destined regions. However, the regions to data. Fortunately, its representation as image be updated are usually small and at arbitrary sequences has a lot of redundancy that can be positions, such as menus or edit boxes. Encod- efficiently removed. Moreover, screen refresh- ing these small regions with arbitrary locations ment with a full-frame unit can make the la- will result in the system suffering from com- tency of screen updating at the client side pression efficiency degradation. relatively constant. While breakthroughs of In contrast to the architecture based on multimedia, hardware, and networking tech- arbitrarily-sized regions, our thin-client system nologies are around the corner, our recent employs a frame-based screen-representation progress on a remote-computing system with model. This model reads all the pixels on the virtualized screen implies that screen compres- screen from the frame buffer at once and sion and transmission can be very efficient feeds the whole screen image into the compres- even with technologies available today. sors and transmitters. At the client side, the screen presenter replaces the whole screen Remote computing with with the newly decoded one. As shown inIEEE MultiMedia virtualized screen Figure 2, both the server and the client store Following the cloud-computing conceptual the same reference frame, which is used to re- architecture depicted in Figure 1, we have move the redundancies between consecutive developed a thin-client, remote-computing sys- frames. Besides the advantage in compression, tem that leverages interactive screen-remoting the frame-based representation model also sim- technologies. plifies the system architecture without the 6
  4. 4. Previousoverhead in scheduling the updates of screen frameregions. Furthermore, this frame-based repre-sentation model could recover from errors due Skip-block Skip codingto packet loss very quickly without retransmis-sions through keyframe refreshing. Picture-block Entropy Compared with existing remote-computing Difference Diffe coding codingsystems, such as VNC and RDP, the proposed dete detection Blocksolution could significantly improve user expe- Current classification frame Text-blockrience in terms of both the smoothness of the Nonskip codingscreen update and the response in interaction,which are enabled through the advancedscreen-compression and transmission technol- which is specifically designed to handle pixels Figure 3. Block diagramogies introduced next. belonging to a background picture in a text- of the proposed screen on-picture region. compression algorithm.Screen compression Figure 3 depicts the proposed screen- Screen images might include, but are not compression algorithm. The captured screenlimited to, webpages, slides, posters, images, image is segmented into nonoverlappingvideos, and anything showing up on your com- 16  16 blocks. After comparison with the pre-puter screen. For natural pictures, many exist- vious frame, each block is identified as either aing image- and video-coding standards (such skip or a nonskip block. Further, a classificationas, JPEG2000 and H.264/Advanced Video Cod- is performed to separate the nonskipped blocksing) have demonstrated excellent coding per- into picture blocks and text blocks, followed byformances. However, they are inefficient for the adaptive-coding schemes associated withcompressing screen images that in most cases the block types. The statistical gradient histo-contain rich text. The challenges for compress- gram is employed for the purpose of blocking screen images are summarized as follows: classification. Obviously, 100 percent accurate block classification is almost impossible be-Š Computational complexity. The screen codec cause of the diversity of text-on-picture regions. has to handle a large amount of data to Fortunately, the proposed escape pixel-coding maintain high resolution and high-frame- scheme can avoid quality loss with a marginal rate screen updates. Meanwhile, the codec rate increase in case of incorrect classification. also has to spare most of the processing This screen codec outperforms traditional resources for other regular applications. natural image and video codecs for coding screen images in terms of rate, quality, andŠ Compression performance. The text, graphics, computational complexity. The encoding and and pictures composing a screen image decoding processes are also GPU-friendly in im- each possess a unique set of characteristics plementation. GPU acceleration with a main- and susceptibilities to encoding artifacts. stream video card could easily double the The screen codec has to handle them effi- frame rates of screen capturing and coding ciently within a single framework, and yet over systems without GPU acceleration. maintain low computational cost. In general, a typical screen image can be seg- Screen transmissionmented into four categories of regions: smooth, The latency of screen transmission is proba-text, picture, and text-on-picture. Based on our bly the most important factor to user experi-statistical analysis, the smooth and picture ence in a remote-computing system. Becauseregions are more suitable for coding in the the screen images are organized as a time-series-transform domain, while the text and text-on- like a video, some existing video-transmission April—June 2011picture regions can be efficiently compressed technologies could be leveraged here. Thein the pixel domain. To simplify the codec ar- strong dependency between predictivelychitecture, we propose a block-based coding al- coded frames makes the video stream sensitivegorithm with only two coding types: picture to transmission errors. Therefore, some buffer-and text. The text-block coding scheme also ing and error-control mechanisms have to beincludes an escape pixel-coding technique, employed, which causes additional delay on 7
  5. 5. Multimedia at WorkFigure 4. Block diagram Codec layer Error-concealed screen reconstructionof the proposed screen Screen-encoding pipelinetransmission Error-concealed Screen Flexible Hybridalgorithm. screen decoder capture slicing encoder Control layer Congestion-distortion optimization In-pipeline congestion Encoded-screen- Received-screen- controller buffer control buffer controller Transport layer Coded screen User Datagram Protocol User Datagram Protocol TCP-friendly rate-control w/automatic w/automatic based congestion controller repeat request repeat request ACK Server Client top of the inherent network-transmission help error-concealed screen reconstruction delay. at the client. Moreover, the screen decoder Furthermore, interactive screen remoting can handle out-of-order screen packets with- has much more stringent requirements on la- out producing drifting errors. tency than real-time video communications. For example, the user usually expects an imme- Š At the control layer, the congestion- diate response on the local display after clicking distortion optimization algorithm sched- a button, similar to what would be expected ules the packets in the sending buffer. when using a local machine. Achieving this per- These packets are scheduled on the basis formance requires quick round-trip message of their impact on network congestion processing and almost instant updates between and display distortion. Some packets may the virtual screen in the cloud and the local dis- be dropped without reencoding the frames play on the device. Therefore, existing video- followed. transmission technologies may not be sufficient to the interactive screen-remoting scenario. Š At the transport layer, the User Datagram Fortunately, the proposed screen codec Protocol transport with automatic repeat doesn’t introduce strong dependency between request is employed to balance the trans- consecutive frames, which can mitigate to a cer- mission rate and error control. To avoid tain extent the drifting errors in decoded screen congestion in the network, TCP-friendly images caused by transmission errors. More- rate control is used to estimate available over, screen remoting presents some unique bandwidth. features different from video-transmission sce- narios. For example, it’s not always necessary to update the whole screen frame all at once. Cloud-mobility convergence Rather, some out-of-order screen regions can The rapid evolution of mobile computing be presented early as they become available. offers a wide variety of freedom to mobile Therefore, it’s possible to achieve extremely- users. Besides communication functions, a low-latency transmission with the help of mobile device could be a computing, sensor, some content-aware scheduling schemes. control, gaming, and natural-interaction plat- Figure 4 illustrates the proposed screen- form. However, it’s difficult for a mobile device transmission algorithm, which is based on the to serve as a dominant device for all the user’sIEEE MultiMedia joint optimization of screen compression and computing needs, mainly because of its limited transmission. The screen images are processed capabilities in terms of computing, storage, in three layers: display, and interaction. On the other hand, cloud computing offers unlimited computing Š At the codec layer, the screen images are and storage capabilities through centralized encoded with a flexible slicing structure to data centers. The capacity of mobile devices 8
  6. 6. Virtual browser Mobile browser Ctrl Msg Ctrl MsgHTML Sample* Sample Ctrl Msg Sample Sample* Rendered files webpage Real-Time Browser Sample Sample RTP Sample Sample Transfer engine encoder scheduler network decoder composition Protocol Cache Encoded sample buffer Rendered screen Phone screen image and extracted location webpage objects Nonupdate screen regionscould be significantly augmented by cloud exemplar mobile applications: Cloud Browser Figure 5. The systemcomputing: and Cloud Phone. architecture of the Cloud Browser.Š Remote-computing terminal. A mobile device Cloud Browser with remote-desktop solutions (for example, As most mobile phones come with prein- RDP and VNC) could use remote computer stalled Web browsers, mobile browsing is be- programs in the cloud, though existing com- coming increasingly popular. However, many puter programs are usually designed for a rel- mobile browsers have limited support for com- atively large display and difficult to adapt to plicated HTML objects, such as full-feature mobile screens. Flash, JavaScript, and Silverlight. Installation of add-ons from some websites in many casesŠ Cloud-services harnessing platform. A mobile isn’t possible. Even worse, the website owners device could access unlimited services in might have to develop and maintain two ver- the cloud through the Internet. In an ex- sions of the same content, one full version for treme case, except for the user interfaces desktop browsers, and the other simplified and sensors, all program execution, data version for mobile browsers. In most cases, storage, and screen rendering could be the mobile version loses rich multimedia infor- done in the cloud. In general, by balancing mation and results in a compromised and program execution, data storage, and screen degraded browsing experience. It’s desirable to rendering between the cloud and the mobile bring the desktop-like browsing experiences to device, we could achieve an optimized com- mobile phones. puting experience adaptive to the capacities Recently, we have built a cloud-assisted of the cloud and the mobile device. mobile browser, the Cloud Browser, which is an extension to the proxy-based Web browser April—June 2011 The virtualization of the screen in the cloud we developed.4 Figure 5 illustrates the architec-introduces a new avenue to further optimize ture of the Cloud Browser, which consists of athe cloud—mobile computing experience and virtual browser deployed in the cloud and a mo-could potentially accelerate cloud—mobile con- bile browser installed on the device. The virtualvergence. On the basis of the proposed interac- browser extracts and renders webpage objectstive screen-remoting system, we showcase two into samples, encodes and multiplexes them 9
  7. 7. Multimedia at Work assemble these samples into a webpage for Cloud Program Phone the users to view to further shorten the web- Virtual Cellphone phones + page rendering time. While the HTML webpage data could be converted to a screen image, the + screen Browser screen image could also be embedded into the Cloud Phone Cloud Phone HTML file as a webpage element to help real- Unlimited service client programs time cloud-client interactive communications. Laptop Cloud Phone The smartphones as a personal computing device are broadly available to mobile users. In addition to a wide variety of phone brands and a number of different operating systemsFigure 6. Conceptual into a composite screen format, and then sends in the market, the rapid development of hard-diagram of the Cloud the data to the client with optimized schedul- ware and application software accelerates thePhone. ing. The mobile browser receives and decodes pace at which phone hardware is upgraded. Is these samples, and assembles them for presenta- it possible to replace the frequent hardware tion to mobile users. The mobile browser also upgrades with only software and services sends back control messages, such as visibility upgrades? Recently, we have built a prototype information about the client display and user- of a complete cloud-service-based phone solu- interaction events, to the virtual browser. tion, the Cloud Phone, with which the phone As described previously, the composite user can access the ever increasing programs screen format is defined as a sequence of sam- and services available in the cloud, even on a ples, where the sample can be either a screen low-cost device without the need to update image or a webpage object. The webpage object phone hardware anymore. consists of object ID, type, and properties such Figure 6 illustrates the conceptual diagram as the position in a screen. We define four types of the Cloud Phone. On the one side, with of webpage objects: hyperlink, text input, pass- the virtual phones deployed in the cloud with word input, and dynamic. The object samples user-preferred mobile operating systems (for ex- are not intended to be used for direct display, ample, Windows Phone, iPhone, and Android) but for improving user interaction experiences and the unlimited apps or services, the Cloud and avoiding unnecessary bandwidth con- Phone service could switch between these sumption. For example, the client can use the phone user interfaces and programs at any object information to execute some simple in- time and instantly. On the other side, the teraction logic to give quick responses to the Cloud Phone client could be implemented in user. Moreover, all or part of the screen could any form, such as a cellphone, browser, or lap- be selectively updated according to the object top. As a complete cloud-service-based phone types and the user’s options. solution, the Cloud Phone could provide the The Cloud Browser is a complete end-to-end following benefits: solution that offloads the rendering of web- pages from the client to the cloud or some Š It could make a low-cost smartphone possi- ble, where the phone is just a terminal proxy servers, reformats the webpages to elimi- with basic communication functions and a nate incompatibility with mobile devices, and screen decoder. Complicated applications maximizes the object-level interactivity while become much simpler by shifting comput- keeping the full desktop browsing experience. ing tasks to the cloud. In many cases, the webpage’s bits after com- pression could be even less than that of the Š It could emulate any existing devices original HTML files. Even better, the proposed (phones, game consoles, PDAs). It also couldIEEE MultiMedia mobile browser could significantly reduce allow the user to enjoy the same personal page loading time by pulling only a visible por- experience on any phone. tion of the screen image rendered in the cloud for that webpage. Moreover, because most Š It could have complete knowledge of users’ screen samples have already been rendered interactions in the cloud for potentially pro- in the cloud, the local client could quickly viding better services. 10
  8. 8. There still remains some future work evolution of cloud computing become arequired to deploy the services of Cloud revolution. MMPhone. For example, the offline experienceshould be taken into account, because the net-work might not always be available. Neverthe- Referencesless, the concept of Cloud Phone inspires us 1. M. Armbrust et al., ‘‘A View of Cloud Comput-to think about the next-generation mobile plat- ing,’’ Comm. ACM, vol. 53, no. 4, 2010,forms with the consideration of cloud—mobile pp. 50-58.convergence. A middleware layer might exist 2. T. Richardson et al., ‘‘Virtual Network Comput-on top of the phone OS, which could be more ing,’’ IEEE Internet Computing, vol. 2, no. 1, 1998.suitable than the general Web user interface 3. Microsoft, ‘‘Remote Desktop Protocol,’’ 2009;for running rich cloud applications. aa383015.aspx.Conclusions 4. H. Shen et al., ‘‘A Proxy-Based Mobile Web Cloud computing is not a simple extension Browser,’’ Proc. ACM Multimedia, ACM Press,of Web services. Breakthroughs may come 2010, pp. 763-766.with the introduction of a new application in-terface model between the cloud and the cli- Contact author Yan Lu at User interaction with the cloud throughclient—cloud communications bounds the user Contact editor Wenjun Zeng at zengw@experience and presents many technical chal- We have shown that advanced multime-dia compression and networking technologiescan bridge the devices and the cloud effectivelyand efficiently, and potentially could help the April—June 2011 11
  9. 9. This article was featured in For access to more content from the IEEE Computer Society, see articles, podcasts, and more.