H I G H - A V A I L A B I L I T Y    C L U S T E R S

                         Architecting Large-Scale
H I G H - A V A I L A B I L I T Y                  C L U S T E R S

           Flexibility: A large-scale streaming sol...
H I G H - A V A I L A B I L I T Y              C L U S T E R S

       distribution is accomplished by keeping track of...
Media Server (4 servers)
  Machine type                   Dell PowerEdge 6450

  Processors                     Four 700 M...
H I G H - A V A I L A B I L I T Y       C L U S T E R S

       running MPEG-2 streams than running MPEG-1 streams—a
Upcoming SlideShare
Loading in …5

Media-Streaming Solutions


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Media-Streaming Solutions

  1. 1. H I G H - A V A I L A B I L I T Y C L U S T E R S Architecting Large-Scale Media-Streaming Solutions By Rizwan Ali; Jenwei Hsieh, Ph.D.; and Edward Yardumian The development of streaming technology has enabled the emergence of media-rich applications, such as media-on-demand, interactive entertainment, distance learning, and e-commerce applications. This article describes a balanced, end-to-end infrastructure designed to support large-scale media streaming. S treaming technology allows digital media, whether audio Quality: Picture and sound quality must meet consumer or video, to be transferred in compressed formats across a expectations. Blotches, jitters, and irregular sound quickly network and played in real time, rather than after an entire turn users away. High-quality streaming requires high rates file has downloaded. Along with the standardization of media for- of data transfer and consistent real-time delivery. For exam- mats and advancements in network infrastructures, streaming has ple, videos compressed with the Moving Picture Experts fostered the emergence of numerous media-rich applications, Group MPEG-2 format require bandwidths ranging between which serve a wide range of purposes. For example, a sports Web 3 Mbps and 16 Mbps. These videos must also be delivered site might provide instant replays of a live football game, including to users and displayed at a consistent frame rate. This replays not broadcast on TV. Another site might use webcasts to jitter-free video playback requires that users receive each allow viewers around the world to watch Broadway shows live. video frame before it needs to be displayed. Media streaming has several advantages over terrestrial broad- Reliability: Program interruptions, delays, and system casts, including user interaction, fewer geographical constraints, failures cannot be tolerated; an effective system must and customized content delivery. As the availability of high-speed remain available despite component failures and sched- Internet access grows, the market presence of media streaming is uled maintenance. Many e-commerce providers have expanding. Furthermore, in the foreseeable future, third-generation learned an important lesson from Web site failures: Long- (3G) wireless networks capable of handling speeds between 384 Kbps term loss of reputation is often more damaging than and 2 Mbps will provide handheld devices with the bandwidth short-term loss of revenue. needed for truly dynamic services and applications such as video- Scalability: Large-scale streaming configurations must be conferencing and multiplayer gaming. Because of its crucial role in capable of scaling quickly to meet growing customer emerging Internet applications, the media-streaming industry is demand. The sustained bandwidth required by 1,000 expected to grow dramatically. streams at 1.5 Mbps (the speed required by MPEG-1 format media) totals 1.5 Gbps; 100,000 streams require sustained Challenges of large-scale streaming bandwidth of 150 Gbps, for both the network and storage A number of technical issues must be resolved before the full poten- systems. This speed is equivalent to 150 Gigabit Ethernet or tial of media streaming can be realized. Architects designing large- 3,125 disk drives running at 6 MB/sec, which is a typical scale media-streaming solutions face several major challenges: disk speed for random access operations. www.dell.com/powersolutions PowerSolutions 61
  2. 2. H I G H - A V A I L A B I L I T Y C L U S T E R S Flexibility: A large-scale streaming solution must support the servers and PowerVault® storage to build an architecture for large- full spectrum of network access speeds—from low bit rates scale media streaming. for slower Internet connections to high bit rates for intranet, cable modem, or asymmetric DSL (ADSL) connections. A flexi- A multi-tier, scale-out solution for streaming ble system must also allow users to easily migrate to the We used a multi-tier, scale-out approach to build an architec- broadband networks becoming more prevalent. ture to support a large-scale streaming solution that meets the requirements for quality, reliability, scalability, and flexibility. To meet these four requirements, a large-scale streaming solution Figure 1 illustrates the system configuration. The system ran a must include the following facilities: suite of streaming software applications from Streaming21, Inc., including the Streaming21® Media Server, Media Director, and Large storage capacity with a hierarchical structure to cost- Streaming Player. effectively archive media files as their number and size increase Streaming21 Media Server High-performance I/O bandwidth to handle hundreds or Streaming21 Media Server controls the delivery of hundreds of thousands of concurrent streams streams through sophisticated scheduling schemes. These schemes A high-performance network subsystem capable of deliver- ensure that all the streams are transferred continuously from storage ing concurrent streams to a large number of users to end users. Media Server relies on the Streaming21 Streaming File System (SFS), a real-time file system that uses intelligent disk block Even with these facilities, a large-scale streaming solution with allocation to improve access efficiency. high-quality service requires careful planning and integration of To provide the requisite network bandwidth, a media server can each system component. An architecture designed for large-scale use multiple network interface cards (NICs) to support the neces- streaming requires a well-balanced, end-to-end approach—from sary number of streams. The streams handled by the media server retrieving content from storage to transferring it across the net- are load balanced across the NICs in the system. Based on the num- work and playing it back smoothly. The following sections focus ber of streams and their bit rates, the system calculates the required on these technical challenges and the use of Dell® PowerEdge® bandwidth and assigns a certain number of streams to each NIC, distributing the load among the NICs and the processors that exe- cute the communications protocols. Since a single server has finite capabilities, multiple media Requests from clients Streams servers can be configured with a storage area network (SAN) to share a common pool of storage. The Streaming21 SFS allows Switch multiple servers to share read-only access to the SAN. Harnessing multiple servers to support the processing and transferring of a large number of streams allows the configuration to scale out. Media director Because every server has access to the same repository of content, any of the media servers in this configuration can fulfill a request. Redirect based Therefore, a server failure would not prevent users from accessing Media on load Media server server a particular media file. One possible system design would dedicate a subset of the media Logical volume access Logical volume access servers to handle streams with low bit rates and another subset to han- Storage switch dle streams with high bit rates. This design would allow server, storage, and network capacity to be scaled separately and incrementally. Storage Storage Streaming21 Media Director array array On top of the cluster farm of media servers, a server running Streaming21 Media Director acts as a traffic redirector—routing Storage area network (SAN) incoming requests to an available media server. Through traffic redirection, the media director balances the load across all of the Figure 1. A solution for large-scale streaming media servers and reroutes streams from failed servers. Load 62 PowerSolutions
  3. 3. H I G H - A V A I L A B I L I T Y C L U S T E R S distribution is accomplished by keeping track of the load on each Network, composed of PowerVault Fibre Channel storage and server, including the number of streams, their media formats, and bit switches. This configuration supported a large number of streams rates. The media director also records the overall capacity and total in several media formats with different bit rates and using various effective network bandwidth of each server. transport protocols. However, the goal was not to support the maximum number of streams possible. Rather, we aimed to Streaming21 Streaming Player explore the issues and challenges faced by architects of large-scale On the end-user side, the Streaming21 Streaming Player is used media-streaming solutions. to control the continuous transfer of media content from any Figure 2 illustrates the test configuration, including the major IP-accessible media server to clients. To support smooth stream- components and network infrastructure. Four Dell PowerEdge ing, buffering (on both the client and server side) and adaptive 6450 servers functioned as media servers. Each of these servers flow control absorb fluctuations in Internet speed and ensure had four 700 MHz Pentium® III Xeon™ processors, 4 GB of RAM, that the client connection is adequate for playback. Media Server two Intel® Pro/1000 Gigabit Ethernet NICs, and two QLogic® maintains the buffer levels of all of the clients by adjusting the Fibre Channel host bus adapters (HBAs). transfer rates of the streams. A PowerEdge 4400 dual-processor server functioned as the media director. The media director was connected directly to a 12-port, Test environment 3Com® SuperStack® II 9300 Gigabit Ethernet switch. All the clients’ For performance testing, we built a multi-tier, scale-out architec- streaming requests were directed to the media director, which for- ture with Dell PowerEdge servers and a PowerVault Storage Area warded them to the four media servers in a load-balancing fashion. A Gigabit Ethernet network backbone provided network connectivity among the media servers, media director, and three 18 Clients 24 Clients 18 Clients Fast Ethernet switches. These switches connected to 60 clients via Fast Ethernet. For the SAN, each media server HBA was connected to two PowerVault 51F Fibre Channel switches using optical cables. The Cisco Catalyst 3524 3Com 3300 Cisco Catalyst 3524 PowerVault 51F switches were then connected to three PowerVault with Gigabit Ethernet with Gigabit Ethernet with Gigabit Ethernet 650F Fibre Channel storage arrays. Each PowerVault 650F had dual storage processors (SPs) with 512 MB of cache per SP, and held ten 3Com 9300 12 port Gigabit Ethernet 9 GB, 10,000-rpm Fibre Channel drives. Figure 3 lists the hardware Media director and software components used in the test configuration. PowerEdge 4400 Test results We initially estimated that the test configuration could easily Media server 1 Media server 2 Media server 3 Media server 4 support more than 1,000 MPEG-1 streams. A special version of PowerEdge 6450 PowerEdge 6450 PowerEdge 6450 PowerEdge 6450 Streaming Player, the Streaming21 Streaming Player Emulator, was used to simulate a large number of clients (see Figure 4). Fabric 1 Fabric 2 The Streaming Player Emulator was installed on 60 Dell PowerVault 51F PowerVault 51F OptiPlex™ GX100s. This provided a close simulation of an actual production environment without configuring thousands of desktop PCs as real clients in the lab. The Streaming Player Emulator behaves similarly to a regular client, except that it does not use the video system to playback the Array 1 Array 2 Array 3 content received from the media server. This lightens the client load, PowerVault 650F PowerVault 650F PowerVault 650F because decoding the compressed media and playing back the stream are CPU-intensive tasks. The emulator was configured to Fast Ethernet Fibre Channel over Optical simulate 20 clients on each OptiPlex GX100. Gigabit Ethernet Fibre Channel over Copper The storage system was filled with a number of full-length media files. The MPEG-1 file was 75 minutes of video at 1.5 Mbps, Figure 2. Test configuration which resulted in a file about 800 MB. The MPEG-2 file was the 64 PowerSolutions
  4. 4. Media Server (4 servers) Machine type Dell PowerEdge 6450 Processors Four 700 MHz processors Memory 4 GB 100 MHz SDRAM Network adapters Dual Intel Pro/1000 Storage adapters Dual QLogic 2200 66 MHz HBAs Operating system Windows NT® 4.0, Enterprise Edition with Service Pack 6a Additional software Streaming21 Media Server, Windows® Media Server, Access Database Object (ADO) components, and Internet Explorer 5.0 Figure 4. Streaming21 Streaming Player Emulator Clients (60 clients) Machine type Dell OptiPlex GX100 MPEG-1 streams supported with TCP/IP Processors Intel Celeron™ 500 MHz processor Number of video files 1 25 Memory 128 MB 1 server1 399 399 Network adapters 3Com 3c920 Integrated Fast Ethernet Controller 4 servers 1050 985 Operating system Windows 2000 Server with Service Pack 1 Figure 5. Number of MPEG-1 streams supported with TCP/IP Network configuration Media Clients were connected to the Fast Ethernet switches via 100 Mbps full duplex. The server connections were fiber optic Gigabit MPEG-1 streams supported with UDP/IP Ethernet Number of video files 1 25 Switches Gigabit Ethernet network backbone with one 12-port Gigabit Ethernet switch (3Com 1 server1 399 399 SuperStack II 9300) and three Fast Ethernet switches with Gigabit uplinks (two Cisco® 4 servers 1105 950 Catalyst® 3524s and one 3Com SuperStack II) Figure 6. Number of MPEG-1 streams supported with UDP/IP SAN configuration Fibre Channel switches Two PowerVault 51Fs with four copper Gigabit Interface Converters (GBICs) and four optical and began the next iteration. During the experiment, we gradually GBICs increased the number of streams until the servers began to demon- Storage Three PowerVault 650F arrays with dual SPs strate instability or the clients began to experience jitters. 512 MB cache per SP Ten 9 GB, 10,000-rpm drives per 650F Figures 5 and 6 show the number of supported MPEG-1 streams using two different transport protocols: TCP/IP and User Datagram Protocol (UDP/IP). Four media servers working together Figure 3. Hardware and software components in test configuration supported 1,050 streams with TCP/IP and 1,105 streams with UDP/IP. The number of disk drives available limited the number same length at 3 Mbps and required approximately 1.6 GB of stor- of streams the system could support. Nonetheless, the results age space. We first determined the number of streams a single demonstrate good scalability. media server could support, and then incrementally added servers Figures 7 and 8 show the results obtained running MPEG-2 until four media servers shared the load. streams with the TCP/IP and UDP/IP protocols. MPEG-2 Each client randomly selected one of the MPEG files on the streams require bandwidth of 3 Mbps, double the bandwidth of SAN. Each emulated client ran in an infinite loop: Once the client MPEG-1 streams. A single media server supported 199 streams.2 finished running the entire MPEG file, it terminated the connection Four media servers delivered higher aggregate bandwidth 1 For configurations with a single media server, the Media Server software license limits the number of MPEG-1 streams that may be supported to 400 streams, even though the server may be capable of supporting more streams. 2 For configurations with a single media server, the Media Server software license limits the number of MPEG-2 streams that may be supported to 200, even though the server may be capable of supporting more streams. www.dell.com/powersolutions PowerSolutions 65
  5. 5. H I G H - A V A I L A B I L I T Y C L U S T E R S running MPEG-2 streams than running MPEG-1 streams—a MPEG-2 streams supported with TCP/IP total of 2.1 Gbps for 700 MPEG-2 streams versus 1.57 Gbps for 1,050 MPEG-1 streams. Two factors explain these results: The Number of video files 1 13 MPEG-2 files used larger transfer blocks, and the smaller num- 1 server2 199 199 ber of streams generated less processing overhead. 4 servers 700 555 Figure 7. Number of MPEG-2 streams supported with TCP/IP Storage system performance To understand the impact of storage system performance on video streaming, we compared the number of streams supported to the number of MPEG-1 files stored in the storage system, MPEG-2 streams supported with UDP/IP since the number of files in the storage system may influence Number of video files 1 13 performance. We first recorded the number of streams sup- 1 server2 199 199 ported with 25 MPEG-1 files in the storage system; these files 4 servers 720 560 occupied 10 percent of the available storage space. We increased the number of MPEG files until the storage system was com- Figure 8. Number of MPEG-2 streams supported with UDP/IP pletely filled with 250 MPEG-1 files. Due to the nature of random access operations, performance decreased as the storage system filled. Accessing the disk drives’ Storage system performance characteristics inner tracks caused lower data transfer rates. The entire system 1000 performed better with fewer MPEG-1 files in the storage system. Number of supported streams Storing a small number of media files close to each other in the 800 faster outer tracks of the disk drives resulted in shorter seek times 600 and higher data transfer rates. Moreover, with a small number of files, the cache memory of the storage processors boosted the media 400 servers’ performance. In production environments, frequently used 200 files could be stored on a separate storage system to maximize caching and minimize seek times. See Figure 9. 0 0 50 100 150 200 250 Number of MPEG-1 files in storage system Delivering high-quality, large-scale media streaming Although building a large-scale streaming solution with guaran- Figure 9. Storage system performance teed high-quality service is difficult, a multi-tier scale-out archi- tecture such as the one used in these tests can support a highly scalable, available, and flexible solution for media streaming. A Edward Yardumian (edward_yardumian@dell.com) is a technolo- scale-out architecture based on PowerEdge servers and PowerVault gist and systems engineer specializing in distributed systems, cluster storage offers the scalability and high availability needed for effec- computing, and Internet infrastructures in the High End Systems tive large-scale media streaming. Group at Dell. Previously, Ed was a lead product engineer for Dell PowerEdge clusters. Rizwan Ali (rizwan_ali@dell.com) is a systems engineer in the Scalable Systems Group at Dell. Rizwan has a B.S. in Electrical Engineering from the University of Minnesota. Jenwei Hsieh, Ph.D. (jenwei_hsieh@dell.com) is a member of the FOR MORE I NFORMATION High End Systems Group at Dell. He has published technical papers extensively on multimedia computing and communications, high- For more information about Streaming21, please speed networking, serial storage interfaces, and distributed network visit http://www.streaming21.com computing. Jenwei has a Ph.D. in Computer Science from the University of Minnesota. 66 PowerSolutions