Aspera on demand for AWS (S3 inc) overview


Published on

Aspera OnDemand for S3 - company intro and AWS specific solution overview

Published in: Technology
1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • That’s why organizations grappling with the storage, management and distribution of unstructured data and other digital content such as the examples shown on this slide chose Isilon IQ.
  • Note: Because of their size and data structure, compression of unstructured data files isn’t often feasible**compressed media files are estimated, encoding and codec variants will create different file sizes
  • Substantiating speed claims
  • First bottleneck's solutionTransfer bulk data over WAN using Aspera fasp,overcomes TCP limitations under network latency and packet loss.Aspera solutions yield 100x performance improvements
  • Fasp part 2 – security & reliability
  • 449 Billion objects and counting1,440 objects for every resident of the US (source: US Population Clock).64 objects for each person on Planet Earth (source: World Population Clock).4 objects for every neuron in your brain (source: Wikipedia / Neuron). About as many S3 objects as there are stars in the Milky Way (source: Wikipedia / Milky Way).
  • First bottleneck's solutionTransfer bulk data over WAN using Aspera fasp,overcomes TCP limitations under network latency and packet loss.Aspera solutions yield 100x performance improvements
  • Aspera on demand for AWS (S3 inc) overview

    1. 1. Aspera OnDemand for AWSS3|DirectBhavik Vyas - Director of Cloud Services & PartnershipsV3.feb2012
    2. 2. Aspera Corporate Overview• Headquarters: Emeryville, CA• Founded: 2004• Ownership: Privately Held• Employees: 95• Patents: Issued and pending in over 32 countries• Customers: over 1,200 customers, over 10,000 licenses• Markets Served: Media and Entertainment, Federal Government, Life Sciences, Healthcare, Cloud Computing, Software and Gaming, Financial Services, Legal, eDiscovery, Engineering, Technology, Telecommunications, Service Providers, Architecture and Design, Enterprise IT• Creators of the fasp™ protocol – Innovative, patented, highly efficient bulk data transport technology – Unique and core to Aspera’s high-performance file transfer software suite – Outperforms software and hardware WAN acceleration solutions – Ranked first in every WAN transfer throughput benchmark Creating next-generation transport technologies that move the world’s digital assets at maximum speed, regardless of file size, transfer distance and network conditions.
    3. 3. What Constitutes “Big Data” at Aspera?
    4. 4. How Big is “Big Data”? (M&E example) A single digital cinema production can be 800K–1M 2K/4K frames
    5. 5. Data Transport Challenges with TCP and alternative technologies• Distance degrades conditions on all networks – Latency (or Round Trip Times) increase – Packet losses increase – Fast networks just as prone to degradation• TCP performance degrades with distance – Throughput bottleneck becomes more severe with increased latency and packet loss• TCP does not scale with bandwidth – TCP designed for low bandwidth – Adding more bandwidth does not improve throughput• Alternative Technologies – TCP-based - Network latency and packet loss must be low TCP powers: – Modified TCP - Improves on TCP performance but insufficient for fast networks • FTP – UDP traffic blasters - Inefficient and waste bandwidth • SCP – Data caching - Inappropriate for many large file transfer workflows • RSYNC • HTTP(s) – Data compression - Time consuming and impractical for certain file types • CIFS – CDNs & co-lo build outs - High overhead and expensive to scale • NFS
    6. 6. What Happened to my Bandwidth? WAN 1000 Mbps – 170ms RTT Seattle – 0.001% packet loss rate ParisWAN Throughput is 1000MbpsMax TCP Throughput ~29MbpsWhere’s my 971Mbps? At 29Mbps: 50GB transfers will take 4 hrs 1TB transfers will take 3.3 days
    7. 7. Aspera’s Solution: high-performance data transport (fasp™)• Maximum line-rate WAN transfer speed – Optimal end-to-end throughput efficiency – Transfer performance scales with bandwidth independent of transfer distance and resilient to packet loss• Congestion Avoidance and Policy Control – Automatic, full utilization of available bandwidth – On-the-fly prioritization and bandwidth allocation• Uncompromising security and reliability – Secure, user/endpoint authentication – AES-128 cryptography in transit & at-rest• Scalable management, monitoring and control – Real-time progress, performance and bandwidth utilization – Detailed transfer history, logging, and manifest• Enterprise-Class File Delivery – Transfers up to thousands of times faster than FTP – Precise and predictable transfer times – Extreme scalability (concurrency and throughput)
    8. 8. Same WAN Scenario with Aspera WAN 1000 Mbps – 170ms RTT – 0.001% packet loss rate Seattle Paris TCP Aspera faspWAN is 1000Mbps• Max TCP Throughput ~29Mbps• Max Aspera Throughput ~995Mbps (gain of x34)• ROI measured in $$ cost of not using 710Mbps At 995 Mbps – 50GB transfer will take ~4 hrs – 1TB transfer will take 3.3 days – 50GB transfer will take ~7 mins – 1TB transfer will take 2.4 hrs
    9. 9. fasp TM – performance breakthroughFTP Across US US – EU US – ASIA Satellite1 GB 1 – 2 hrs 2 – 4 hrs 4 – 20 hrs 8 – 20 hrs10 GB 15 – 20 hrs 20 – 40 hrs Impractical Impractical100 GB Impractical Impractical Impractical Impractical TCP transfer times limited by packet loss, delay (network distance) NOT BWfasp™ 2 Mbps 10 Mbps 45 Mbps 100 Mbps 200 Mbps 1 Gbps1 GB 70 min. 14 min. 3.2 min. 1.4 min. 42 sec. 8.4 sec.10 GB 11.7 hrs 140 min. 32 min. 14 min. 7 min. 1.4 min.100 GB 23.3 hrs 5.3 hrs 2.3 hrs 1.2 hrs 14 min. Aspera transfer times shorten linearly with bandwidth Independent of packet loss, delay (network distance) Cross US – Add 1to 5% Intercontinental – Add 1 to 10% Satellite – Add 1 to 10%
    10. 10. Aspera software product & technology portfolio Distribute Collaborate Automate Complete portfolio of servers and end point Global person-to-person and project-based Web-based application and SDK for creating and clients for high-speed digital content delivery and exchange and collaboration of files and directories, managing automated workflows, from simple file distribution. of any size, over any distance, over any network. forwarding, to complex process orchestration. Enterprise and Connect Server faspex Server Orchestrator • Universal file transfer server and web-based • Secure digital delivery and collaborative file • Intuitive graphical workflow designer interface and directory listing transfers with remote users and partners • File processing decision tree and flow Client and Point-to-point • Integrated e-mail notifications for delivery and • Rich and flexible plug-in architecture for third- • Uni- and bi-directional transfer clients successful download party process integration • Comprehensive administration, user • Comprehensive library of plug-ins for Connect management & access control transcoding, virus checking, quality checking, • Web browser plug-in for high-speed uploads faspex Multi-Server / HA archive, notifications and downloads • Automated bi-directional relays between sites • High volume processing Mobile • Detailed dashboard, workflow, and step-level and multiple servers • High-speed transfer for mobile devices progress reporting. • 3-tier architecture with support for clustering and Sync high availability • Open development framework for designing • Highly scalable, multidirectional file replication and integrating highly processing and Cargo and synchronization automation pipelines • Automated client downloads TransportOur unique, patented transport technologies provide unparalleled speed, efficiency, concurrency and bandwidth control over any size, distance, and networkfasp™ fasp3™ Aspera On-Demand S3|DirectPatented, file-based bulk data transport Next-gen protocol for any bulk data High-speed transfer direct to cloud storage (S3)fasp-AIR™ fasp-MC™ Console transport managementUploads and downloads over 3G, LTE and Wi-Fi networks High-speed delivery over multicast Centralized web-based management, monitoring, and reporting
    11. 11. fasp: Patented Bulk-Data Transport Options faspTM • Aspera’s original, unique, patented transport for moving bulk data at maximum speed • Independent of file size, distance, or network type • Independent of network delay and robust to extreme packet loss • Built in adaptive rate control achieves extraordinary quality of service and predictable delivery times • Complete security and data integrity verification fasp3TM • Aspera’s 3rd generation patented transport for moving any bulk data, not just files fasp-AIRTM • Designed and optimized for high-speed uploads and downloads over 3G, LTE and Wi-Fi networks. • Supports Android and iOS devices • SDK allows app developers to integrate superior transport capabilities into their own applications fasp-MCTM • Enables high-speed transport over IP Multicast • Shortest end-to-end distribution time • Eliminates any pre- and post-transmission delays • Unrivaled repair efficiency and network utilization
    12. 12. Aspera Console: File Management & Reporting• Centralized network-wide management and transfer monitoring• Track and manage all Aspera transfers throughout the enterprise• Remote configuration all managed Aspera Server nodes• Create and manage users & user groups• Initiate and manage transfer• Precise control of transfer & bandwidth utilization parameters• Comprehensive dashboard reporting: • Real-time transfers and bandwidth utilization • Transfer event notifications • Full transfer logging • Extensive custom reporting capabilities
    13. 13. Aspera DistributeComplete portfolio of servers and end point clients for high-speed digital content delivery and distribution. Client and Point-to-Point Enterprise Server • High-speed faspTM desktop transfer clients • High-speed faspTM transport of large data sets • Easy to use drag and drop user interface and built in CLI • Enterprise-wide high-volume content ingest and distribution • Transfer initiation and automated queuing and scheduling • Comprehensive automation and scheduling • Advanced bandwidth utilization management • Advanced user and server management • Graphically-adjustable transfer speed and priority • Complete monitoring and reporting • Built in email notification and delivery confirmation • High-availability configuration (active/active or active/passive) Connect Connect Server • Install-on-demand Web browser plug-in • Incorporates all the features of the Enterprise Server • High-performance fasp™ transfers directly from browsers • Web portal for high-speed upload / download of files, • Built-in transfer monitor for visual, on-the-fly transfer rate control directories, and multi-item transfers and monitoring • Rich set of APIs, can be integrated / embedded in 3 rd party • HTTP fallback mode for highly-restrictive network environments web applications • Complete Javascript API for embedding in custom Web portals • Supports Aspera Connect browser plug-in with automatic download and installation Mobile Sync • High-speed fasp-AIR™ uploads from an iOS device over • High speed faspTM synchronization of remote files and wireless networks, including cellular and Wi-Fi directories • Easy-to-use interface that directly accesses the picture and • Highly scalable – supports today’s extremely large data sets video library, as well as the iPhone built-in camera • One-to-one, one-to-many, and full-mesh synchronization • Real-time transfer progress and performance display • One time, scheduled, or continuous in both push and pull mode
    14. 14. Aspera Collaborate Global person-to-person and project-based exchange and collaboration of files and directories, of any size, over any distance, over any network. faspexTM Server Cargo • High-performance fasp™ transport • Automatic downloads of faspexTM packages from Aspera • Person-to-person and project-based file-exchange faspexTM servers • Easy-to-use web-based interface • Supports secure embargoes on content access with built-in • Integrated email notification encryption • Advanced policy-based encryption • Easy GUI controls for automatic decryption of all content in • Drop Boxes for ad-hoc submission and distribution a downloaded package • Fully configurable custom Meta data support • Configurable download target location per faspex server • Supports Cargo for automatic package download and and account distribution • Support for a configurable number of concurrent downloads • Enterprise-scale user management and access control and queuing • Unlimited scale-out with multi-server relay, clustering, and • Real-time transfer rate control and monitoring Pause, cancel HA support and resume functionality with automatic retry • Supports all LDAP directory services for import, synchronization and direct authentication • Broad Platform support Outlook Plugin (Coming soon) • High-speed transfer of large file attachments seamlessly faspexTM Multi-Server / HA from within Outlook • Automated bi-directional relays between sites and multiple • Leverages the strength of the Aspera fasp™ transport servers protocol and existing faspex™ software and infrastructure • 3-tier architecture with support for clustering and high availability
    15. 15. Aspera Design: Developer Network & SDKA complete set of SDKs provides developers with guides, reference information, and sample code to assist them with integrating Asperatechnology into their own applications. Aspera fasp™ technology can be used in desktop, network-based, and web applications in placeof FTP, HTTP, or custom TCP-based copy protocols.Aspera Transfer APIs Aspera Mobile APIs Aspera Web Services Android SDK A SOAP based web service API that allows initiation, Aspera Android SDK provides a Java API to transfer files using monitoring and controlling of fasp based file transfers. fasp-AIR™. Aspera Web iPhone SDK Javascript API exposed by Aspera Connect client. It allows Aspera iPhone SDK provides an Objective C API to transfer files integration of fasp based file transfers into web using fasp-AIR. applications. fasp Manager A class library that allows intiations, monitoring and Aspera Application APIs controlling of fasp based file transfers. faspex™ Web API Aspera Multicast SDK The Aspera faspex Web API provides a set of services that enables A Java class library that allows initiation and management users to create and receive digital deliveries via a Web interface, while of IP multicast based data transmissions using Aspera taking advantage of fasp high-speed transfer technology fasp-MC™. Other Information Supporting Tools and Libraries General Reference Supporting tools and libraries let you perform other common Reference on error codes, log file locations, configuration files and tasks surrounding file transfers. more.
    16. 16. Aspera OnDemand for AWS
    17. 17. Why AWS for Media Production & Distribution?• Content Creation – Compute Intensive: EC2 (10’s, 100’s, 1000’s of CPUs) • Transcoding, encoding, watermarking, video editing • Rendering & HPC applications• Mission Critical Storage and Distribution – Long-term archive & backup – Near-line storage for compute – B2B/B2C media ingest & distribution• Monetization & Play Out – Release, project and event specific marketing & social media – Brand awareness & franchise continuity – CDN and Delivery • AWS Cloud Front
    18. 18. Big-Data Cloud Transfer BottlenecksCharacterizing & UnderstandingObject Storage for Big-Data
    19. 19. Two Major Bottlenecks: WAN Transfer & Local HTTP I/O 1st Bottleneck - WAN 2nd Bottleneck – Data Center Transfers over the WAN are TCP “Last-foot” local transfers from EC2 to based (FTP, SCP, HTTP etc) S3 can use multiple HTTP connections Server S3 (EC2) WAN
    20. 20. 1st Bottleneck: WAN Transfers#1 WAN Transfer: Local machine to EC2 Effective throughput• Single HTTP transfer• Typical internet conditions 0.5 to 5 Mbps  50-250ms latency & 0.1- 3% packet loss• 15 parallel http streams 7 to 75 Mbps• Aspera fasp transfer Up to 700 Mbps WAN
    21. 21. fasp – Secure, Reliable & Line-Rate WAN Speed• Secure user/endpoint authentication• AES-128 encryption in transit & at rest• Packet-level data integrity verification• Automatic resume of partial or failed transfers• Any storage i/o (NAS, SAN, DAS, Object, etc)• Any platform (Windows, Linux, MAC, UNIX, iOS, Android)• Any method (CLI, Desktop, Mobile, Web, Embedded)
    22. 22. 2nd Bottleneck: Data Center/ Local HTTP I/O 2nd Bottleneck – Data Center “Last-foot” local transfers from Server (EC2) to Storage (S3) using one/many HTTP connections Server S3 (EC2)
    23. 23. Some History - AWS S3: 762 Billion objects and countingNow 762 Billion.• This represents year-over-year growth of 192%• At 449 Billion objects (in July 2011) this was: – 1,440 objects for every resident of the US – 64 objects for each person on Planet Earth – ~ the number of stars in the Milky Way
    24. 24. 2006-2009 Big-Data & AWSMarch 2006: Computer Music / Audio Modeling – AWS Launches the S3 object storage systemJune 2009: – AWS announces physical import/export for AWSSeptember 2009: – Aspera launches Aspera OnDemand for AWS Video & Photos & Graphics Imaging Genetic Sequencing PDFs
    25. 25. Sept 2009: Aspera Launches On-Demand for AWS• On-Demand: powered by Aspera’s patented fasp™ technology – Next-generation transport protocol for digital media – Eliminates the latency & packet loss bottlenecks of TCP – Reliable & secure asset delivery system – Replaces FTP, HTTP, NFS, CIFS, tape and disks – Seamless integration w/ all Aspera Clients and Console Management• Aspera On-Demand lowered the network barrier to AWS adoption: – Solved Bottleneck #1 – But didn’t transfer data directly to/from S3 – Bottleneck #2
    26. 26. Aspera On-Demand Version 1 – No S3 Support… Aspera On-Demand v.1 did not read/write data to S3 faspAspera Connect Aspera S3 EC2Browser Plugin On-Demand Server EC2Aspera Client WAN HTTP(s) I/O only – Object Store - Key/Value Pair Global availability – 11 9’s durabilityAspera Mobile Local HD Not persistent Elastic Block Store – EBS (NAS) Regional only – Limited to 1TB blocks – single mount
    27. 27. 2011: Big-Data REALLY Meets AWS…Dec 2010: AWS Announces Major S3 Upgrade Computer Music / Audio Modeling – S3 object size increased • 5GB to 5TB • AWS introduces multipart HTTP uploader • API’s available in Java, .NET, PHP & RESTFantastic… but now what? Video & Photos & Graphics Imaging • Still HTTP over the WAN (SLOW) • Still have to “glue” any fasp high speed transfer to S3 I/O in custom s/w – big speed bump! • Find an expert s/w team Genetic • Build upon the multipart API Sequencing PDFs • Concurrently stream data to S3 • Integrate into operations
    28. 28. S3 & Big-Data: API/ Commercial Application Options Multi-Part Uploader API (from AWS) Commercial ToolsStep Action 1 Initiate multipart upload by providing your AWS credentials 2 Provide required bucket name and key name 3 Save the upload ID for each subsequent multipart upload operation Upload parts providing part upload information (upload ID, bucket 4 name, part number) 5 Save the responses (ETag value and the part number) 6 Repeat tasks 4 and 5 for each part of your object 7 Execute a final call to complete the multipart upload S3 Multi-part upload HTTP !"#
    29. 29. What are the Challenges of Storing Big Files in the Cloud?• Designed as scalable distributed object stores – Target applications require simple read/write operations of binary "blobs”, indexed by a single primary key – Should work well for storing large numbers of media files, compared to traditional file systems• BUT – “Blob" sizes are small (<64 MB) => large media files must be “chunked” – Data I/O use the standard HTTP protocol – VERY SLOW at distance – API for managing data requires a team of experts• M&E/ Big-Data services require high-speed software bridge over the WAN – Large files to be moved at full bandwidth capacity w/ global access – Must overcome the WAN and the I/O bottleneck – Must allow for writing media files of any size – Must be transparent to the end user uploading / downloading (GUI, command line, browser, etc.)
    30. 30. Solving the Big-Data i/o at ScaleWith the help of AWS, Aspera did a full characterization of AWS S3 i/o:• Upload/Download performance vs. thread count• Upload/Download performance vs. chunk size• 24hr upload stability w/ fixed thread size• 24hr download stability w/ fixed chunk• Upload/Download performance vs. duration• DNS lookup performance• Performance w/ concurrent access to single S3 bucket• Performance w/ max connections per host
    31. 31. The Result? Aspera On-Demand S3-Direct HTTP S3 Parts API Aspera Connect Browser Plugin S3 S3Aspera Server/ Client Optimized S3 i/o Aspera On-Demand Aspera Mobile Server fasp S3 Server RAM WAN fasp-S3 Gateway EC2Aspera On-Demand S3-direct:• Full client-side r/w of S3• Synchronous transfer from Client to S3 (via EC2 Aspera On-Demand)• Real-time optimization of HTTP threads• Real-time optimization of chunk size Aspera fasp data session
    32. 32. Overcoming Both Bottlenecks - Transferring Data to S3 over WAN#1 - Transfer Data to EC2 over WAN Effective throughput• http transfer over WAN (single stream)• Typical internet conditions 0.5 to 5 Mbps • 50-250ms latency & 0.1- 3% packet loss• 15 parallel http streams 7.5 to 100 Mbps• Aspera fasp transfer over WAN to EC2 up to 700 Mbps#2 - Transfer Data from EC2 to S3 Effective throughput• Standard single stream http 20 to 100 Mbps• Aspera S3 Proxy up to 700 Mbps • With parallel I/O http streams fasp™ 45 Mbps 100 Mbps 200 Mbps 1 Gbps 5 Gbps 10Gbps 1 GB 3.2 min 1.4 min 42 sec 8.4 sec 1.6 sec 0,8 sec 10 GB 32 min 14 min 7 min 1.4 min 16 mins 8.2 sec 100 GB 5.3 hrs 2.3 hrs 1.2 hrs 14 min 2.7 mins 82 sec 1TB 2.1 days 23 hrs 11.7 hrs 2.3 hrs 28 mins 14 mins
    33. 33. Aspera On-Demand S3-direct EC2 S3Aspera Connect Aspera Client Aspera Mobile Aspera ConsoleBrowser Plugin &/or Server (Command/ Control/ Report) Aspera On- RD Deman Aspera Aspera d On- On- Server Demand S Demand Aspera Server EC2 HTTP Server EC2 On-Demand Server EC2 (FTP/ HTTP fasp replacement) S3 S3 fasp Persistent Data WAN Faspex Local On-Demand HD (Collaboration) Elastic Block Store – EBS (NAS) EBS RDS – MySQL Relational Database Service
    34. 34. Big-Data: Accessed & Delivered by Aspera
    35. 35. For more information