Hp aspera-big data cloud-v2

1,770 views

Published on

Published in: Art & Photos, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,770
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
36
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Due to many uncontrolled parameters such as … it is a difficult task to compare the performance of different protocols. For the same protocol, its performance can vary … To exclude these uncontrolled factors, we repeat the same test many times in a relative long time and then compare the mean, worse and best scenario.We also manually verified the performance on the iPhone …
  • fasp Technology and Software Suite for Predictable, High Speed File-based TransferUnique in the world, patented transport technology providing unparalleled speed, efficiency, concurrency and bandwidth controlFully integrated, cross platform software suite for interoperable file transfer — any size, any distance, and network BWSecure standard for the industryIntegrated global management, tracking and reportingExtends to all Cloud based storageExtends to all major mobile platformsKey Business Areas Enabled with Latest Aspera SoftwareHigh Speed Content Delivery, Synchronization and Distribution (including Cloud!)Ad Hoc Content Ingest, Delivery and CollaborationIntegrated File Workflow Automation and OrchestrationMobile Platform File Delivery
  • First bottleneck's solution Transfer bulk data over WAN using Aspera fasp, overcomes TCP limitations under network latency and packet loss.Aspera solutions yield 100x performance improvements
  • Hp aspera-big data cloud-v2

    1. 1. Enabling The Big Data Cloud for HPCand Collaboration With High-Speed DataTransport
    2. 2. PRESENTER AND AGENDAPRESENTER AGENDADaniel KumiDirector, New Market Developmentdaniel@asperasoft.com • Who and Why Aspera? • WAN Transport • Wireless Transport • Cloud and Big Data – Data Transfer Challenges for HPC and Collaboration • Aspera On Demand • Aspera-HP Discussion
    3. 3. TrendsBig Data Explosion • 90% of data today file-based or unstructured • Mix of file sizes—but larger and larger files the normDiversity of IP Networks—Media, Bandwidth Rates, and Conditions • Variable bandwidth rates (slow to super-fast) • Bandwidth rates increasing—costs decreasing • Network media remains diverse (terrestrial, satellite, wireless) • Conditions vary—all networks prone to degradation over distanceData Freighting Challenges—moving Big Data over WANs • Teams are geographically dispersed • Over distance, network conditions degrade • Contemporary TCP acceleration solutions not designed for big data transfer and replicationCloud Computing Grows Up • Amazon Web Services (AWS) S3 cloud storage – 2010: 262 billion objects, 2012: 905 billion objects • More choices: Microsoft Azure, OpenStack, HP Cloud • No longer a niche – Netflix (transcoding), MTV (global video distribution), BGI (genomic sequencing)
    4. 4. ASPERA’S MISSION Creating next-generation transport technologies that move the world’s digital assets at maximum speed,regardless of file size, transfer distance and network conditions.
    5. 5. Aspera: moving the world’s digital assets at maximum speed 50% YOY growth in revenue and employees Over 12,000 licenses sold, and over 1,500 customers world wide Expanded to Asia PAC and Latin America through direct and channel Patents issued or pending in 32 countriesContinuing to innovate: fasp3™, fasp-MC™, mobile transport, cloud enablement
    6. 6. MARKETS SERVEDMedia and Entertainment, Federal Government, Life Sciences, .Healthcare, Cloud Computing, Software and Gaming,Financial Services, Legal, eDiscovery, Engineering, Technology,Telecommunications, Service Providers,Architecture and Design, Enterprise IT, Oil and Gas
    7. 7. Aspera Ecosystem of Partners
    8. 8. Aspera in Life SciencesLife Sciences
    9. 9. BIG DATA TRANSFER CHALLENGE
    10. 10. What Happened to my Bandwidth? WAN 1000 Mbps Seattle • 170ms RTT • 0.001% packet loss rate ParisWAN Throughput is 1000MbpsMax TCP Throughput ~29MbpsWhere’s my 970Mbps? At 29Mbps 50GB transfer will take 4 hrs 1TB transfer will take 3.3 days
    11. 11. BIG-DATA and WAN TRANSFER WITH TCPTCP WAS DESIGNED IN THE EARLY 80’S • When data was small & bandwidth was limited • Fantastic for reliable data delivery • Not fast enough for big-dataTCP IS THE ENGINE THAT DRIVES • FTP, HTTP & HTTPS • RSYNC, SCP & DICOM • CIFS & NFSTCP DOES NOT LIKE NETWORK LATENCY/ RTT • Geographic distance increases latency • Network congestion increases latencyTCP DOES NOT LIKE PACKET LOSS • Loss is caused by congestion • Different network capacity • Wireless and satellite communications
    12. 12. So if TCP doesn’t work, what’s the answer?The Aspera Solution
    13. 13. Same WAN Scenario with Aspera fasp WAN 1000 Mbps • 170ms RTT Seattle • 0.001% packet loss rate ParisWAN is 1000MbpsMax TCP Throughput ~29MbpsMax Aspera Throughput ~995Mbps (gain of x34)ROI measured in $$ cost of not using 971Mbps At 995 Mbps • 50GB transfer will take ~4 hrs • 1TB transfer will take 3.3 days • 50GB transfer will take ~7 mins • 1TB transfer will take 2.4 hrs
    14. 14. FASP™ — HIGH-PERFORMANCE DATA TRANSPORTMAXIMUM LINE-RATE WAN TRANSFER SPEED • Transfer performance scales with bandwidth independent of transfer distance and resilient to packet loss • Optimal end-to-end throughput efficiencyCONGESTION AVOIDANCE AND POLICY CONTROL • Automatic, full utilization of available bandwidth • On-the-fly prioritization and bandwidth allocationUNCOMPROMISING SECURITY AND RELIABILITY • Secure, user/endpoint authentication • AES-128 cryptography in transit & at-restSCALABLE MANAGEMENT, MONITORING AND CONTROL • Real-time progress, performance and bandwidth utilization • Detailed transfer history, logging, and manifestENTERPRISE-CLASS FILE DELIVERY • Transfers up to thousands of times faster than FTP/HTTP(S) • Precise and predictable transfer times • Extreme scalability (concurrency and throughput)
    15. 15. FASP vs TCP PERFORMANCEfasp Bandwidth ROI FTP: Limited by Distance & Packet Loss, Not B/W FTP Across US US – EU US – ASIA Satellite 1 GB 1 – 2 hrs 2 – 4 hrs 4 – 20 hrs 8 – 20 hrs 10 GB 15 – 20 hrs 20 – 40 hrs Impractical Impractical 100 GB Impractical Impractical Impractical Impractical Aspera: Scales Linearly with Bandwidth fasp™ 2 Mbps 10 Mbps 45 Mbps 100 Mbps 200 Mbps 1 Gbps 1 GB 70 min. 14 min. 3.2 min. 1.4 min. 42 sec. 8.4 sec. 10 GB 11.7 hrs 140 min. 32 min. 14 min. 7 min. 1.4 min. 100 GB 23.3 hrs 5.3 hrs 2.3 hrs 1.2 hrs 14 min. Distance & Packet Loss Independent
    16. 16. 6 Gbps Scalable WAN Throughput~6Gbps Big-Data Throughput x3000 improvement vs. TCP • Latency independent • 1TB data moved in 20 min • Loss independent • 2 days with TCP over LAN conditions Scale to ~10Gbps with IQ Accelerator
    17. 17. Aspera mobile app SUITE New FEATURES AND CAPABILITIESTechnology Advantages • Up to 3x faster than 3G, up to 100x faster than 802.11n • Complete integration into native device environmentiOS faspex™ App for iPhone and iPad • High-speed faspex mobile transfers • Native iOS app look and feel, and Photo Gallery integration • Familiar faspex™ workflow with visual style of iOS email app • Supports faspex™ security models (e.g. encryption at rest)Android Client App • High-speed transfers to/from Enterprise or Connect Servers • Upload content from camera, photo gallery, or file system • Browse Connect Server’s listings and download to the device • Upload geo-tagged content using device’s positionApplication Uses • Remote news collection, upload and publishing to central hub • Viewing and approving content as part of a file based workflow • Remote medical image viewing and diagnosis • Use fasp-AIR SDK to embed high-speed transfers into your mobile apps
    18. 18. fasp-AIR Benchmarks on Verizon 4GIn some cases (highlighted in orange), speeds will varygreatly, depending on available bandwidth and the underlyingcondition of the wireless network.
    19. 19. Aspera Software Product & Technology Portfolio DISTRIBUTE COLLABORATE AUTOMATE Complete portfolio of Global person-to-person Web-based application servers and clients for and project-based and SDK for creating andhigh-speed data delivery exchange and managing automated file- and distribution. collaboration of files based workflows. and directories. TRANSPORT Our unique, patented faspTM transport technologies provide unparalleled speed, efficiency, and bandwidth control over any file size, transfer distance, network condition and storage location (on- premise or cloud).
    20. 20. fasp™ Software Environment
    21. 21. ASPERA DEVELOPER NETWORKA complete set of SDKs provides developers with guides, reference information, and sample code to assist them withintegrating Aspera technology into their own applications. Aspera fasp™ technology can be used in desktop, network-based, and web applications in place of FTP, HTTP, or custom TCP-based copy protocols. ASPERA TRANSFER APIs ASPERA MOBILE APIs Aspera Web Services Android SDK A SOAP based web service API that allows initiation, Aspera Android SDK provides a Java API to transfer files using monitoring and controlling of fasp based file transfers. fasp-AIR™. iPhone SDK Aspera Web Aspera iPhone SDK provides an Objective C API to transfer files Javascript API exposed by Aspera Connect client. It allows using fasp-AIR. integration of fasp based file transfers into web applications. ASPERA APPLICATION APIs Connect 2.8 developer Preview 2 Introducing the new Connect 2.8 developer preview! faspex™ Web API Integrate the functionality of Aspera Connect 2.8, a fasp- The Aspera faspex Web API provides a set of services that enables based file transfer client, into your own web applications, users to create and receive digital deliveries via a Web interface, while while customizing it to your unique brand. taking advantage of fasp high-speed transfer technology. fasp Manager A class library that allows intiations, monitoring and OTHER INFORMATION controlling of fasp based file transfers. Supporting Tools and Libraries Aspera Multicast SDK Supporting tools and libraries let you perform other common tasks A Java class library that allows initiation and management surrounding file transfers. of IP multicast based data transmissions using Aspera General Reference fasp-MC™. Reference on error codes, log file locations, configuration files and more.
    22. 22. CLOUD COMPUTING & BIG DATA
    23. 23. Cloud Computing – Why is it so compelling?The potential of infinite computing resources, on demand • Eliminates the need to plan ahead • Allows companies to meet demand • Without the lead-time bottleneckThe elimination of an up-front commitment • Reduce capital outlay and investment risk • Start small & increase h/w resources to match need • Auto-scale to meet demandPay-for-use resource model • CPU’s by the hour • Storage by the day • Bandwidth by the GB Cloud
    24. 24. SO? WHAT CAN I DO WITH IT? • Compute Intensive: 10’s, 100’s, 1000’s of CPU cores DATA PROCESSING& CONTENT CREATION • Transcoding, rendering, encoding, watermarking • Big-data analytics & HPC • Near-line for editing, creative apps and processing STORAGE FOR ARCHIVE & D/R • B2B / B2C data workflow • Offsite storage for disaster recovery and business continuity • OTT, play out, release, project & event specific marketing DATA & CONTENT DISTRIBUTION • Collaborative data exchange • CDN and global delivery
    25. 25. GETTING BIG DATA IN AND OUT OF THECLOUD KNOWING WHEN TO CHOSE THE RIGHT TOOL
    26. 26. CHALLENGES OF STORING BIG FILES IN THE CLOUD?BEWARE THE OBJECT STORE: • Not like traditional NAS or SAN • Bigger, better, but possibly much more complex • a.k.a. Google File System, Amazon S3, Hadoop Distributed File System • Simple read/write of data ―blobs‖, indexed by a key • Multiple replicas are distributed across storage for durability and optimized for access • Should work well for storing large numbers of filesUNDERSTAND CHUNKS, BLOCKS and BLOBS • You need to deal with chunks, blocks and blobs • ―Chunk‖ sizes are small (64 MB/128 MB) • Large media files must be ―chunked‖ (1TB file = transporting and reassembling 10,000+ chunks!) • Multi-chunk APIs impede workflow and are complex • Data I/O use the standard HTTP(s) protocol • VERY SLOW at distance • Single HTTP stream slow even locally (<100 Mbps).BIG-DATA SERVICES WILL NEED A HIGH-SPEED BRIDGE TO THE CLOUD • Large files moved at full bandwidth capacity with global access • Overcome the WAN and storage bottleneck • Support files of any size or quantity • Transparent to the end user/data owner (GUI, command line, API, browser, etc.) • No hardware to support B2B, B2C, C2B workflow
    27. 27. Big data cloud storage challenge Cloud
    28. 28. S3 & BIG-DATA: TYPICAL APPLICATION OPTIONS
    29. 29. SOLUTION: ASPERA ON-DEMAND DIRECT-TO-S3 Cloud
    30. 30. OVERCOMING BOTH BOTTLENECKS #1 — TRANSFER DATA TO EC2 OVER WAN EFFECTIVE THROUGHPUT• http transfer over WAN (single stream) <10 Mbps• Typical internet conditions • 50–250ms latency & 0.1–3% packet loss <10 to 100 Mbps• 15 parallel http streams• Aspera fasp transfer over WAN to EC2 up to 1Gbps (per EC2 Extra Large Instance) #2 — TRANSFER DATA FROM EC2 TO S3 EFFECTIVE THROUGHPUT• Standard single stream http 10 to 100 Mbps• Aspera S3 Proxy up to 1Gbps • With parallel I/O http streams (per EC2 Extra Large Instance) ASPERA + AWS | ~10 TB transferred per 24 hours | PER EC2 INSTANCE
    31. 31. ASPERA DIRECT-TO-S3 — LINE RATE ACCESS TO THE CLOUDUNRIVALED ASPERA PERFORMANCE • Built on Aspera fasp™ technology for maximum transfer speed • Regardless of file size, transfer distance and network conditions • Precise bandwidth control ensures the available bandwidth is utilized to achieve maximum transfer speeds, while being fair to other business-critical network trafficSEAMLESS INTEGRATION WITH S3 • Integrated with S3 multi-part HTTP for maximum ―last foot‖ performance • Simple configuration of S3 credentials, for both shared and dedicated docroot • Transfers directly into S3 are seamless and transparent to userENTERPRISE-GRADE SECURITY AND RELIABILITY • Secure authentication with encryption in transit & at rest (AES-128, FIPS 140-2, HIPPA Compliant) • Packet-level data integrity verification • Automatic resume of partial or failed transfers • Full support for AWS S3 Service-side-encryption at restINTEROPERATES WITH ALL ASPERA HOST OPTIONS • Any platform (Windows, Linux, MAC, UNIX, iOS, Android) • Any Aspera Clients (CLI, Desktop, Point-to-Point, Mobile, Web, Embedded) • Any Aspera Servers (Enterprise, Connect, faspex)
    32. 32. ASPERA FOR AWS: DIRECT-TO-S3 : SCALE OUT ARCHITECURE FOR ULTRA HIGH THROUGHPUT1. Upload using typical multi-part HTTP client Scale out HTTP –2. fasp high-speed upload Direct-to-S3 multipart 2 fasp Aspera Transfer Server Aspera Herndon, VA Client 1 HTTP – multipart Client, Dallas, TX
    33. 33. ASPERA SOFTWARE ON DEMAND Aspera Server Aspera faspex Aspera Shares Aspera Console Universal file transfer server Global Person-to-person file Global Person-to-person file Global transfer supports desktop, web, mobile & ingest & distribution transfer & exchange monitoring, reporting & control embeddedKEY FEATURES • On demand high-performance data transport to and from remote infrastructures • Unlimited scale out of transfer capacity with additional AMIs • Support for all Aspera Server software and use cases • Additional Client Options: Mobile, Outlook Plug-in & Cargo (Aspera faspex) • Flexible Storage Options: Local, EBS, AWS S3 • Seamlessly interoperates with on-premise Aspera deployments • Integrated Management and MonitoringAPPLICATIONS AND USE CASE • High Performance Computing On Demand • Content Aggregation, Transformation and Distribution • Time-boxed event or project-based collaboration, ad-hoc distribution or content ingest
    34. 34. ASPERA shares 1.0 MULTI-NODE FILE SHARINGGlobal Multi-node, High-speed File Sharing • Easy-to-use web application with secure access to a consolidated view of all shared content • Fully integrated Aspera Connect browser plug-in for high-speed, secure file transfers • Access to content distributed across any locations (enterprise servers, private, public and hybrid clouds) • Extremely scalable with web application being decoupled from storage nodes • Powerful security model administered through a single management point combining authorization, user management, and access control • Search, filtering, and sorting capabilities make it easy to find individual files or folders in a very large content storeApplication Examples • Internal distribution of digital assets within the enterprise • Collaboration enablement for geographically-distributed teams • Project team and third-party content or data gathering • Global browser-based distribution of data across external teams and partners
    35. 35. Aspera Shares Features and BenefitsEasy to use Web interface • Users can easily navigate across files and folders to locate and initiate a high-speed file upload or download • A single view consolidates all shared content making it easy to search and navigate across multiple nodes • Powerful search, filtering, and sorting capabilities simplify finding individual files or folders in a very large content store • Operations can be performed on a single file or folder, or directly on the search results Powerful security and access model • Secure access administered through a single management point combining authorization, user management, and access control • Add and manage users and groups locally, as well as through Active Directory and LDAP • Complete control over access, such as which nodes are visible, and which nodes, directories and files are accessible • Granular control over all end-user operations such as browsing, uploading, downloading, making new directories, renaming or deleting files and directories
    36. 36. HYBRID CLOUD DEPLOYMENT (PUBLIC/PRIVATE)Shares app transparently communicates with Aspera server Nodes in cloud and in enterpriseUser browses content across authorized sharesHigh-speed data transfers with DatacenterHigh-speed data transfers with Direct-to-S3 Client, NY, NY fasp Shares fasp DMZ Node Node Herndon, VA Datacenter, Emeryville, CA
    37. 37. Aspera faspex™ OverviewGlobal Person-to-Person File Delivery Person-to-person File Exchange and Collaboration • Built on Aspera Enterprise Server technology • High-performance fasp™ transport • Person-to-person and project-based file-exchange • Easy to use Web-based interface, email, mobile and desktop client interfaces • Easily create and manage work groups for file-based collaboration • Enterprise-scale user management and access controlApplication examples • Internal distribution of digital assets within the enterprise • Collaboration enablement for geographically-distributed teams • File-based review, approval and quality assurance workflows • Digital delivery and collaborative file transfer with external Aspera faspex™ partners
    38. 38. Aspera faspex™ key features and benefitsUnrivaled Aspera performance • Maximum transfer speed regardless of file size, transfer distance and network conditions • Secure authentication, full data encryption in transit and at rest • 100% reliable data deliveryGlobal File Exchange and Collaboration • Custom Metadata Collection with integrated reporting • Comprehensive and granular package access and expiration policies • Integrated email notification • API and automation capabilities for file-based process /workflow integrationEnterprise-scale solution • Comprehensive administration, user management & access control • Highly concurrent user connection architecture • Remotely monitored and controlled via Aspera Console – Aspera’s centralized web-based transfer management application.
    39. 39. Comprehensive SecurityFull support of the Aspera fasp™ security model • Secure authentication, encryption of the data using strong cryptography, per-packet integrity verification to protect against man-in-the-middle compromise, and is FIPS-140 2 compliantSupports all LDAP directory services for import, synchronization, and direct authentication • Open Directory, Open LDAP, Active DirectoryConfigurable security options • Automatic deactivation after set number of failed login attempts • Concurrent login prevention, session timeout, strong passwords • Configurable per IP and mask upload, download, login permission • Sending and receiving from 3rd party non registered users, with policy-based expirationConfigurable data encryption in flight and at rest • Package contents can be encrypted over-the-wire, or optionally at rest for complete security • Encrypted packages can be decrypted on the fly when downloaded by the recipient using one of the faspex™ clients such as the iOS faspex™ Client, Aspera Add-in for Outlook, Aspera Connect browser plug-in, or Aspera Cargo downloader • The sender may choose when and to whom to distribute the secret, and thus prevent unauthorized users from accessing the content, and control when recipients are able to decrypt the downloaded content
    40. 40. ASPERA faspex 3.0 NEW FEATURES AND CAPABILITIES Person-to-person File Exchange and CollaborationRemote File Sources (local, enterprise, cloud) • Browse and publish from any storage location • RESTful API for integration and full process automationAspera On Demand • Available on demand via AWS subscription • Seamless integration with Amazon S3 • Supports all faspex client and mobile optionsSupport for Active / Active and Tiered Architecture • Active-active high availability configuration with shared storage (on Linux) • Transfer server and faspex web application can be deployed on separate hosts for maximum scale outNew Client Applications • Faspex app for iOS support mobile use cases • MS Outlook add-in allows transfers directly from within Outlook with transparent uploads/downloads via faspexSelf-registration • Enables rapid addition of third parties with both moderated and un-moderated configuration options
    41. 41. Enterprise Performance, Scalability and ReliabilityUnlimited scale-out • faspex™ application and Enterprise Server can be installed on separate hosts • Allows clustering of Enterprise Server nodes for unlimited scale-out of transfer capacity (I/O and network capacity) Multi-server relay • Master server and Relay server configuration • Transfers from one user to another are relayed to the users’ home server • Simple administration of relay server through the master with automatic synchronization of user accounts between master and peersHigh-availability configuration • faspex™ server is designed to be deployed in an active / passive HA configuration. Now supports Active/Active • Ensures continuous availability of the faspex™ application • Seamless automatic retry and resume of transfer sessions on failover assuming shared storage
    42. 42. Faspex On demand with Aspera direct-to-S31. fasp high-speed upload Direct-to-S32. faspex notification3. fasp high-speed down load Direct-from-S3 faspex fasp fasp Person-to- Person 3 HTTP – multipart 2 1 Connect Connect Browser Browser Plug-in Plug-in Herndon, VA Client, NY, NY Client, NY, NY
    43. 43. BIG-DATA: ACCESSED & DELIVERED BY ASPERA File transfer
    44. 44. THANK YOU!Daniel KumiDirector, New Market Developmentdaniel@asperasoft.com

    ×