Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

10Gbps transfers

544 views

Published on

An overview of why TCP doesn't work in high speed networks, while providing an overview of FileCatalyst Direct and how it goes about providing 10Gbps transfers

Published in: Software
  • Be the first to comment

  • Be the first to like this

10Gbps transfers

  1. 1. May Webinar: 10Gbps transfers Presenter: Christian Charette, John Tkaczewski Duration: 30 minutes
  2. 2. Presenter Introduction John Tkaczewski • Co-Founder and President of Unlimi-Tech Software • Oversees creative and technical vision for FileCatalyst Webmail and FileCatalyst Workflow • Blog: http://www.filecatalyst.com/blog Christian Charette • Director of Software Development • Project lead on 10 Gbps research | © Copyright 2 FileCatalyst, 2013
  3. 3. Agenda • FileCatalyst Direct Overview • Quick refresher why TCP doesn’t work in high speed networks • 10Gbps test infrastructure – Ie: what are we using to perform the demo? • Watch the data fly! • Questions (please type them in as we go) | © Copyright 3 FileCatalyst, 2013
  4. 4. FileCatalyst Direct Overview • Software architecture is Client-Server. • One side acts as the main file repository (FileCatalyst Server) • One or many clients connect to server, and push/pull data • Clients include: – automated interfaces (API, CLI, Javascript, C++) – web components (applets) – feature rich applications (HotFolder, Express, mobile) • Primarily written in Java, works on every major platform (Windows, Linux, Mac OSX, iOS, Android, and more) • Transport protocol utilizes proprietary UDP, with error correction and congestion control built into the application layer (instead of relying on TCP network layer) | © Copyright 4 FileCatalyst, 2013
  5. 5. Putting 10 Gbps in context • 10 Gbps transfers an average BlueRay disk (26GB) in under 30 seconds • Cost of 10 Gbps dedicated lines is dropping – becoming cost effective to many companies (~10K/month US to UK) • 1 Gbps fiber connections becoming available to residential consumers | © Copyright 5 FileCatalyst, 2013
  6. 6. TCP vs UDP for packet loss & latency • Quick refresher: TCP can’t cut it. • Data provided as part of Photonics North 2012 presentation, FTP data uses FileZilla client connecting to PureFTPD | © Copyright 6 FileCatalyst, 2013
  7. 7. TCP vs UDP for packet loss & latency • TCP cannot handle packet loss -- even for networks with small latency. • High latency networks fall apart under real world conditions. | © Copyright 7 FileCatalyst, 2013
  8. 8. TCP vs UDP for packet loss & latency • You cannot transfer files at 10 Gbps with TCP. It just doesn’t work. | © Copyright 8 FileCatalyst, 2013
  9. 9. 10 Gbps transfers • 10 Gbps research project started in May 2011. • Project completed 6 months later with Nov 2011 release of FileCatalyst Direct v3.0, able to transfer at full 10gbps speed. • 1½ years of successful deployments of 2-10 Gbps in the field. • Tested in 2012 by independent researchers (ex-CEO of competitor Tixel GmbH), validated as most stable transfer results between 5 managed transport solutions. Comparison of Contemporary Solutions for High Speed Data Transport on WAN 10 Gbit/s Connections, Dmitry Kachan, Eduard Siemens, Vyacheslav Shuvalov | © Copyright 9 FileCatalyst, 2013
  10. 10. What are we using to run this demo? FileCatalyst HotFolder Ares x.x.x.58 | © Copyright 10 FileCatalyst, 2013 FileCatalyst Server Hermes x.x.x.59
  11. 11. What are we using to run these tests? • Machines utilize standard components circa 2011. • Each machine costs approx. $5000 to put together. – Intel Westmere 5600 series (used in demo testbed) – AMD Magny-Cours 6100 Opterons FileCatalyst HotFolder Ares x.x.x.58 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 | © Copyright 11 FileCatalyst, 2013 FileCatalyst Server Hermes x.x.x.59 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2
  12. 12. What are we using to run these tests? • Machine has 48GB total, 36GB reserved for RAMFS (so 12GB for OS + FileCatalyst). • For the purpose of these tests, going to read/write to RAM drive. • Why? Want to remove disks as part of the equation for demo to simplify things (everybody has different IO solutions in place). • If your disks can handle the speeds, our software can transfer it. FileCatalyst HotFolder Ares x.x.x.58 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 | © Copyright 12 FileCatalyst, 2013 FileCatalyst Server Hermes x.x.x.59 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2
  13. 13. What are we using to run these tests? • In demo, to simulate WAN environment, a Linktropy box from Apposite Technologies is used – Control link speed, packet loss, and link latency FileCatalyst HotFolder Ares x.x.x.58 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 | © Copyright 13 FileCatalyst, 2013 FileCatalyst Server Hermes x.x.x.59 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 WAN Emulator Apposite Technologies
  14. 14. So what are we going to see? FileCatalyst HotFolder Ares x.x.x.58 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 | © Copyright 14 FileCatalyst, 2013 FileCatalyst Server Hermes x.x.x.59 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 WAN Emulator Apposite Technologies
  15. 15. So what are we going to see? • (1) Going to test most basic scenario: – Transfer static file from point A to B – 30GB file… slightly larger than the average Blue Ray disk. FileCatalyst HotFolder Ares x.x.x.58 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 | © Copyright 15 FileCatalyst, 2013 FileCatalyst Server Hermes x.x.x.59 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 WAN Emulator Apposite Technologies
  16. 16. So what are we going to see? • (2) Going to test standard live media scenario: – Growth files (5) simulate live video feeds saved to system – Move files in real-time to destination as files are output FileCatalyst HotFolder Ares x.x.x.58 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 | © Copyright 16 FileCatalyst, 2013 FileCatalyst Server Hermes x.x.x.59 Dual Intel Xeon 5690 48 GB of RAM (36 GB as RAMDRIVE ) Intel 10Gbps cards Ubuntu 10.04.1 LTS Java 1.7 64-bit FileCatalyst v3.2 WAN Emulator Apposite Technologies
  17. 17. What are we using to run these tests? • Linktropy setup to simulate LA -> NY transfers – Latency of 60 ms – 0.1% packet loss • Configurations used for single file transfer @ 10gbps: – Server & HotFolder each using 4GB of RAM – MTU 9000 (packet 8972 bytes) – Blocks: 10MB – Number of sender threads: 50 – Sender sockets: 5 – Receiver sockets: 3 – RTT congestion control (because test scenario utilizes packet loss) – Congestion control set to 7 (more aggressive) | © Copyright 17 FileCatalyst, 2013
  18. 18. DEMOS DEMOS | © Copyright 18 FileCatalyst, 2013
  19. 19. Upcoming Events: - NYC Digital Asset Managers, (May 29th) - Broadcast Asia, Singapore (June 18-21) - IBC, Amsterdam (Sept 13-17) - Interop New York (Oct 2-3) - Broadcast India, Mumbai (Oct 9-11) And of course don’t forget our monthly webinar....
  20. 20. Thank you for attending our May webinar! Questions? Please type in your questions.

×