Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Grid technology for next gen media processing

  • Login to see the comments

Grid technology for next gen media processing

  1. 1. Grid technology for next-gen media processing Jens Buysse - Stijn De Smet - Koen Segers- Bruno Volckaert
  2. 2. Overview  MediaGrid concept  Distributed video transcoding  Enabling technologies  Setup overview  Test results  Simulation results  Conclusions 2
  3. 3. MEDIAGRID CONCEPT 3
  4. 4. Originating problems  Tape-based media to file-based media  Multitude of file-based media transfers and processing Storage / retrieval / transfer of media  Conforming  Transcoding  Upscaling  Editing   Geographically disperse facilities / resources / media storage 4
  5. 5. Grid technology as solution?  Grid technology a Grid is a distributed processing architecture where heterogeneous resources are shared between different participating organizations, across an interconnecting network  Resources  Storage (media archive, temporary storage, etc.)  Computational (rendering farm, work stations, etc.)  Specialized (broadcasting, ingesting, etc.)  High speed interconnecting network (1-10 Gbit/s) 5
  6. 6. MediaGrid M M M hires hires hires 6
  7. 7. MediaGrid M M M hires hires hires Grid Middleware EDL 7
  8. 8. MediaGrid: enabling virtual organisations VO 1 VO 2 8
  9. 9. DISTRIBUTED VIDEO TRANSCODING 9
  10. 10. Grid technology proof-of-concept  Investigated the viability of Grid technology for processing tasks in media production / distribution companies  Transcoding of media  Upscaling of media Video transcoding deals with converting a video signal into another one with different format, such as different bit rate, frame rate, frame size, or even compression standard  Video transcoding is a resource intense process  I/O  Processing needs 10
  11. 11. Need for transcoded / rescaled video VRT online media YouTube http://www.deredactie.be http://www.youtube.com 11
  12. 12. Distributed video transcoding  How can we accelerate this process? Server 4 Server 3 Server 1 Server 2 00:00:00  00:51:53 00:00:00 00:13:15 00:13:15 00:26:30 00:26:30 00:39:45 00:39:45 00:51:53 12
  13. 13. ENABLING TECHNOLOGIES 13
  14. 14. Enabling technologies  OS  SuSe enterprise  Transcoding software  Transcode library  Grid Middleware  TORQUE (openPBS)  Maui scheduler  Grid distributed transcoder: custom Java application  Data retrieval / storage technology  GPFS 14
  15. 15. Enabling technologies: TORQUE  TORQUE : open PBS TORQUE Server Maui Sheduler pbs_mom Job Queue 1 Policy User Queue 2 pbs_mom 15
  16. 16. Enabling technologies  Job / batch / workflow submitter  Consider job dependencies 1 2 Stock 1 3 4 5 6 Stock 2 7 8 Stock 3 16
  17. 17. Enabling technologies  Grid distributed transcoding application 17
  18. 18. Enabling technologies  Grid distributed transcoding application 18
  19. 19. SETUP OVERVIEW 19
  20. 20. Setup overview  … TORQUE  … with GPFS cluster as media storage  … Java distributed transcoding front-end  … on each computational resource Transcode libraries  … the will to transcode in a distributed fashion 20
  21. 21. First distributed transcoding workflow TORQUE 1. Split phase 00:00:00 00:13:15 00:13:15 00:26:30 2. Transcoding 3. Merge phase Node 1 00:26:30 00:39:45 00:39:45 00:51:53 00:00:00 00:51:53 Node 2 User Node 3 00:13:15 00:26:30 00:26:30 00:39:45 00:39:45 00:13:15 00:00:00 00:51:53 Node 4 00:00:00 00:51:53 GPFS 21
  22. 22. Current distributed transcoding workflow TORQUE Nav.log 1. Preprocess phase 00:00:00 00:13:15 2. Demux phase 3. Transcoding 4. Merge / multiplex Node 1 Audio.mp3 00:13:15 00:26:30 00:00:00 00:51:53 Node 2 User 00:26:30 00:39:45 Node 3 00:39:45 00:51:53 Node 4 00:00:00 00:51:53 GPFS 22
  23. 23. Future distributed transcoding workflow TORQUE Node 1 User Node 2 WAN 1. Prefetch Node 3 2. Preprocess 3. Demux 4. Transcode 5. Merge / 00:00:00 00:51:53 GPFS Remote multiplex 00:00:00 00:51:53 GPFS local 23
  24. 24. Discussion  Old version  Video files were physically split  Split / merge step could introduce artifacts  Current version  File is inspected and navigation file created allowing for easy frame- addressing  Audio ripped and transcoded in separate step  No artifacts  Less media-transfers than in previous versions  Future version  Pre-fetching / replication of media to remote sites 24
  25. 25. TEST RESULTS 25
  26. 26. Test topology Torque Server Traffic Shaping GPFS node 26
  27. 27. Test results  Input media Vob file  MPEG-2 video encoding  AC3 audio encoding  Size: 1,64 GB   Output media Avi file  Xvid video encoding  MP3 audio encoding  Size: 700 MB   Currently no HD video input modules!  Not the most optimized video transcoders  Focus on measuring benefits of distributing 27
  28. 28. Results – Grid overhead  Grid Overhead 28
  29. 29. Results – Preprocess phase  Preprocess 29
  30. 30. Results – Audio ripping phase  Rip audio 30
  31. 31. Results – Merging phase  Merging phase 31
  32. 32. Results – 1Gbit/s WAN 32
  33. 33. Results – Parameterised WAN interconnection 33
  34. 34. Video (up)scaling Video scaling is converting video signals from one size or resolution to another: usually quot;upscalingquot; or quot;upconvertingquot; a video signal from a low resolution (e.g. standard definition) to one of higher resolution (e.g. high definition television). 00:00:00 00:51:53 00:00:00 00:51:53 720X576 984x752 34
  35. 35. Video (up)scaling results – 52Mbit/s WAN 35
  36. 36. Video (up)scaling results – 52Mbit/s WAN 36
  37. 37. SIMULATION RESULTS 37
  38. 38. Simulation results  We introduced a WAN connection to a remote computational resource provider TORQUE Node 1 Node 2 User • 1 Gbit/s • 100 Mbit/s Node 3 • 52 Mbit/s • 35 Mbit/s Node 4 GPFS 38
  39. 39. Simulation results – total job turnaround time 39
  40. 40. Comparison with measured results 40
  41. 41. Comparison with measured results 41
  42. 42. Simulation results  Simulations provide very accurate total job turnaround times  Real-life transcoding behaves erroneously when interconnecting GPFS with computational resource provider by means of WAN link lower than 35Mbit/s Control Traffic Control Traffic Click Router Data Data Simulation results show what would happen to job turnaround  GPFS time for lower WAN interconnections 42
  43. 43. Simulation results – low-speed WAN interconnection 43
  44. 44. Simulation results – 10 chunks 44
  45. 45. CONCLUSIONS 45
  46. 46. Conclusions  Grid technology is a viable technology for dealing with media production / distribution tasks  Inherent support for parallelism can seriously decrease the total processing time  Need for adaptation of media tasks  Grid overhead is no issue  Outsourcing task processing to remote resource providers  Viable when interconnection is sufficient  Technical limitations (e.g. GPFS time-outs)  MediaGrid simulator can provide accurate performance predictions 46
  47. 47. Questions ? Feel free to e-mail: Bruno.Volckaert@intec.UGent.be

×