Grid technology for next gen media processing

1,627 views
1,439 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,627
On SlideShare
0
From Embeds
0
Number of Embeds
28
Actions
Shares
0
Downloads
57
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Grid technology for next gen media processing

  1. 1. Grid technology for next-gen media processing Jens Buysse - Stijn De Smet - Koen Segers- Bruno Volckaert
  2. 2. Overview  MediaGrid concept  Distributed video transcoding  Enabling technologies  Setup overview  Test results  Simulation results  Conclusions 2
  3. 3. MEDIAGRID CONCEPT 3
  4. 4. Originating problems  Tape-based media to file-based media  Multitude of file-based media transfers and processing Storage / retrieval / transfer of media  Conforming  Transcoding  Upscaling  Editing   Geographically disperse facilities / resources / media storage 4
  5. 5. Grid technology as solution?  Grid technology a Grid is a distributed processing architecture where heterogeneous resources are shared between different participating organizations, across an interconnecting network  Resources  Storage (media archive, temporary storage, etc.)  Computational (rendering farm, work stations, etc.)  Specialized (broadcasting, ingesting, etc.)  High speed interconnecting network (1-10 Gbit/s) 5
  6. 6. MediaGrid M M M hires hires hires 6
  7. 7. MediaGrid M M M hires hires hires Grid Middleware EDL 7
  8. 8. MediaGrid: enabling virtual organisations VO 1 VO 2 8
  9. 9. DISTRIBUTED VIDEO TRANSCODING 9
  10. 10. Grid technology proof-of-concept  Investigated the viability of Grid technology for processing tasks in media production / distribution companies  Transcoding of media  Upscaling of media Video transcoding deals with converting a video signal into another one with different format, such as different bit rate, frame rate, frame size, or even compression standard  Video transcoding is a resource intense process  I/O  Processing needs 10
  11. 11. Need for transcoded / rescaled video VRT online media YouTube http://www.deredactie.be http://www.youtube.com 11
  12. 12. Distributed video transcoding  How can we accelerate this process? Server 4 Server 3 Server 1 Server 2 00:00:00  00:51:53 00:00:00 00:13:15 00:13:15 00:26:30 00:26:30 00:39:45 00:39:45 00:51:53 12
  13. 13. ENABLING TECHNOLOGIES 13
  14. 14. Enabling technologies  OS  SuSe enterprise  Transcoding software  Transcode library  Grid Middleware  TORQUE (openPBS)  Maui scheduler  Grid distributed transcoder: custom Java application  Data retrieval / storage technology  GPFS 14
  15. 15. Enabling technologies: TORQUE  TORQUE : open PBS TORQUE Server Maui Sheduler pbs_mom Job Queue 1 Policy User Queue 2 pbs_mom 15
  16. 16. Enabling technologies  Job / batch / workflow submitter  Consider job dependencies 1 2 Stock 1 3 4 5 6 Stock 2 7 8 Stock 3 16
  17. 17. Enabling technologies  Grid distributed transcoding application 17
  18. 18. Enabling technologies  Grid distributed transcoding application 18
  19. 19. SETUP OVERVIEW 19
  20. 20. Setup overview  … TORQUE  … with GPFS cluster as media storage  … Java distributed transcoding front-end  … on each computational resource Transcode libraries  … the will to transcode in a distributed fashion 20
  21. 21. First distributed transcoding workflow TORQUE 1. Split phase 00:00:00 00:13:15 00:13:15 00:26:30 2. Transcoding 3. Merge phase Node 1 00:26:30 00:39:45 00:39:45 00:51:53 00:00:00 00:51:53 Node 2 User Node 3 00:13:15 00:26:30 00:26:30 00:39:45 00:39:45 00:13:15 00:00:00 00:51:53 Node 4 00:00:00 00:51:53 GPFS 21
  22. 22. Current distributed transcoding workflow TORQUE Nav.log 1. Preprocess phase 00:00:00 00:13:15 2. Demux phase 3. Transcoding 4. Merge / multiplex Node 1 Audio.mp3 00:13:15 00:26:30 00:00:00 00:51:53 Node 2 User 00:26:30 00:39:45 Node 3 00:39:45 00:51:53 Node 4 00:00:00 00:51:53 GPFS 22
  23. 23. Future distributed transcoding workflow TORQUE Node 1 User Node 2 WAN 1. Prefetch Node 3 2. Preprocess 3. Demux 4. Transcode 5. Merge / 00:00:00 00:51:53 GPFS Remote multiplex 00:00:00 00:51:53 GPFS local 23
  24. 24. Discussion  Old version  Video files were physically split  Split / merge step could introduce artifacts  Current version  File is inspected and navigation file created allowing for easy frame- addressing  Audio ripped and transcoded in separate step  No artifacts  Less media-transfers than in previous versions  Future version  Pre-fetching / replication of media to remote sites 24
  25. 25. TEST RESULTS 25
  26. 26. Test topology Torque Server Traffic Shaping GPFS node 26
  27. 27. Test results  Input media Vob file  MPEG-2 video encoding  AC3 audio encoding  Size: 1,64 GB   Output media Avi file  Xvid video encoding  MP3 audio encoding  Size: 700 MB   Currently no HD video input modules!  Not the most optimized video transcoders  Focus on measuring benefits of distributing 27
  28. 28. Results – Grid overhead  Grid Overhead 28
  29. 29. Results – Preprocess phase  Preprocess 29
  30. 30. Results – Audio ripping phase  Rip audio 30
  31. 31. Results – Merging phase  Merging phase 31
  32. 32. Results – 1Gbit/s WAN 32
  33. 33. Results – Parameterised WAN interconnection 33
  34. 34. Video (up)scaling Video scaling is converting video signals from one size or resolution to another: usually quot;upscalingquot; or quot;upconvertingquot; a video signal from a low resolution (e.g. standard definition) to one of higher resolution (e.g. high definition television). 00:00:00 00:51:53 00:00:00 00:51:53 720X576 984x752 34
  35. 35. Video (up)scaling results – 52Mbit/s WAN 35
  36. 36. Video (up)scaling results – 52Mbit/s WAN 36
  37. 37. SIMULATION RESULTS 37
  38. 38. Simulation results  We introduced a WAN connection to a remote computational resource provider TORQUE Node 1 Node 2 User • 1 Gbit/s • 100 Mbit/s Node 3 • 52 Mbit/s • 35 Mbit/s Node 4 GPFS 38
  39. 39. Simulation results – total job turnaround time 39
  40. 40. Comparison with measured results 40
  41. 41. Comparison with measured results 41
  42. 42. Simulation results  Simulations provide very accurate total job turnaround times  Real-life transcoding behaves erroneously when interconnecting GPFS with computational resource provider by means of WAN link lower than 35Mbit/s Control Traffic Control Traffic Click Router Data Data Simulation results show what would happen to job turnaround  GPFS time for lower WAN interconnections 42
  43. 43. Simulation results – low-speed WAN interconnection 43
  44. 44. Simulation results – 10 chunks 44
  45. 45. CONCLUSIONS 45
  46. 46. Conclusions  Grid technology is a viable technology for dealing with media production / distribution tasks  Inherent support for parallelism can seriously decrease the total processing time  Need for adaptation of media tasks  Grid overhead is no issue  Outsourcing task processing to remote resource providers  Viable when interconnection is sufficient  Technical limitations (e.g. GPFS time-outs)  MediaGrid simulator can provide accurate performance predictions 46
  47. 47. Questions ? Feel free to e-mail: Bruno.Volckaert@intec.UGent.be

×