平行化你的工作 part1

802 views

Published on

A short talk I gave to R&D department in Digimax, Inc. It's dedicated to briefly describe how to parallelize your routine work in animation industry by making use of some tools.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
802
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

平行化你的工作 part1

  1. 1. 平行化你的工作 Part 1 Drake Guan Digimax, Inc. 2013/06 (http://about.me/drakeguan)
  2. 2. http://en.wikipedia.org/wiki/Parallel_computing https://en.wikipedia.org/wiki/Distributed_computing
  3. 3. Parallel instead of Distributed http://bit.ly/17WQ0gR
  4. 4. Scope !   GNU Parallel !   Parallel Python (pp) !   Alfred What can you expect? It’s just a part 1. Actually, it is animation studio oriented. Creativity is more important than simply techniques.
  5. 5. GNU Parallel GNU parallel is a shell tool for executing jobs in parallel using one or more computers. http://www.gnu.org/software/parallel/
  6. 6. http://www.gnu.org/software/parallel/
  7. 7. Online Doc Examples 是你的好朋友 http://www.gnu.org/software/parallel/
  8. 8. 你只要記得這兩個範例就行了 http://www.gnu.org/software/parallel/man.html#example__working_as_xargs__n1__argument_appending
  9. 9. 範例1: 轉檔 # straightforward but sequential way ~: for i in *.exr; do convert $i –depth 8 –alpha off –gamma 2.2 –quality 100 $(basename $i .exr).jpg; done # parallel way ~: ls *.exr | parallel do convert {} –depth 8 –alpha off –gamma 2.2 –quality 100 {.}.jpg • 8 bits: -depth 8 • No alpha: -alpha off • Gamma encoding: -gamma 2.2 • JPEG quality: -q 100
  10. 10. 範例2: 更改檔名 ~: parallel mv {} {.}.html ::: *.htm ~: find /shows/qq/mov/ -name ‘*.avi’ | parallel mv {} {.}.mov ~: find . | parallel mv {} backup_{} ~: parallel mv {} {.}_backup.jpg ::: *.jpg
  11. 11. 範例4: 瘋狂 grep RenderMan Studio 的文件很多,你想找出裏頭僅存的 Alfred 資料. ~: find /rms_doc/ -type f | parallel –k –n 1000 grep –H –n Alfred {} •  File only: -type f •  Keeping order: -k •  Multiple arguments: -n 1000
  12. 12. Parallel Processing and Multiprocessing in Python http://wiki.python.org/moin/ParallelProcessing
  13. 13. 多到爆的模組,看了就不想用!
  14. 14. Parallel Python (pp) http://www.parallelpython.com/
  15. 15. Quick start guide 是我們的好朋友
  16. 16. mkmov: images -> video ! Supering: logo, footage info, author, date, frame count, … !   Resizing: to the extend of 16 if necessary (player’s constraint) !   Gamma encoding !   HDR -> LDR !   Movie maker
  17. 17. mkmov code snippets
  18. 18. 你要懂的 pickling !   Python 先天上有 GIL (Global Interpreter Lock) 的限制:一 個 thread 要被 Python interpreter 執行,得先要個這個 lock. !   Parallel Python and others, 都是走 multi process (or fork). !   Picking: 醃製/醬瓜,用來保存食物的方法(酸,但可以吃) !   Python Pickle: 用來保存(serializing)記憶體狀態的方法 !   Parallel Python modules 幾乎都用 pickle 來處理平行化時的 message passing.
  19. 19. Alfred
  20. 20. Alfscript
  21. 21. dgAlfred !   讓每個人都可以透過 alfred 使用我們的 render farm. ! dgAlfred 幫你翻譯(產生) alfscript. !   所有 alfscript 的語法都支援. 每個指令都有個 class 的對應. !   是 ypcat 的傑作! 程式碼非常短!
  22. 22. dgalfred -> genTask(…)
  23. 23. Applications of dgalfred ! rfMaya.py ! rfNuke.py ! rfShake.py ! rfMkDoremi.py ! rfMopBake ! rf -> render farm
  24. 24. That’s Part 1.

×