1. Writing
Data Analysis Pipeline
As Ruby Gem
Shi-Gang Wang

2. About me
{
name: ‘ Shi-Gang Wang ( Sean ) ’,
email: ‘ seanwang@goldenio.com ’,
working_at: ,
role: [‘ software engineer ’],
language: ‘ ruby ’,
github: ‘ https://github.com/seansg ’
}

3. Outline
❖ What is pipeline
❖ Disassemble pipeline
❖ Queue a pipeline

4. ?

6. pineapple.txt

7. pineapple.txt
cat pineapple.txt

8. pineapple.txt
cat pineapple.txt
cat pineapple.txt | grep apple

9. pineapple.txt
cat pineapple.txt
cat pineapple.txt | grep apple
cat pineapple.txt | grep apple | wc -l

10. Write scripts to do one thing
Make scripts to work together
=> Pipeline

12. Take
as an example

13. CAGNUT
❖ Computational and Analytical Gear for Nucleic acid
Utilitarian Techniques
❖ DNA analysis pipeline
❖ Burrows-Wheeler Aligner (BWA) — in C
❖ Sequence Alignment/Map tools (SAMtools) — in C
❖ Genome Analysis Toolkit (GATK) — in Java
❖ Picard — in Java
❖ Generate bash scripts

14. A Genome Analysis
Flowchart
https://software.broadinstitute.org/gatk/img/BP_workflow_3.6.png

16. Demo

17. How to write the pipeline?

18. How to write the pipeline?
How to disassemble the
pipeline?

19. Think about the pipeline
structure
Pipeline
Tools
CAGNUT
Core

20. Think about the pipeline
structure
Pipeline
CAGNUT
Core
Tools

21. Write all parts as ruby gems

22. Benefits of ruby gems
❖ Reuse
❖ Debug
❖ Maintain
❖ Share

23. Difficulties
❖ Usage
❖ Integration of tools
❖ Execution order
❖ Automation

24. Prepare work — Define help
❖ “Help” can help you understand how to use the
commands of pipeline

25. Prepare work — Namespace

26. Skills of writing the gems
Part 1 — Tool gems
Part 2 — Pipeline gem
Part 3 — Cagnut core gem

27. Part 1 — Tool gems
❖ Tool written in Singleton
❖ Tool methods written in class
❖ Job scripts generation
Pipelin
e
Tools
CAGNUT
Core

28. Tool written in Singleton

29. Tool method written in class

30. Get specific variables in other
class
❖ Use Forwardable

31. Job scripts generate
❖ Use Tilt
❖ Generic interface to
multiple Ruby
template engines

32. Part 2 — Pipeline gem
❖ Require tool gems
❖ Create workflow with tool gems
❖ Generate the job list
Pipelin
e
Tools
CAGNUT
Core

33. Require tool gems
❖ Loading bundle env

34. Create workflow with tool
gems
❖ Composed by tool gems
❖ Order
❖ Dependency

35. Generate the job list

36. Part 3 — CAGNUT core gem
❖ Project template prepare
❖ Parameters handling
❖ Tool-specific methods overwrite
❖ Jobs control
Pipelin
e
Tools
CAGNUT
Core

37. Project template prepare
❖ Define bundle as Thor command

38. Parameter handing
❖ Use OptionParser

39. Tool-specific method
overwrite
❖ One tool, One configuration
❖ Using “Prepend” to overwrite
dev.af83.com/2012/10/19/ruby-2-0-module-prepend.html

40. Jobs Control — desktop run
❖ wait $!
❖ detach

42. If the data is large
or
much larger,
like the human genome

43. The size of the human
genome is
3 x109 base pairs (bps)

44. Each base pair takes 2 bits
(you can use 00, 01, 10, and 11 for T, G, C
and A)
2 x 3 x 109 bits
= 6 x109 bits
= 7.5x108 bytes
= ~700 MB

45. In a perfect world: ~700 MB
(just 3 billion letters)
In the real world: ~200 GB
(right off the genome sequencer)

46. Crash your desktop/laptop!

47. Long wait …

48. Resource allocation

49. Resource allocation
❖ Specifying the memory used by the
program
❖ Using Queueing System

50. What is Queueing System?

51. Queueing System
BD C A
Waiting JobsJob Finished Job
System
❖Queue
❖the list of waiting jobs
❖Queueing System
❖Waiting Jobs + Servers
Server n
Server 2
Server 1

52. In a desktop computer

53. Cluster Queues

54. Queueing System
❖ Props
❖ Jobs scheduling
❖ Load balancing
❖ Batch jobs execution

55. Queueing System
❖ Portable Batch System (PBS)
❖ Sun Grid Engine (SGE)
❖ Load Sharing Facility (LSF)

56. Submit jobs to Queueing
System
❖ Take LSF as an example
❖ Creating a job script
❖ Submitting the job

57. Demo
❖ Submit jobs to cluster

58. Acknowledgement
https://cagnut.golden.io
https://goldenio.com

59. Thanks

60. Backup