Hoffman Lab Tech Talk

1. Luigi - A Python framework for data
processing an pipelining
Mickaël Mendez
Tech Talk – Jun 27, 2016

2. Summary
● What is a pipeline?
● Luigi
● Tips and Tricks
● Conclusion

3. A Simple Pipeline
● Count the number of interval per chromosome

4. A more robust pipeline

5. Add logs

6. GNU Make 1/2
http://www.gnu.org/software/make/manual/make.html

7. GNU Make 2/2
http://www.gnu.org/software/make/manual/make.html

8. Luigi
● Python Framework
● Developed at Spotify©
“
“
http://luigi.readthedocs.io/en/stable/

9. Luigi – Main features
● Task templating
● Dependency graphs
● Resumption of data flows after intermediate
failure
● Command line integration
● Run Tasks in parallel

10. Luigi – Cheat sheet

11. Luigi - Example

12. Luigi -Example

13. Running a pipeline
● python luigi_word_count.py
● On mordor:
– qlogin -pe smp 10 #From the main node
– Python luigi_word_count.py --workers 10
● A specific task:
– python luigi_word_count.py CountLetters

14. Luigi – Parameters

15. Configurations
● Allows to specify the parameters in an external
file
● luifi.cfg
[MyTask]
a_parameter=21

16. Tips and Tricks

17. Tips

18. Conclusion
● Luigi is a workflow manager
● It handles dependencies, job failures and
parallelism
● It is well maintained (More than 100
contributors)
● Mailing list is very active
● Used by popular companies (foursquare...)