DevOps and Automation for Bioinformaticians
•Download as PPT, PDF•
3 likes•584 views
Slide deck from my recent presentation at the Rocky Mountain Bioinformatics Conference in Snowmass, CO.
Recommended
More Related Content
What's hot
What's hot (18)
Viewers also liked
Viewers also liked (20)
Similar to DevOps and Automation for Bioinformaticians
Similar to DevOps and Automation for Bioinformaticians (20)
Recently uploaded
Recently uploaded (20)
DevOps and Automation for Bioinformaticians
- 1. http://www.bioteam.net Dev Ops and Automation for Bioinformaticians Simon Twigger, Ph.D. Rocky Mountain Bioinformatics Conference, 2014 HPC & Bioinformatics Consulting Clients across Government, Pharma, Biotech, Academia
- 2. Development/Deployment Challenge Team Local IT ‘The Cloud’Complex App & Environment Intimate link between your software and where its being run
- 3. Chef • Uses a Ruby DSL • Cookbooks + recipes to describe the infrastructure https://www.getchef.com/ •Run chef to configure your installation, set up users, define ENV, checkout code, install and build your software •Save to your lab chef-repo and preserve configuration •Wide variety of public cookbooks available at the supermarket…
- 4. Vagrant • Host Virtual Machines on your desktop • Infrastructure configuration described in a Vagrantfile, saved and tracked in github, etc. • Use the Chef recipes to provision the server • Same tools can be used to build for VMs, AWS, https://www.vagrantup.com/
- 5. Continuous Integration/Deployment • Don’t hope that it still works… • Automate your test suite (you do have a test suite…?) • Don’t hope that it all still comes together… • Automate your deployment (at least to a staging/test environment) https://jenkins-ci.org/ https://codeship.io
- 6. Invest in DevOps • Invest some time up front to learn some DevOps tools/approaches • Chef, Puppet, Ansible, even plain old Bash • Manage your infrastructure like you manage your source code • Chef Recipes • Ansible Playbooks • Vagrant • Consider Jenkins as a CI (and CD) solution • Make Source Control a requirement • Make testing a requirement simon@bioteam.net
- 7. Invest in DevOps • Invest some time up front to learn some DevOps tools/approaches • Chef, Puppet, Ansible, even plain old Bash • Manage your infrastructure like you manage your source code • Chef Recipes • Ansible Playbooks • Vagrant • Consider Jenkins as a CI (and CD) solution • Make Source Control a requirement • Make testing a requirement simon@bioteam.net
Editor's Notes
- Presentation for the Rocky Mountain Bioinformatics Conference, December 2014. Bioteam – small team of consultants that get to do some cool stuff with some cool clients (http://bioteam.net) Our breadth of clients provides us with great visibility into a wide spectrum of Bioinformatics and Bio-IT environments, what works and where the industry is moving. Getting a grip on infrastructure, embracing change and flexibility, leveraging the cloud are all high on the list. What can be done to make this easier?
- The problem: You are likely building some software, you start with some code, then download and add a library, set some environment variables, link to a database, some local data, etc. and pretty soon you have a complex application and a non-trivial environment. However, you want to distribute this app to your team, scale it to your local IT environment or move it to the cloud All those custom configurations and tweaks now come back to haunt you, you’d like to try some new compute options but you are stuck in the tar-pit of manual configuration unable to break free! If you want to be able to share your code you have to be able to share your whole environment too… How to do this? Just like in the lab - You need a recipe that describes exactly how to create your environment, something that’s repeatable and sharable
- Chef gives you a way to both describe your configuration and setup and create it You write ‘recipes’ and build cookbooks that describe your system in a repeatable fashion e.g. A basic recipe to deploy a genomics software package Create users Create install directory Grab a local copy of the software tarball Unzip and install This cookbook can be shared and run repeatedly with the same result (Idempotent) You have a way to describe and build your software but you need this running on a server or a dev box, how to do this?
- You are likely building on Linux of some type so you need sandbox, a place to develop your software that mimics the environment its going to end up in – Vagrant provides this. Built off Oracle’s VirtualBox VM you can download and run many common flavors of UNIX (Ubuntu, CentOS) Create a Vagrant file which brings in the desired base OS, set up networking, provision the VM and away you go Plugins allow you to build directly on VMWare or AWS if you want. Vagrant Up creates a complete environment on demand, you can pass in alternative configuration variables into your chef recipes to add in extra flexibility – e.g. here we’re defining a specific human reference genome to be loaded onto the server.
- You now have a way to describe and build your infrastructure on command, the next step is to build in some resiliency against change (new code, new dependencies, etc.). Continuous Integration (CI) tools take you code and can automatically check it out and run your test suite so you can be sure it still works (or know that its broken so you can fix it). You can also get one of those cool ‘Build: Passing’ badges on your github repo and be like all the cool kids. The final step is continuous deployment (CD) to give you confidence that you can still deploy the whole enchilada to somewhere useful. This will complete the loop – check out your code, run the tests and if they pass, deploy the new code so your new features can be put to use straight away. Using CD to production has a few more caveats to watch out for but deploying to a staging site or similar is a great way to get new code in the hands of your team or client so they can test it, provide feedback before you do move it to production. Example tools for CI/CD include commercial tools like Codeship (http://Codeship.io) and Jenkins (http://jenkins-ci.org) an excellent open source route.
- Consider investing some time in DevOps and get a handle on your infrastructure and your development pipeline. Yes, it is likely a change from how you used to do things, Yes, it will take you a bit of time to get up to speed. Yes it will save you time down the road, Yes your team will thank you when all they have to type is ‘vagrant up’ to get your new software running. Yes, your users will thank you for taking the time to put this in place so they can easily download and use your new algorithms. Yes, you will be psyched when you can run the same code on AWS or Google Compute Engine and try your software on a high memory node or compute optimized node to really make it go! Plus, these are the tools and techniques that the software industry is using, bring this knowledge to your academic institution/pharma/biotech and be better placed to take those skills to where ever your computational career may take you… https://www.getchef.com/ http://www.ansible.com/ - Slightly simpler than Chef with fewer dependencies, etc., newer kid on the block, not so much documentation and supporting resources but this is what we’re spending our time learning these days. https://www.vagrantup.com/ CI/CD Solutions https://codeship.com/ http://jenkins-ci.org/ Also: https://travis-ci.org/ https://www.solanolabs.com/ https://www.atlassian.com/software/bamboo This wouldn’t be complete without mentioning Docker (https://www.docker.com/) the newest kid on the ‘lets make deployment easier’ block! Clearly the newest hotness, Illumina has been making some headway using this as the underpinnings of their Basespace platform, check out their Native Apps documentation for more info: https://developer.basespace.illumina.com/docs/content/documentation/native-apps/native-app-overview