dkNET Webinar - Appyters: Turning Jupyter Notebooks into Data-Driven Web Apps 05/28/2021
Abstract Jupyter Notebooks have transformed the communication of data analysis pipelines by facilitating a modular structure that brings together code, markdown text, and interactive visualizations. Here, we extended Jupyter Notebooks to broaden their accessibility with Appyters. Appyters turn Jupyter Notebooks into fully functional standalone web-based bioinformatics applications. Appyters present to users an entry form enabling them to upload their data and set various parameters for a multitude of data analysis workflows. Once the form is filled, the Appyter executes the corresponding notebook in the cloud, producing the output without requiring the user to interact directly with the code. Appyters were used to create many bioinformatics web-based reusable workflows, including applications to build customized machine learning pipelines, analyze omics data, and produce publishable figures. These Appyters are served in the Appyters Catalog at https://appyters.maayanlab.cloud. In summary, Appyters enable the rapid development of interactive web-based bioinformatics applications. The top 3 key questions that Appyters can answer: 1. I wrote my workflow as a Python Jupyter Notebook, is there an easy way that I can quickly convert this notebook into a web app so that others can use my workflow to process their data? 2. I have bulk RNA-seq data that I collected and would like to analyze. The genomics core provided me with the aligned reads file, but I am not sure about the next steps. Can I use an Appyter to analyze my data? 3. I am interested in doing some data analysis using the TCGA RNA-seq data, but I am having trouble accessing and formatting the data I need from the new GDC data portal. Is there an Appyter that I can use to access these RNA-seq data? Presenter: Avi Ma'ayan, PhD, Mount Sinai Endowed Professor in Bioinformatics, Professor in Department of Pharmacological Sciences, and Director of Mount Sinai Center of Bioinformatics, Icahn School of Medicine at Mount Sinai Upcoming webinars schedule: https://dknet.org/about/webinar
Recommended
Recommended
More Related Content
More from dkNET
More from dkNET (20)
Recently uploaded
Recently uploaded (20)
dkNET Webinar - Appyters: Turning Jupyter Notebooks into Data-Driven Web Apps 05/28/2021
- 1. Appyters: Turning Jupyter Notebooks into Data Driven Web Apps dkNET, May 28, 2021 Daniel J. B. Clarke, Avi Ma’ayan Mount Sinai Center for Bioinformatics Department of Pharmacological Sciences, Icahn School of Medicine at Mount Sinai, New York, NY 10029 USA
- 2. Ma’ayan et al. Science 310, 1078 (2005) Large Scale Cell Signaling Network Extracted from Literature Green arrows- activation Red plungers- inhibition Blue- neutral/scaffolding
- 3. KEA: Web-Based System for Analyzing Lists of Proteins Utilizing Kinase-Substrate Interactions from the Literature Bioinformatics 11, 87 (2010) https://maayanlab.cloud/kea3/
- 4. https://maayanlab.cloud/kea3/# KEA3: Improved KEA Enrichment Analysis via Omics Integration Nucleic Acids Research, gkab359, Published: 21 May 2021
- 5. ChEA: ChIP-X Enrichment Analysis to Infer Transcription Factors Regulating Changes in Gene Expression • 197 unique transcription factors • 212 publications • 46,112 genes • Average targets per transcription factor ≈ 1,300 • Total interactions 439,913 Bioinformatics (2010) 26(19): 2438-2444 https://maayanlab.cloud/chea3/
- 6. ChEA3: Improved Transcription Factor Enrichment Analysis via Omics Integration https://maayanlab.cloud/chea3 Nucleic Acids Research, 47, W1, W212–W224 (2019)
- 7. Ma’ayan et al. Trends Pharmacol Sci. 2014 Sep;35(9):450-60.
- 21. https://maayanlab.cloud/biojupies BioJupies Demo Cell Syst. 2018 Nov 28;7(5):556-561.e3.
- 26. So What is an Appyter and How You Can Create One? Clarke et al., Appyters: turning Jupyter Notebooks into data-driven web apps, Patterns (2021), https://doi.org/10.1016/j.patter.2021.100213
- 27. Manual Curation of >500 High Quality Diabetes Signatures from GEO
- 29. https://appyters.maayanlab.cloud/ The Appyters Catalog Currently Contains 80 Appyters Clarke et al., Appyters: turning Jupyter Notebooks into data-driven web apps, Patterns (2021), https://doi.org/10.1016/j.patter.2021.100213
- 30. https://appyters.maayanlab.cloud/ Other Notable Appyters Clarke et al., Appyters: turning Jupyter Notebooks into data-driven web apps, Patterns (2021), https://doi.org/10.1016/j.patter.2021.100213
- 31. Construct a meta-Jupyter Notebook report Use the Appyter SDK to convert the meta-Jupyter Notebook into a web app Customized & persistent user report User brings their data Tweakable parameters So What is an Appyter and How Can You Create One? Daniel Clarke
- 32. Installing appyter and getting started % pip install appyter jupyter % jupyter notebook To access the notebook, open this file in a browser: http://localhost:8888/?token=*** Python3.8+ should be installed. Familiarity with jupyter is a must. Familiarity with jinja2 will help. Start jupyter notebook
- 33. Preparing the notebook & starting the appyter! This snippet is available on our github https://github.com/maayanlab/appyter Now it’s just a normal notebook with appyter features! % appyter --profile=biojupies My First Appyter.ipynb ======== Running on http://127.0.0.1:5000 ======== Nothing much yet.
- 34. Making something useful Let’s look at the COVID Tracking Project data https://covidtracking.com/data/download
- 35. Making Something Useful Continued
- 36. Making Something Useful Continued Let’s make it so the user can choose their own state!
- 37. Parameterizing Code The resulting source code after substituting the default The output, given the source code The appyter page after adding the field Altering the field and clicking submit will generate the substituted notebook and execute it. Jinja2 Substitution Jinja2 variable assignment of Appyter field
- 38. Permitting File Uploads NOTE: Make sure your file input is typical so people can easily format their data for your appyter
- 39. Prettification and Preparing to Publish Customize the section with icon and such Pure appyter cell
- 40. Turning Notebook into a ‘Publication’ You can even use template substitutions in markdown! NOTE: Even though it’s markdown -- you use a ‘code cell’ with `%%appyter markdown` if you need template substitutions. Don’t forget figure legends, and citation information.
- 42. Running in Production, or Publishing on The Catalog Running in Production % appyter --profile=biojupies --debug=false My First Appyter.ipynb Publishing More information including the catalog of existing appyters, documentation about creating and publishing your own appyters at https://appyters.maayanlab.cloud
- 43. Single Source of Truth: Jupyter Notebook (ipynb) CLI Web Form (construct notebook) appyter.ipynb Orchestration (execution queuing) Version controlled & programmatically validated for appyter-catalog appyter.json FAIR Metadata version license tags author “Meta” Jupyter Notebook REST API Production Configuration (supervisord, nginx) Development Tools (profiles, extras) nbexecute Web View (view notebook) nbconstruct
- 44. Multi-Appyter Catalog Continuous Integration -LB S3-compat Orchestrator Catalog nbexecute PostgREST docker-compose Kubernetes Pull Request Github Actions Build nbexecute nbconstruct (defaults) Manual Review Merge, production pulls appyter.json
- 45. • We systematically convert publicly available omics datasets into an abstract format centered on genes and drugs. • The bioinformatics tools that we developed have made a big impact on the research community. • Appyters can enable the rapid development of bioinformatics applications. • Appyters are easier to maintain and can run anywhere. • Case study demonstrated how we created a useful Appyter for the diabetes research community. Summary
- 46. Acknowledgements Ma’ayan Lab Sherry Jenkins, MS - Project Manager Daniel Clarke, MS - Data Science Analyst Alexander Lachmann, PhD - Assistant Professor Kathleen Jagodnik, PhD - Postdoctoral Fellow Minji Jeon, PhD - Postdoctoral Fellow Megan Wojciechowicz, MS - PhD Student John Erol Evangelista, MS - Bioinformatician Sherry Xie, BS - Bioinformatician Eryk Kropiwnicki, MS - Bioinformatician Maxim Kuleshov, MS - Bioinformatician DRs. Alan Attie and Mark Keller - UWM