Cryo-EM Science Gateway for Big Data
•
0 likes•116 views
COSMIC2 is a research and drug discovery tool for cryoelectron microscopy high resolution protein structure imaging that extends availability of high performance computing beyond the "power user". COSMIC2 fuels innovation in drug target discovery by making it easier for a wider range of researchers to use sophisticated computing tools and resources. Courtesy Mona Wong-Barnum of San Diego Supercomputing Center (SDSC)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Cryo-EM Science Gateway for Big Data
Similar to Cryo-EM Science Gateway for Big Data (20)
Recently uploaded
Recently uploaded (20)
Cryo-EM Science Gateway for Big Data
- 1. July 20, 2017 COSMIC2 A Science Gateway for Cryo-Electron Microscopy with Globus for Terabyte-sized Datasets
- 2. What is a Science Gateway? • A science gateway is a website that ”allows science & engineering communities to access shared data, software, computing services, instruments, educational materials, and other resources specific to their disciplines.” 2
- 3. Structural Biology • DNA • Macromolecules & assemblies • Midst of revolution 3
- 4. Cryo Electron Microscopy (Cryo-EM) • Electron Microscopy at liquid nitrogen temperature • More “natural” specimen • Atom-‐level resolution • “Method of the Year” (Nature Methods, 2015) 4
- 5. Challenges • Large datasets – Resolution : 7420 x 7676 pixels per image – 1 day ~ 5 TB – 1 week (typical) ~ 10 – 30 TB 5
- 6. Challenges • Large datasets – Resolution : 7420 x 7676 pixels per image – 1 day ~ 5 TB – 1 week ~ 10 – 30 TB • Requires HPC (High Performance Computing) 6
- 7. Challenges • Large datasets – Resolution : 7420 x 7676 pixels per image – 1 day ~ 5 TB – 1 week ~ 10 – 30 TB • Requires HPC (High Performance Computing) • Command line tools 7
- 8. COSMIC2 Science Gateway • Web-‐based UI 8
- 9. COSMIC2 Science Gateway • Web-‐based UI • Built upon CIPRES Workbench Framework 9
- 10. COSMIC2 Science Gateway • Web-‐based UI • Built upon CIPRES Workbench Framework • Integrated Globus Auth & Globus Transfer 10
- 11. COSMIC2 Science Gateway • Web-‐based UI • Built upon CIPRES Workbench Framework • Integrated Globus Auth & Globus Transfer • http://cosmic-‐cryoem.org 11
- 12. Workbench Framework • Phylogenetic science gateway – Operations since 2007 – 930+ million TeraGrid/XSEDE jobs submitted – Over 23,000 unique users world-‐wide – Supported 3,200+ publications 12
- 13. Workbench Framework • Phylogenetic science gateway – Operations since 2007 – 930+ million TeraGrid/XSEDE jobs submitted – Over 23,000 unique users world-‐wide – Supported 3,200+ publications • Open-‐source, Java-‐based 13
- 14. Workbench Framework • Phylogenetic science gateway – Operations since 2007 – 930+ million TeraGrid/XSEDE jobs submitted – Over 23,000 unique users world-‐wide – Supported 3,200+ publications • Open-‐source, Java-‐based • Look-‐and-‐feel of familiar email clients 14
- 15. Workbench Framework • Phylogenetic science gateway – Operations since 2007 – 930+ million TeraGrid/XSEDE jobs submitted – Over 23,000 unique users world-‐wide – Supported 3,200+ publications • Open-‐source, Java-‐based • Look-‐and-‐feel of familiar email clients • XML configurations 15
- 16. Workbench Framework • Phylogenetic science gateway – Operations since 2007 – 930+ million TeraGrid/XSEDE jobs submitted – Over 23,000 unique users world-‐wide – Supported 3,200+ publications • Open-‐source, Java-‐based • Look-‐and-‐feel of familiar email clients • XML configurations • 16
- 17. Customizing CIPRES for COSMIC2 • Backend scripts 17
- 18. Customizing CIPRES for COSMIC2 • Backend scripts • New package wrapping Globus Auth & Transfer 18
- 19. Customizing CIPRES for COSMIC2 • Backend scripts • New package wrapping Globus Auth & Transfer • Directory-‐based datasets 19
- 20. Customizing CIPRES for COSMIC2 • Backend scripts • New package wrapping Globus Auth & Transfer • Directory-‐based datasets • GitHub repo 20
- 22. major functions • User management – Globus Auth • Data management – Functions : upload, copy, move, delete • Task management – Functions : create, clone, set parameters, view, save, delete 22
- 23. major functions • Task management (cont.) – Relion tools • 2D classification • 3D classification with 3D model • Iterative refinement of 3D volumes • Job management – Functions : view status, download results 23
- 24. Globus Auth • Federated Identity Management 24
- 25. Globus Auth • Federated Identity Management • OAuth2 and OpenID 25
- 26. Globus Auth • Federated Identity Management • OAuth2 and OpenID • HTTPS encryption 26
- 27. Globus Auth • Federated Identity Management • OAuth2 and OpenID • HTTPS encryption • Required by Globus Transfer 27
- 28. 28 User login with Globus Auth
- 29. 29 User login with Globus Auth
- 30. 30 User login with Globus Auth
- 31. 31 User login with Globus Auth
- 32. 32 User login with Globus Auth
- 33. Globus Transfer • Asynchronous, reliable, secure, efficient, large data transfer 33
- 34. Globus Transfer • Asynchronous, reliable, secure, efficient, large data transfer • Speed – UCSD -‐> SDSC : 1 TB / 1 hour – PSC -‐> SDSC : 1 TB / 1.5 hours 34
- 35. Globus Transfer • Asynchronous, reliable, secure, efficient, large data transfer • Speed – UCSD -‐> SDSC : 1 TB / 1 hour – PSC -‐> SDSC : 1 TB / 1.5 hours • 256 PB transferred • 36 billon files • 99.5% uptime 35
- 36. Globus Transfer • Asynchronous, reliable, secure, efficient, large data transfer • Speed – UCSD -‐> SDSC : 1 TB / 1 hour – PSC -‐> SDSC : 1 TB / 1.5 hours • 256 PB transferred • 36 billon files • 99.5% uptime • Requires Globus Auth 36
- 38. Globus Transfer • “end-‐points” • Source end-‐point : Globus Connect Personal/Server 38
- 39. Globus Transfer • “end-‐points” • Source end-‐point : Globus Connect Personal/Server • Destination end-‐point : XSEDE Data Oasis 39
- 40. Globus Transfer • “end-‐points” • Source end-‐point : Globus Connect Personal/Server • Destination end-‐point : XSEDE Data Oasis • One-‐time setup 40
- 41. Globus Transfer • “end-‐points” • Source end-‐point : Globus Connect Personal/Server • Destination end-‐point : XSEDE Data Oasis • One-‐time setup – Community account certificate 41
- 42. Globus Transfer • “end-‐points” • Source end-‐point : Globus Connect Personal/Server • Destination end-‐point : XSEDE Data Oasis • One-‐time setup – Community account certificate – End-‐point sub-‐directory 42
- 43. Globus Transfer • “end-‐points” • Source end-‐point : Globus Connect Personal/Server • Destination end-‐point : XSEDE Data Oasis • One-‐time setup – Community account certificate – End-‐point sub-‐directory • Gateway user access management 43
- 44. Globus Transfer • End-‐point activation 44
- 45. Globus Transfer • End-‐point activation – Setup proxy certificate 45
- 46. Globus Transfer • End-‐point activation – Setup proxy certificate – Retrieve “activation_requirements” 46
- 47. Globus Transfer • End-‐point activation – Setup proxy certificate – Retrieve “activation_requirements” – Get public key (“delegate_proxy”) 47
- 48. Globus Transfer • End-‐point activation – Setup proxy certificate – Retrieve “activation_requirements” – Get public key (“delegate_proxy”) – Create signed delegated X.509 proxy 48
- 49. Globus Transfer • End-‐point activation – Setup proxy certificate – Retrieve “activation_requirements” – Get public key (“delegate_proxy”) – Create signed delegated X.509 proxy • Transfer configurations (sync_level=2) 49
- 50. Globus Transfer • End-‐point activation – Setup proxy certificate – Retrieve “activation_requirements” – Get public key (“delegate_proxy”) – Create signed delegated X.509 proxy • Transfer configurations (sync_level=2) • Transfer ID 50
- 51. Globus Transfer • End-‐point activation – Setup proxy certificate – Retrieve “activation_requirements” – Get public key (“delegate_proxy”) – Create signed delegated X.509 proxy • Transfer configurations (sync_level=2) • Transfer ID • User notification 51
- 52. Globus Transfer • End-‐point activation – Setup proxy certificate – Retrieve “activation_requirements” – Get public key (“delegate_proxy”) – Create signed delegated X.509 proxy • Transfer configurations (sync_level=2) • Transfer ID • User notification • Status check 52
- 53. 53 Globus Transfer for large datasets
- 54. 54 Globus Transfer for large datasets
- 55. 55 Globus Transfer for large datasets
- 56. 56 Globus Transfer for large datasets
- 57. 57 Globus Transfer for large datasets
- 58. 58 Globus Transfer for large datasets
- 59. 59 Globus Transfer for large datasets
- 60. 60 Globus Transfer for large datasets
- 61. 61 Globus Transfer for large datasets
- 62. 62 Globus Transfer for large datasets
- 65. Lessons learned • Strong team effort – Domain expert – CIPRES team – Job submission – Globus Auth & Transfer integration 65
- 66. Future Plans • Beta release • Grant funding • Streaming data from microscopes • Additional analysis features • Expand compute resources 66
- 67. Acknowledgements • Team : Michael Cianfrocco, Choonhan Youn, Rick Wagner • CIPRES team : Mark Miller, Teri Schwartz, Paul Hoover • SDSC : Nancy Wilkins-‐Diehr & Mike Norman • XSEDE ECSS • SGCI EDS 67
- 68. Thank You! Questions? • mona@sdsc.edu 68