D Baker - Galaxy Update

635 views

Published on

Presentation at BOSC2012 by D Baker - Galaxy Update

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
635
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • What's the goal of scalability?Should run for a million users like it runs for a single userComplex task for any software Should run for a million users like it runs for a single userComplex task for any software
  • D Baker - Galaxy Update

    1. 1. Galaxy Update Dannon Baker Galaxy Team Emory University
    2. 2. Topics• API• Automatic Parallelization• Tool Shed• ?
    3. 3. API Overview• RESTful API supporting CRUD operations• Uses generated keys for per-user authentication • No username/password • No credential caching (via keys)• Request parameters and responses are in JSON (JavaScript Object Notation)
    4. 4. Quickstart
    5. 5. Raw GET Example» GET /api/histories?key=966354fc14c9e427cee380ef50a72a21 « [ « { « url: /api/histories/d0bfe935d0f5258d, « id: d0bfe935d0f5258d, « name: Demo History 1 « } « ]
    6. 6. Making the CallsWrapper methods exist (in /scripts/api/) to make calls easier:python scripts/api/{action}.py <api key> http://<ip>/api/{module}/[id] [args]action: create | display | update | deleteapi_key: obtained from the UImodule: datasets | forms | histories | libraries | permissions | quotas |requests | roles | samples | tools | users | visualizations | workflowsunit: dataset_id / history_id / library_id / …args: name / key-value pair / …
    7. 7. Sample InvocationsCreate a history ./create.py <api_key> https://localhost:8080/api/histories name="from API”Display all histories ./display.py <api_key> https://localhost:8080/api/historiesDisplay information about a history ./display.py <api_key> https://localhost:8080/api/histories/6b5cf7e7ef797b21View datasets in a given history ./display.py <api_key> https://localhost:8080/api/histories/6b5cf7e7ef797b21/contentsDelete a history ./delete.py <api_key> https://localhost:8080/api/histories/976a9ce09b49502a
    8. 8. End-to-end pipelines• Push data from instruments to Galaxy• Execute a workflow• Associate with a user or add to a Data Library• Export outputs• Or a totally custom interface to Galaxy
    9. 9. Automatic Parallelisminput BLAST/strip? output
    10. 10. Parallelism• Take maximum advantage of available resources• Less costly fault recovery (spot instances)• Overhead in splitting time• Increased temporary storage requirement
    11. 11. Use it nowuse_tasked_jobs = TrueTools supported: BLAST, BWA, Bowtie Yours?
    12. 12. Try it for your tool<parallelism method = "multi” split_inputs = "query" split_mode = "number_of_parts” split_size = "4" shared_inputs = "subject" merge_outputs = "output1”/>
    13. 13. Advanced Splitting• FUSE • No disk write required • But it is slightly slower to read• More splitters: • Chromosome based?
    14. 14. Galaxies on Galaxies on private clouds public clouds private Tool Sheds G a la x y T o o lhttp://usegalaxy.org S he d ... http://usegalaxy.org/community 1 2 3 ... ∞ p r iv a t e G a la x y in s t a lla t io n s
    15. 15. Tool Shed - Developer
    16. 16. Tool Shed - Developer
    17. 17. Tool Shed - User
    18. 18. Tool Shed
    19. 19. Tool Shed
    20. 20. Tool Shed - User
    21. 21. Tool Shed• Simple installation to galaxy of tools, workflows • Dependencies• Automatic update notifications• Multiple Tool Versions installed
    22. 22. Tool Shed References• Main Tool Shed: usegalaxy.org/toolshed• ISMB Tech Track w/ Greg – Sunday, TT10
    23. 23. if more_time:• S3ObjectStore• Documentation • <galaxy_dir>/doc – `make html`• Join us in #galaxyproject on irc.freenode.net• Acknowledgements: • Galaxy Team, Community, people who send pull requests

    ×