Successfully reported this slideshow.
Mobyle Administrator Workshop              Institut Pasteur 09/28/2012
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Set the environmentstart VirtualBoxFile>import appliance: choose Mobyle.ovaclick Mobyleclick Start
Set up the environmentlogin: mobylemot de passe: mobyleftp://ftp.pasteur.fr/pub/gensoft/projects/mobyle/Mobyle-workshop-su...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Architecture             User                                          Componant                                    Ressou...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
RequirementsRequirements:1. Python (2.5< version<3.0)2. Apache3. libxml2 and its python binding lxml4. simpleTAL (>= 4.1 &...
Requirement installationsudo apt-get install apache2python-lxml python-simpletalpython-pygraphviz squizz(passwd: mobyle)
Mobyle distributionfrom 1.5 version Mobyle exists in two flavors:● Mobyle+BCBB-1.xx.tar.gz .   With BMID (programs editor)...
Download Mobyle distributionwith Firefox go to the Mobyle Download pagehttps://projets.pasteur.fr/projects/mobyle/wiki/dow...
setup.py generalitiessetup.py is the python standard way to buildand install python project.setup.py● build● install--help...
setup.py--install-core=/opt/mobyle--install-htdocs=/var/www/mobyle/htdocs--install-cgis=/var/www/mobyle/cgis--install-bmid...
setup.cfg: a way to automate installation[install]install_core=/opt/mobyleinstall_htdocs=/var/www/mobyle/htdocsinstall_cgi...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Apache: basic configurationWe will set up a Mobyle virtual hostcd /etc/apache2/sites-available/sudo vim mobyleapache confi...
Apache: basic configuration<VirtualHost *:80>                        Do not copy-paste directly from the slides,    Docume...
Apache: basic configuration  ScriptAlias /sitemap.xml /var/www/mobyle/cgis/sitemap.py  ScriptAlias /cgi-mobyle/ /var/www/m...
Apache: startActivate modulessudo a2enmod rewrite headersActivate Mobyle virtual hostsudo a2dissite 000-defaultsudo a2ensi...
Set permissions for ApacheTo enable the access to data and log directories from Apache you have tomake them writable to th...
Mobyle: basic configurationcd /opt/mobylesudo cp Example/Local/Config/Config.template.py  Local/Config/Config.pysudo vim L...
Apache: advanced config & security                                          Do not copy-paste directly from the slides,Rew...
Mobyle portal is readyin Firefox, connect to your Mobyle instance:http://localhost/cgi-bin/portal.py
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Services deploymentMOBYLEHOME/Services/                 MOBYLEHOME/Local/Services/            |_ Programs                 ...
How to deployMobyle configuration:                        LOCAL_DEPLOY_INCLUDE = { programs : [ * ] ,                     ...
Deploy services on your server!●      download from the FTP release folderswget   ftp://ftp.pasteur.fr/pub/gensoft/project...
Deploy services on your server!●  extract the files from the archivestar -xf Programs-5.0.tgztar -xf Viewers-1.0.1.tar.gzt...
Deploy services on your server!●  configure deploymentsudo vim /opt/mobyle/Local/Config/Config.pyLOCAL_DEPLOY_INCLUDE = { ...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Connect Mobyle to an execution system ● By default Mobyle executes jobs on local   system. ● Mobyle can execute jobs on cl...
Mobyle: execution system configuration                                           config system 1                          ...
set up execution systemEXECUTION_SYSTEM_ALIAS = {             DRMAA_sge : SgeDRMAAConfig(                           /path/...
set up execution systemDISPATCHER = DefaultDispatcher( { clustalw : ( EXECUTION_SYSTEM_ALIAS[ DRMAA_sge ] , mobyle ), clus...
binary pathIn Mobyle configuration we can add some path to the general web server PATH. BINARY_PATH = [ "/usr/bin", "usr/l...
Set BINARY_PATH on your serverTo run correctly PHYLIP programs on your server:sudo vim /opt/mobyle/Local/Config/Config.pyB...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Control access to servicesThere are several levels of control to theaccess to programs or workflows.● all services, a port...
Control access to servicesDisable/enable all the portal:   #DISABLE_ALL = False   DISABLE_ALL = True       The list of ser...
Control access to servicesRestrict the access to one programs or workflows to oneIP or a set of IPAUTHORIZED_SERVICES = {h...
Control access to servicesThe last way to control the access is to black list a useremail or an ip.The black_list.py is lo...
Data helpers: Bank configurationembl:{       dataType : Sequence ,       bioTypes : [Nucleic,DNA] ,       label    : EMBL ...
Data helpers: Bank configurationsudo vim /opt/mobyle/Local/Config/Config.pyDATABANKS_CONFIG = {   imgt:{       dataType:Se...
Data helpers: format detector/convertor sudo vim /opt/mobyle/Local/Config/Config.py DATA_CONVERTER={     Sequence: [ squiz...
SessionsThere is 2 kind of sessions:● Authenticated● AnonymousAuthenticated session allow user to retrieve Itsuser space a...
Session configuration Anonymous Session# no : the anonymous sessions are not allowed# yes : the anonymous sessions are all...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
New tools integration● Command line based programs are integrated in  Mobyle with the help of an XML file.● This file is u...
Grammar● The format of the XML files for Mobyle services is  defined by schema definitions (for programs and other  servic...
Program XML structure<program>     <head>PROGRAM HEADER, CONTAINING GENERALINFORMATION  </head>  <parameters>LIST OF INPUT...
Program XML: header <head>   <name>muscle</name>   <version>3.8.31</version>   <doc>     <title>Muscle</title>     <descri...
Program XML: an input parameter          <parameter issimple="1" ismandatory="1">           <name>sequence</name>         ...
Program XML: another "simple"parameter<parameter>  <name>maxiters</name>  <prompt lang="en">Maximum number of iterations (...
Program XML: an output parameter <parameter isstdout="1">   <name>muscleHtmlout</name>   <prompt lang="en">Alignment</prom...
Program XML: paragraphs<paragraph>  <name>inputs</name>  <prompt lang="en">Inputs options</prompt>  <parameters>    <param...
Program XML: type and format<type>  <datatype>                       Datatypes are the base of interoperability    <class>...
How to debug a program interface● mobvalid checks that the XML obeys the grammar    rules.●   But it does not check the em...
build_logTo activate the build_log set DEBUG or better PARTICULAR_DEBUG to 2 or 3 ● DEBUG=2 build the command line but not...
build_log########################## validation beginning              ##########################------------- sleep_out --...
build_log###################### xml controls beginning ######################------------- aalpha -------------service.pre...
build_log--------------- slept ---------------commandIsInserted Trueservice.getArgpos( paramName ) 10rawVdef = Noneconvert...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Sharing services with MobyleNetMobyle provides a technical solution to enable the sharing ofservices (programs, workflows)...
Import/Export Mobyle services with  MobyleNetPORTAL_NAME = "moi_meme"                                          Defines the...
Import/Export Mobyle services withMobyleNetsudo vim /opt/mobyle/Local/Config/Config.pyPORTAL_NAME = "training_session_MYNA...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Mobyle: Daily maintenanceTo supervise a Mobyle server there are:  ● logs  ● some mob* tools
Mobyle: Daily maintenanceThe logs are located in LOGDIR.By default, there are 3 files logs:● access_log: the job launched●...
access_log                  job/workflow name               user email               submission portalFri, 24 Feb 2012 09:...
error_logMobyle.MobyleJob : WARNING : MobyleJob.py: L 717 : Fri, 30 Mar 2012 10:57:56 : PotentialCollision: [/htdocs/data/...
child_log------------------- extend_align : T08716220794916 -------------------commlib returns cant find connectionerror: ...
mobjobw    To supervise the current active jobs:--------------------------------------------------------------------------...
mobclean● clean job and sessions.● never clean authenticated sessions.● clean jobs which are finished from more than      ...
mobkillTo kill an active job.It use the execution informations to call the right method(cluster, local, ...) to kill the j...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Link my web application to MobyleIt is possible with Mobyle to link from another webapplication so that you open the porta...
Link my web application to MobyleAn example file is available in: $MOBYLEHOME/Doc/MobLinkExample.html<form action="/cgi-bi...
Link my web application to Mobyletest it:sudo cp /opt/mobyle/Doc/MobLinkExample.html/var/www/mobyle/htdocs/You can also us...
Overview●   Mobyle architecture.●   Installation.●   Mobyle and Apache configuration.●   How to deploy new services.●   Ho...
Referencing● sitemaps is a protocol/format to facilitate the referencing    of a site by declaring the URLs published by t...
StatisticsIn addition to the access_log, you can create usagestatistics with the help of Google Analytics:
StatisticsTo use Google Analytics with Mobyle, just:● Create an account on Google Analytics● Configure your portal in Conf...
Upcoming SlideShare
Loading in …5
×

Mobyle administrator workshop

5,669 views

Published on

training session on the setup/configuration of Mobyle, as well as the development of new program wrappers/viewers/workflows.

  • Be the first to comment

  • Be the first to like this

Mobyle administrator workshop

  1. 1. Mobyle Administrator Workshop Institut Pasteur 09/28/2012
  2. 2. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  3. 3. Set the environmentstart VirtualBoxFile>import appliance: choose Mobyle.ovaclick Mobyleclick Start
  4. 4. Set up the environmentlogin: mobylemot de passe: mobyleftp://ftp.pasteur.fr/pub/gensoft/projects/mobyle/Mobyle-workshop-supplement.tar.gz
  5. 5. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  6. 6. Architecture User Componant Ressources accessible by the web server Web Ressources accessible by cluster nodesWeb Portal (static + cgis) Computational ResourcesMobyle Core Librairies submissionServices Definitions (Execution nodes)(submission mode)Mobyle Persistance Layer Biological Data Bioinformatics Banks softwares- Users Spaces- Data- Jobs
  7. 7. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  8. 8. RequirementsRequirements:1. Python (2.5< version<3.0)2. Apache3. libxml2 and its python binding lxml4. simpleTAL (>= 4.1 & <5.0)5. simplejson6. graphviz and its python binding pygraphvizoptional1. squizz. (sequence format detector/converter)2. golden. (bank indexer and retriever)3. dnspython. (check user email server)4. Python Imaging Library. (captcha)5. a Distributed Ressources Manager SGE,torque,... with drmaa + python-drmaa.
  9. 9. Requirement installationsudo apt-get install apache2python-lxml python-simpletalpython-pygraphviz squizz(passwd: mobyle)
  10. 10. Mobyle distributionfrom 1.5 version Mobyle exists in two flavors:● Mobyle+BCBB-1.xx.tar.gz . With BMID (programs editor) and BMPS(user graphical workflows)● Mobyle-1.xx.tar.gz . Without BMID (programs editor) and BMPS(user graphical workflows)
  11. 11. Download Mobyle distributionwith Firefox go to the Mobyle Download pagehttps://projets.pasteur.fr/projects/mobyle/wiki/downloador directly with Firefox or wgetftp://ftp.pasteur.fr/pub/gensoft/projects/mobyle/Mobyle+BCBB-1.5-RC1.tar.gzand extract the archivetar -xzf Mobyle+BCBB-1.5-RC1.tar.gzgo to this directorycd Mobyle+BCBB-1.5-RC1
  12. 12. setup.py generalitiessetup.py is the python standard way to buildand install python project.setup.py● build● install--help to get the list of available optionspython setup.py install --help
  13. 13. setup.py--install-core=/opt/mobyle--install-htdocs=/var/www/mobyle/htdocs--install-cgis=/var/www/mobyle/cgis--install-bmid--install-bmpssudo setup.py install --install-core=/opt/mobyle --install-htdocs=/var/www/mobyle/htdocs --install-cgis=/var/www/mobyle/cgis --install-bmid--install-bmps
  14. 14. setup.cfg: a way to automate installation[install]install_core=/opt/mobyleinstall_htdocs=/var/www/mobyle/htdocsinstall_cgis=/var/www/mobyle/cgisinstall_bmid=Trueinstall_bmps=Truesudo python setup.py install
  15. 15. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  16. 16. Apache: basic configurationWe will set up a Mobyle virtual hostcd /etc/apache2/sites-available/sudo vim mobyleapache configuration file for Mobyle virtual host: https://projets.pasteur.fr/projects/mobyle/filesapache .htaccess file for Mobyle https://projets.pasteur.fr/projects/mobyle/files
  17. 17. Apache: basic configuration<VirtualHost *:80> Do not copy-paste directly from the slides, DocumentRoot /var/www/mobyle/htdocs as it will insert invisible control characters in the text file which cause Apache errors. <Directory "/var/www/mobyle"> You can directly use the text version of Options FollowSymLinks this file available in Mobyle-workshop- supplement.tar.gz AllowOverride Limit FileInfo </Directory> <Directory "/var/www/mobyle/htdocs"> Options Indexes MultiViews FollowSymLinks DirectoryIndex index.xml index.html Order allow,deny allow from all </Directory>
  18. 18. Apache: basic configuration ScriptAlias /sitemap.xml /var/www/mobyle/cgis/sitemap.py ScriptAlias /cgi-mobyle/ /var/www/mobyle/cgis/ ScriptAlias /cgi-bin/ /var/www/mobyle/cgis/ <Directory "/var/www/mobyle/cgi-bin"> AllowOverride None Options FollowSymLinks Order allow,deny Allow from all </Directory> ErrorLog "/var/log/apache2/mobyle_error_log" CustomLog "/var/log/apache2/mobyle_access_log" common</VirtualHost>
  19. 19. Apache: startActivate modulessudo a2enmod rewrite headersActivate Mobyle virtual hostsudo a2dissite 000-defaultsudo a2ensite mobyleStart Apachesudo service apache2 restart
  20. 20. Set permissions for ApacheTo enable the access to data and log directories from Apache you have tomake them writable to the "Apache user" (www-data on Ubuntu)sudo chown -R www-data /var/www/mobyle/htdocs/datasudo -u www-data mkdir /var/log/mobyle
  21. 21. Mobyle: basic configurationcd /opt/mobylesudo cp Example/Local/Config/Config.template.py Local/Config/Config.pysudo vim Local/Config/Config.pyROOT_URL = "http://localhost/"HTDOCS_PREFIX= ""CGI_PREFIX= "cgi-bin"MAINTAINER= [""]HELP= [""]MAILHOST= "localhost"
  22. 22. Apache: advanced config & security Do not copy-paste directly from the slides,RewriteEngine on as it will insert invisible control characters#Do not show hidden files content in the text file which cause Apache errors. You can directly use the text version ofRewriteCond %{REQUEST_URI} /. [OR] this file (htaccess) available in Mobyle-RewriteCond %{REQUEST_URI} /ADMINDIR workshop-supplement.tar.gz and copy it as .htaccess in MobyleRewriteRule .* - [F,L] htdocs folder#allow saving resultsRewriteCond %{REQUEST_URI} ^/data/jobs(.*)RewriteCond %{QUERY_STRING} ^save$RewriteRule (.*)/([^/]+)$ $1/$2 [E=SAVEDFILENAME:$2]Header set Content-Disposition "attachment; filename="%{SAVEDFILENAME}e"" env=SAVEDFILENAME
  23. 23. Mobyle portal is readyin Firefox, connect to your Mobyle instance:http://localhost/cgi-bin/portal.py
  24. 24. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  25. 25. Services deploymentMOBYLEHOME/Services/ MOBYLEHOME/Local/Services/ |_ Programs |_ Programs | |_ Entities | |_ Entities | |_ *.xml | |_ Env |_ Workflows | |_ *.xml | |_ Entities |_ Workflows | |_*.xml | |_ Entities |_ Viewers | |_ Env | |_ viewer1.xml | |_*.xml | |_ viewer |_ Viewers |_ Tutorials | |_ viewer1.xml |_ tutorial1.xml | |_ viewer |_ tutorial |_ Tutorials |_ tutorial1.xml |_ tutorial
  26. 26. How to deployMobyle configuration: LOCAL_DEPLOY_INCLUDE = { programs : [ * ] , workflows: [ * ] , viewers : [ * ] , tutorials : [ * ] , } LOCAL_DEPLOY_EXCLUDE = { programs : [ ] , workflows: [ ] , viewers : [ ] , tutorials : [ ] , }tool deployment: mobdeploy -s local -p all deploy mobdeploy -s local -p prog1,prog2 deploy mobdeploy deploy mobdeploy -s local -t all deploy
  27. 27. Deploy services on your server!● download from the FTP release folderswget ftp://ftp.pasteur.fr/pub/gensoft/projects/mobyle/Programs-5.0.tgzwget ftp://ftp.pasteur.fr/pub/gensoft/projects/mobyle/Tutorials-1.5.tgzwget ftp://ftp.pasteur.fr/pub/gensoft/projects/mobyle/Workflows-1.0.0.tar.gzwget ftp://ftp.pasteur.fr/pub/gensoft/projects/mobyle/Viewers-1.0.1.tar.gz
  28. 28. Deploy services on your server!● extract the files from the archivestar -xf Programs-5.0.tgztar -xf Viewers-1.0.1.tar.gztar -xf Workflows-1.0.0.tar.gztar -xf Tutorials-1.5.tgz● move them to the right placessudo mv Programs-5.0/*.xml /opt/mobyle/Services/Programs/sudo mv Programs-5.0/Entities /opt/mobyle/Services/Programs/sudo mv Programs-5.0/Env /opt/mobyle/Local/Services/Programs/Envsudo mv Workflow-1.0.0/*.xml /opt/mobyle/Services/Workflows/sudo mv Workflow-1.0.0/Env /opt/mobyle/Local/Services/Workflows/sudo mv Viewers-1.0.1/*.xml /opt/mobyle/Services/Viewers/sudo mv Tutorials-1.5/*.xml /opt/mobyle/Services/Tutorials/
  29. 29. Deploy services on your server!● configure deploymentsudo vim /opt/mobyle/Local/Config/Config.pyLOCAL_DEPLOY_INCLUDE = { programs : [ * ] , workflows: [ * ] , viewers : [ * ] , }LOCAL_DEPLOY_EXCLUDE = { programs : [ mafft ] , workflows: [ ] , viewers : [ ] , }● deploysudo -u www-data /opt/mobyle/Tools/mobdeploy deploy
  30. 30. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  31. 31. Connect Mobyle to an execution system ● By default Mobyle executes jobs on local system. ● Mobyle can execute jobs on cluster. ○ supported DRMs: sge, open grid scheduler, torque, lsf. ● Mobyle interact with drms via libdrmaa. ● Mobyle may interact with severals drms at same time.
  32. 32. Mobyle: execution system configuration config system 1 Execution System 1 Web Portal config system 2 and Distpatcher Execution System 2 Core library config system 3 Execution System 3 3 actors: ● ExecutionConfig: each system need to have a configuration ● EXECUTION_SYSTEM_ALIAS: give a name to an execution system ● DISPATCHER: aim to route jobs to an execution system
  33. 33. set up execution systemEXECUTION_SYSTEM_ALIAS = { DRMAA_sge : SgeDRMAAConfig( /path/to/sge/libdrmaa.so , root = $SGE_ROOT, cell = default ) , DRMAA_torque: PbsDRMAAConfig( /path/to/pbs/libdrmaa.so , hostname ), LSF : LsfDRMAAConfig( /path/to/LSF/libdrmaa.so , lsf_envdir = $LSF_ENVDIR , lsf_serverdir =$LSF_SERVERDIR), SYS : SYSConfig() , }
  34. 34. set up execution systemDISPATCHER = DefaultDispatcher( { clustalw : ( EXECUTION_SYSTEM_ALIAS[ DRMAA_sge ] , mobyle ), clustalo : ( EXECUTION_SYSTEM_ALIAS[ DRMAA_torque ] , mobyle ), toppred : ( EXECUTION_SYSTEM_ALIAS[ DRMAA_torque ] , short ), blast2: ( EXECUTION_SYSTEM_ALIAS[ DRMAA_sge ] , long ) DEFAULT : ( EXECUTION_SYSTEM_ALIAS[ SYS ] , )})For the workshop we wont use a cluster, so we will execute all jobs on localsystem.DISPATCHER = DefaultDispatcher( { DEFAULT : ( EXECUTION_SYSTEM_ALIAS[ SYS ] , )})
  35. 35. binary pathIn Mobyle configuration we can add some path to the general web server PATH. BINARY_PATH = [ "/usr/bin", "usr/local/bin", "/local/Bioinfo/bin" ]instead of tag env in programs/workflows descriptions,● this new path is available for all services,● these locations are added to the PATH (do not replace it), their order is preserved.In debian/ubuntu the phylip package binaries are installed in "/usr/lib/phylip/bin"and all binaries are accessible through the phylip wrapper, e.g. protdist -> phylipprotdistso we must either:● modify each interfaces belonging to Phylip package (interfaces)● add this specific path in our Mobyle PATH
  36. 36. Set BINARY_PATH on your serverTo run correctly PHYLIP programs on your server:sudo vim /opt/mobyle/Local/Config/Config.pyBINARY_PATH = ["/usr/local/bin", "/usr/lib/phylip/bin"]
  37. 37. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  38. 38. Control access to servicesThere are several levels of control to theaccess to programs or workflows.● all services, a portal, a specific services● for every one, for some ip only● and finally forbid the access to Mobyle to one ip or one email.
  39. 39. Control access to servicesDisable/enable all the portal: #DISABLE_ALL = False DISABLE_ALL = True The list of services is empty The job submission is disabledDisable a service: DISABLED_SERVICES = [ local.blast2, clustalw*, genouest.blast2 ]Disable a portal: DISABLED_SERVICES =[ genouest.*]
  40. 40. Control access to servicesRestrict the access to one programs or workflows to oneIP or a set of IPAUTHORIZED_SERVICES = {http://mobyle.pasteur.fr/data/services/servers/local/programs/netNglyc.xml : [ 157.99.*.* ]} ○ The service will appear only for the users in the subnet ○ Only the users in the subnet can submit this job
  41. 41. Control access to servicesThe last way to control the access is to black list a useremail or an ip.The black_list.py is located in MOBYLEHOME/Localusers = [ blub@web.de, kt7mail@gmail.com, caca@crotte.fr, *@aaa*.com , toto@*, titi@*, *@bidule.fr, ]host = [ 142.161.25.184, 141.161.25.102]a message will appear to the user:you have abused our service. Your are not allowed to run on this server for now.For more informations contact mobyle@pasteur.fr".This message is customizable in function emailUserMessagelocated in Local/Policy.py file
  42. 42. Data helpers: Bank configurationembl:{ dataType : Sequence , bioTypes : [Nucleic,DNA] , label : EMBL Nucleotide Sequence Database, command : [/usr/local/bin/golden, %(db)s:%(id)s] },genbank:{ dataType : Sequence, bioTypes : [Nucleic,DNA], label : Genbank NIH DNA sequence database, command: [/usr/bin/seqret, %(db)s:%(id)s -osformat2 fasta -auto -stdout] },fasta:{ dataType:Sequence, label: GenOuest Data Banks, command: [ "/opt/mongo/mongo.pl", "%(id)s" ] }}
  43. 43. Data helpers: Bank configurationsudo vim /opt/mobyle/Local/Config/Config.pyDATABANKS_CONFIG = { imgt:{ dataType:Sequence, bioTypes:[DNA], label: IMGT, command: [golden, %(db)s:%(id)s] }, uniprot_sprot:{ dataType:Sequence, bioTypes:[Protein], label: SWISSPROT, command: [golden, %(db)s:%(id)s] }}
  44. 44. Data helpers: format detector/convertor sudo vim /opt/mobyle/Local/Config/Config.py DATA_CONVERTER={ Sequence: [ squizz_sequence(/usr/bin/squizz) ] , Alignment: [ squizz_alignment(/usr/bin/squizz)] }
  45. 45. SessionsThere is 2 kind of sessions:● Authenticated● AnonymousAuthenticated session allow user to retrieve Itsuser space at each session of workIn Anonymous session the user will not able toretrieve its user space after closing its webbrowser. Even if he set his email, the email isjust used to communicate with the him.
  46. 46. Session configuration Anonymous Session# no : the anonymous sessions are not allowed# yes : the anonymous sessions are allowed, without any verification# captcha : the anonymous sessions are allowed, but with a captcha challenge ( default )ANONYMOUS_SESSION = "captcha" Authenticated Session # no : the authenticated session are not allowed. # yes : the authenticated session are allowed and activated without any restriction. # email : the authenticated session are allowed but an email confirmation is needed to activate it (default). AUTHENTICATED_SESSION = "email"
  47. 47. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  48. 48. New tools integration● Command line based programs are integrated in Mobyle with the help of an XML file.● This file is used to: ○ generate the user interface (form) ○ transform a form submission into a command line call ○ capture the results ○ display the results in the user interface (job page) ○ index the program (enabling classification and search)
  49. 49. Grammar● The format of the XML files for Mobyle services is defined by schema definitions (for programs and other services), stored in $MOBYLEHOME/Schema,● A service cannot be deployed if the XML file does not respect this format,● this is a partial safeguard against unpredictable behaviors which can occur if the program is not correctly described,● you also need to be careful with the python code used throughout the XML description to validate the data, compute preconditions, and generate the command line.
  50. 50. Program XML structure<program> <head>PROGRAM HEADER, CONTAINING GENERALINFORMATION </head> <parameters>LIST OF INPUT AND OUTPUT PARAMETERS, WHICHCAN BE NESTED IN PARAGRAPHS </parameters></program>
  51. 51. Program XML: header <head> <name>muscle</name> <version>3.8.31</version> <doc> <title>Muscle</title> <description> <text lang="en">MUSCLE is a program for creating multiple alignments of aminoacid or nucleotide sequences.</text> </description> <authors>Edgar, R.C.</authors> <reference doi="10.1093/nar/gkh340">Edgar, Robert C. (2004), MUSCLE: multiplesequence alignment with high accuracy and high throughput, Nucleic Acids Research 32(5),1792-97.</reference> <doclink>http://www.drive5.com/muscle/</doclink> <homepagelink>http://www.drive5.com/muscle/</homepagelink> <sourcelink>http://www.drive5.com/muscle/downloads.htm</sourcelink> </doc> <category>alignment:multiple</category> <command>muscle</command> </head>
  52. 52. Program XML: an input parameter <parameter issimple="1" ismandatory="1"> <name>sequence</name> <prompt lang="en">Sequences (-in)</prompt> <type> <datatype> <class>Sequence</class> </datatype> <dataFormat>FASTA</dataFormat> </type> <precond> <code proglang="perl">not defined($profile1) and not defined($profile2)</code> <code proglang="python">profile1 is None and profile2 is None</code> </precond> <format> <code proglang="perl">"-in $value"</code> <code proglang="python">" -in " + str(value)</code> </format> <argpos>10</argpos> </parameter>
  53. 53. Program XML: another "simple"parameter<parameter> <name>maxiters</name> <prompt lang="en">Maximum number of iterations (-maxiters)</prompt> <type><datatype><class>Integer</class></datatype></type> <vdef><value>16</value></vdef> <format> <code proglang="perl">(defined $value and $value != $vdef) ? " -maxiters $value" : ""</code> <code proglang="python">( "" , " -maxiters " + str( value ) )[ valueis not None and value != vdef]</code> </format> <comment> <text lang="en">You can control the number of iterations that MUSCLEdoes by specifying the -maxiters option.</text> [...] </comment></parameter>
  54. 54. Program XML: an output parameter <parameter isstdout="1"> <name>muscleHtmlout</name> <prompt lang="en">Alignment</prompt> <type> <datatype> <class>MuscleHtmlAlignment</class> <superclass>AbstractText</superclass> </datatype> </type> <precond> <code proglang="perl">$outformat == html </code> <code proglang="python">outformat == html</code> </precond> <filenames> <code proglang="perl">(defined $outfile) ? "$outfile" : "muscle.out"</code> <code proglang="python">( outfile , "muscle.out")[outfile is None]</code> </filenames> </parameter>
  55. 55. Program XML: paragraphs<paragraph> <name>inputs</name> <prompt lang="en">Inputs options</prompt> <parameters> <parameter issimple="1" ismandatory="1"> <name>sequence</name> [...] </parameter> <parameter> <name>seqtype</name> [...] </parameter> </parameters></paragraph>
  56. 56. Program XML: type and format<type> <datatype> Datatypes are the base of interoperability <class>Sequence</class> between the services and also between </datatype> services and "helpers": <dataFormat>FASTA</dataFormat></type> For "file" types (Sequence, Alignment, Structure, etc): ● the source datatype has to be identicalBioTypes:* DNA, RNA, Protein or included (subclass of) the target datatype ● the source format has to be included inExamples of formats:* FASTA, PDB, CLUSTAL, etc. the possible formats of the target ● the source biotype has to be included in the possible biotypes of the target Datatypes are also used to check the validity of "simple" types (Integer, String, etc.)
  57. 57. How to debug a program interface● mobvalid checks that the XML obeys the grammar rules.● But it does not check the embedded python code neither the syntax nor the logic.(mandatory parameter, format, precond, control, ...)● These 2 aspects can be inspected with the build_log.● The buil_log log all steps performed by Mobyle to build the unix command line.● The build_log as only purpose to help us to debug a program xml, its very verbose and useless in production.
  58. 58. build_logTo activate the build_log set DEBUG or better PARTICULAR_DEBUG to 2 or 3 ● DEBUG=2 build the command line but not execute it. It allow us to debug an xml even we have not the corresponding binary on the computer, or the service is very time or cpu consuming. ● DEBUG=3 do the same thing except it execute the command line.ensure that only you can use the service. Otherwise all jobs logs will be mixed.(You can use RESTRICT_ACCESS for this)example sleep.xml--------------------- MobyleJob set user value for time--------------------self._service.setValue( time , 20.0 )--------------------- MobyleJob set user value for suffix--------------------self._service.setValue( suffix , s )
  59. 59. build_log########################## validation beginning ##########################------------- sleep_out -------------value= None : type= <type NoneType>service.precondHas_proglang( sleep_out , python ) = Nonevalue is Nonecall service.validate( sleep_out )check if the Parameter have a secure filenamefilename= sleep.out safeMask = sleep.outfilename = sleep.out ...........OKcheck if the Parameter have a secure paramfileno paramfile
  60. 60. build_log###################### xml controls beginning ######################------------- aalpha -------------service.precondHas_proglang( aalpha , python ) = NoneconvertedVdef = 1e-07 value = 1e-07eval( value>= 0 and value <=1 ) = Truehas scale= False------------- nb_expe_4 -------------service.precondHas_proglang( nb_expe_4 , python ) = Trueprecond= SAXS_4 is not Noneeval precond= Falsenext parameter
  61. 61. build_log--------------- slept ---------------commandIsInserted Trueservice.getArgpos( paramName ) 10rawVdef = NoneconvertedVdef = NonemyEvaluator.setVar( vdef , None )myEvaluator.isDefined( slept ) = FalserawVdef = NoneconvertedVdef = NonemyEvaluator.setVar( value , None )service.formatHas_proglang( slept , python ) = Truevalue = None type = <type NoneType>vdef = None type = <type NoneType>format = " && echo "I slept %f %s""%( time , suffix )commandLine = sleep 20.0 && echo "I slept 20.000000 s"------------ end of parameter loop -------------PATH= /bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/binEnvironment ={}command line= sleep 20.0 && echo "I slept 20.000000 s"
  62. 62. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  63. 63. Sharing services with MobyleNetMobyle provides a technical solution to enable the sharing ofservices (programs, workflows) between multiple Mobyleservers.
  64. 64. Import/Export Mobyle services with MobyleNetPORTAL_NAME = "moi_meme" Defines the information sent toPORTALS={ remote execution portals to identify yourself ami_1:{ url: http://ami1.fr/cgi-bin/, help : support@ami1.fr, repository: http://ami1.fr/, Defines the list of remote services services: { programs :[golden,dnapars,boxshade,protpars ], that can be run from your portal workflows:[ workflow_phylogeny ] } }, ami_2:{ url:http://ami2.fr/mobyle/cgi-bin/, help : support@ami2.fr, repository : http://ami2.fr/mobyle, services: {programs:[protpars] } }} Defines the list of "your" which you allow to be run from remote MobyleEXPORTED_SERVICES = [ abiview,toppred ] servers
  65. 65. Import/Export Mobyle services withMobyleNetsudo vim /opt/mobyle/Local/Config/Config.pyPORTAL_NAME = "training_session_MYNAME"PORTALS={ pasteur:{ url:http://mobyle.pasteur.fr/cgi-bin/, help:mobyle@pasteur.fr, repository:http://mobyle.pasteur.fr/, services: {programs: [protpars]} }}sudo -u www-data /opt/mobyle/Tools/mobdeploy deploy
  66. 66. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance● Link my web application to Mobyle● Referencing and statistics
  67. 67. Mobyle: Daily maintenanceTo supervise a Mobyle server there are: ● logs ● some mob* tools
  68. 68. Mobyle: Daily maintenanceThe logs are located in LOGDIR.By default, there are 3 files logs:● access_log: the job launched● error_log: the errors● child_log: the uncaught error produced by detached scripts.and depending of the DEBUG levelbuild_log: the building of the unix commandline
  69. 69. access_log job/workflow name user email submission portalFri, 24 Feb 2012 09:41:27 pratt F04027998503923 bneron@pasteur.fr 157.9.0.1 musky-devTue, 28 Feb 2012 13:34:30 mafft-cons-tree N21174955267906 bneron@pasteur.fr 157.9.0.1 musky-devTue, 28 Feb 2012 13:34:30 mafft P21175159178972 bneron@pasteur.fr 157.9.0.1 musky-devTue, 28 Feb 2012 13:34:31 quicktree C21175509293079 bneron@pasteur.fr 157.9.0.1 musky-devTue, 28 Feb 2012 13:34:31 cons D21175620271921 hmenager@pasteur.fr 157.9.0.2 rita-branche job submission date job/workflow key user address
  70. 70. error_logMobyle.MobyleJob : WARNING : MobyleJob.py: L 717 : Fri, 30 Mar 2012 10:57:56 : PotentialCollision: [/htdocs/data/jobs/mafft/J12393850061893/mafft.out] input files match the "result" outputparameter mask /htdocs/data/jobs/mafft/J12393850061893/mafft.outMobyle.Session.AnonymousSession : ERROR : Session.py: L 491 : Thu, 29 Mar 2012 14:06:57 :session/J05040996510983 : the data af50ab3b273d6f3bd97b396ed3771bc7.data ( 3388999 ) cannotbe added because the session size exceed the session limit ( 1048576 )Mobyle.Execution.DRMAA : CRITICAL : DRMAA.py : L 281 : Thu, 22 Mar 2012 13:40:50 : error duringdrmaa intitialization for job 4: code 2: unable to contact qmaster using port 6444 on host "musky"Traceback (most recent call last):File "mobyle/Src/Mobyle/Execution/DRMAA.py", line 277, in getStatuss.initialize()...raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value))DrmCommunicationException: code 2: unable to contact qmaster using port 6444 on host "musky"
  71. 71. child_log------------------- extend_align : T08716220794916 -------------------commlib returns cant find connectionerror: unable to contact qmaster using port 6444 on host "musky"------------------- dnadist : P23432022073984 -------------------------------------- mafft : R24219133367062 -------------------------------------- neighbor : Y28504744119883 -------------------
  72. 72. mobjobw To supervise the current active jobs:--------------------------------------------------------------------------------V06795772934914 -- SGE_DRMAA/mobyle -- runningmorePhyML -- toto@ed.ac.uk -- 09/25/12 16:45:19 -- LOCAL -- STANDALONE -- 130.250.199.199UNKNOWNmorePhyML -i aln.phylipi -d aa -a e -f m -u aln.phylipi.tree.txt--------------------------------------------------------------------------------Y14426818867922 -- SGE_DRMAA/mobyle -- pendingblast2 -- titi@gmail.com -- 09/26/12 07:40:05 -- genouest -- STANDALONE -- 14.199.100.4UNKNOWNblastall -p blastp -d uniprot -e 0.001 -i my_sequence.fasta--------------------------------------------------------------------------------Z69834269642389 -- SGE_DRMAA/mobyle -- runningmuscle--titi@gmail.com--09/26/12 07:40:05--LOCAL--blast_to_multialign/V87685541428494 -- 14.199.100.4UNKNOWNmuscle -quiet -in sequence.data--------------------------------------------------------------------------------V87685541428494 -- SGE_DRMAA/mobyle -- runningblast_to_multialign-- titi@gmail.com -- 09/26/12 07:40:05 --LOCAL-- STANDALONE--14.199.100.4UNKNOWNno command line
  73. 73. mobclean● clean job and sessions.● never clean authenticated sessions.● clean jobs which are finished from more than days RESULT_REMAIN● clean anonymous sessions when they not reference jobs anymore.mobcleanremove job older than REMAIN_RESULTS and sessions which not reference jobs anymore.mobclean -v -l /path/mobclean_logsame as above but the output wiil be verbose and redirected in mobclean_log filemobclean -j -d 12 -nperform a dry run on jobs older than 12 days, doe not do anything on sessions.
  74. 74. mobkillTo kill an active job.It use the execution informations to call the right method(cluster, local, ...) to kill the job.mobkill V06795772934914 Y14426818867922 V87685541428494If the job is a workflow, kill all subtasks.
  75. 75. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance.● Link my web application to Mobyle.● Referencing and statistics
  76. 76. Link my web application to MobyleIt is possible with Mobyle to link from another webapplication so that you open the portal with a pre-filledform.e.g.:● from a databank browser, you can give the possibility to the user to send his selection directly to a program (or a workflow to Mobyle),● The Mobyle portal will directly open the corresponding form prefilled with the set of values you ask for,● The user can then modify further this predefined configuration if he wishes to, and launch the analysis.
  77. 77. Link my web application to MobyleAn example file is available in: $MOBYLEHOME/Doc/MobLinkExample.html<form action="/cgi-bin/portal.py" method="post" enctype="multipart/form-data"target="_blank"><div><label>Here is the db that has to be set in golden prefilled form<input name="load::golden::db" value="uniprot"></label></div><div><label>Here is the db that has to be set in golden prefilled form<input name="load::golden::query" value="104K_THEPA"></label></div><input type="submit" name="Open" value="open"></div></form>
  78. 78. Link my web application to Mobyletest it:sudo cp /opt/mobyle/Doc/MobLinkExample.html/var/www/mobyle/htdocs/You can also use links that perform the same HTTPrequest (see the example file), but it is far less elegantsince the parameters remain in the portal URL.
  79. 79. Overview● Mobyle architecture.● Installation.● Mobyle and Apache configuration.● How to deploy new services.● How to connect Mobyle to an execution system.● Mobyle configuration.● New tools integration. ○ concepts (grammar, typing, validation,...). ○ BMID tutorial.● Share services between Mobyle servers.● Maintenance.● Link my web application to Mobyle.● Referencing and statistics.
  80. 80. Referencing● sitemaps is a protocol/format to facilitate the referencing of a site by declaring the URLs published by the website directly to some referencing entities (search engines),● it is possible to enhance the referencing of your Mobyle server by using the provided sitemap tool of Mobyle: http://localhost/cgi-bin/sitemap.py● it provides the list of programs/workflows which can directly be accessed from the webhttp://www.sitemaps.org/and just search for "sitemap" on any search engine ;)
  81. 81. StatisticsIn addition to the access_log, you can create usagestatistics with the help of Google Analytics:
  82. 82. StatisticsTo use Google Analytics with Mobyle, just:● Create an account on Google Analytics● Configure your portal in Config.pyGACODE = XXXXXXXXXX

×