Installing and Building GTLAB Marlon Pierce Indiana Univeristy

GTLAB and OGCE OGCE contains multiple sub-projects Portlet-based Grid portal (with GridSphere and Tomcat). Workflow suite (services and add-ins to the portal) Information Web services Gadget container Cyberaide JavaScript libraries GTLAB These are packaged with Maven and include everything you need except Java and (for some services) MySQL. We try to make things installable with minimal fuss. Consistent directory structures across projects Edit one config file (pom.xml) Run one command (mvn clean install) You may need to futz a little with MySQL

OGCE contains multiple sub-projects

Portlet-based Grid portal (with GridSphere and Tomcat).

Workflow suite (services and add-ins to the portal)

Information Web services

Gadget container

Cyberaide JavaScript libraries

GTLAB

These are packaged with Maven and include everything you need except Java and (for some services) MySQL.

We try to make things installable with minimal fuss.

Consistent directory structures across projects

Edit one config file (pom.xml)

Run one command (mvn clean install)

You may need to futz a little with MySQL

Overview and Examples

User’s Browser Portlet, Gadget Containers iGoogle, Facebook GTLAB Gadgets TeraGrid Services (GRAM, GridFTP, INCA) Google GData Services, Twitter Feeds, etc RPS, RDS, GPIR Services GTLAB’s Relationship to Other Components

Various GTLAB applications deployed as portlets: Remote directory browsing, proxy management, and LoadLeveler queues.

GTLAB Applications as Google Gadgets: MOAB dashboard, remote directory browser, and proxy management.

MyProxy Example <h:form id=&quot;form&quot;> <h:outputText value=&quot;Username:&quot;/> <h:inputText value=&quot;#{resource.username}&quot; required=&quot;true&quot;/> <h:outputText value=&quot;Password: &quot;/> <h:inputSecret value=&quot;#{resource.password}&quot; required=&quot;true&quot;/> <h:outputText value=&quot;MyProxy Server: &quot;/> <h:inputText value=&quot;#{resource.myproxyHostname}&quot; required=&quot;true&quot;/> <o:submit id=&quot;submit&quot; value=&quot;Submit&quot; binding=&quot;#{builder.body}&quot; action=&quot;#{builder.junkAction}&quot; > <o:multitask id=&quot;multi&quot; persistent=&quot;true&quot; taskname=&quot;#{resource.taskname}&quot;> <o:myproxy id=&quot;mypr&quot; hostname=&quot;#{resource.myproxyHostname}&quot; lifetime=&quot;2&quot; password=&quot;#{resource.password}&quot; port=&quot;7512&quot; username=&quot;#{resource.username}&quot;/> </o:multitask> </o:submit> </h:form>

Executing Multiple Task <o:submit id=&quot;submit&quot; value=&quot;Submit&quot; binding=&quot;#{builder.body}&quot; action=&quot;#{builder.junkAction}&quot; > <o:multitask id=&quot;multi&quot; persistent=&quot;true&quot; taskname=&quot;#{resource.taskname}&quot;> < o:jobsubmit id=&quot;js&quot; myfaces=&quot;true&quot; arguments=&quot;#{resource.arguments}&quot; executable=&quot;#{resource.executable}&quot; hostname=&quot;#{resource.hostname}&quot; provider=&quot;#{resource.provider}&quot; stdout=&quot;#{resource.stdout}&quot;/> < o:filetransfer id=&quot;ft&quot; myfaces=&quot;true&quot; from=&quot;#{resource.from}&quot; to=&quot;#{resource.to}&quot; /> <o:dependency id=&quot;dep&quot; task=&quot;ft&quot; dependsOn=&quot;js&quot;/> </o:multitask> </o:submit>

Resource Prediction Service Client with GTLAB <o:submit id=&quot;submit&quot; value=&quot;Submit&quot; binding=&quot;#{builder.body}&quot; action=&quot;#{builder.junkAction}&quot;> <o:multitask id=&quot;multi&quot; persistent=&quot;true&quot; taskname=&quot;#{resources.taskname”> <o:rpsOptimal resource=&quot;task1-resources&quot; id=&quot;rpsoptimal-exec&quot; myfaces=&quot;true&quot;/> </o:multitask> </o:submit>

Example with Dependency <o:submit id=&quot;submit” …> <o:multitask id=&quot;multi” …> <o:rpsAddModel resource=&quot;task1-resources&quot; id=&quot;rpsAddModel-exec&quot; myfaces=&quot;true&quot;/> <o:rpsManager resource=&quot;task1-resources&quot; id=&quot;rpsManager-exec&quot; myfaces=&quot;true&quot;/> <o:dependency id=&quot;deps-rpsstuff” task=&quot;rpsManager-exec&quot; dependsOn=&quot;rpsAddModel-exec&quot;/> </o:multitask> </o:submit>

More Detail: Available Tags and Features

GTLAB Features Extends Java Server Faces. Tag components wrap major COG Abstraction Layer features And Web Service clients, GridShib , etc. Supports tag inter-dependencies . Allows you to do standalone development. Use JSF portlet bridge to convert into portlets No new coding, just add/modify XML config files and jars. Process is automated Or forget about portlets Convert into Google Gadgets Develop as a Facebook application Etc.

Extends Java Server Faces.

Tag components wrap major COG Abstraction Layer features

And Web Service clients, GridShib , etc.

Supports tag inter-dependencies .

Allows you to do standalone development.

Use JSF portlet bridge to convert into portlets

No new coding, just add/modify XML config files and jars.

Process is automated

Or forget about portlets

Convert into Google Gadgets

Develop as a Facebook application

Etc.

Getting GTLAB See http://www.collab-ogce.org/ogce/index.php/GTLAB You can use your favorite SVN client to check out. svn co https://ogce.svn.sourceforge.net/svnroot/ogce/GTLAB (latest) svn co https://ogce.svn.sourceforge.net/svnroot/ogce/tags/GTLAB-tg09 (tagged) “ Latest” will give you easy access to any updates “ svn update” Best option if you want to actively develop and get fixes right away.

See http://www.collab-ogce.org/ogce/index.php/GTLAB

You can use your favorite SVN client to check out.

svn co https://ogce.svn.sourceforge.net/svnroot/ogce/GTLAB (latest)

svn co https://ogce.svn.sourceforge.net/svnroot/ogce/tags/GTLAB-tg09 (tagged)

“ Latest” will give you easy access to any updates

“ svn update”

Best option if you want to actively develop and get fixes right away.

No SVN? Get the TAR SourceForge’s SVN/CVS viewer now provides a “Download GNU Tar” option. http://ogce.svn.sourceforge.net/viewvc/ogce/GTLAB/ for latest. http://ogce.svn.sourceforge.net/viewvc/ogce/tags/GTLAB-tg09/ for tag

SourceForge’s SVN/CVS viewer now provides a “Download GNU Tar” option.

http://ogce.svn.sourceforge.net/viewvc/ogce/GTLAB/ for latest.

http://ogce.svn.sourceforge.net/viewvc/ogce/tags/GTLAB-tg09/ for tag

Levels of Abstraction in GTLAB code Note JSF is compatible with JSP, so you can mix and match. And you can embed JavaScript YUI, Scriptaculous user interface goodies. Google and Facebook APIs. You can use prepackaged gadgets/portlets. You can develop new applications using existing tag libraries. You can make new tags for your services You can hack or steal the code

Note JSF is compatible with JSP, so you can mix and match.

And you can embed JavaScript

YUI, Scriptaculous user interface goodies.

Google and Facebook APIs.

Tag Description MyProxy Gets a proxy credential (COG) JobSubmit Runs remote commands (COG) FileOperation Create, delete, list remote files and dirs (COG) FileTransfer GridFTP downloads, 3 rd party transfers (COG) XSubmissionOutput Format outputs of job and file operations ResourceDiscovery Clients for the RDS ResourcePrediction Clients for the RPS MyCommunityProxy Get and decorate TG Community credentials with Grid Shib Community Log Log use of TG Community credentials

These will work as gadgets, portlets, or standalone applications GTLAB Example Description MyProxyExample Get a credential and store in OGCE global proxy store. JobSubmitExamples Various examples showing how to submit a grid job and get output. Dashboard Examples Runs showq on Big Red, Cobalt, and Mercury and formats output. DirectoryBrowser Displays and downloads files on a remote host via GridFTP MultipleTaskExample Shows how to couple file transfers, operations, and

Gadgets and Portlets

GTLAB Gadgets Code Base GTLAB Portlets Code Base Tomcat Web Server Tomcat Web Server + Portlet Container iGoogle Client Aggregator Compile and deploy into server Decorate with bridge and container jars, xml config files. GTLAB gadgets can also run as portlets with no code changes by using the JSF portlet bridge.

Making GTLAB Google Gadgets A Gadget is a standalone application that is integrated into your personal iGoogle display. They can run on your Web server . You can use SSL/HTTPS Google Gadgets have two versions http : these use Google’s JavaScript APIs url : these wrap external applications with Iframes. We currently support “url” gadgets.

A Gadget is a standalone application that is integrated into your personal iGoogle display.

They can run on your Web server .

You can use SSL/HTTPS

Google Gadgets have two versions

http : these use Google’s JavaScript APIs

url : these wrap external applications with Iframes.

We currently support “url” gadgets.

Example Gadget Config File <?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; ?> <Module> <ModulePrefs title=&quot;MyProxy Gadget Example&quot; scrolling=&quot;true&quot; height=&quot;500&quot;/> <Content type=&quot;url&quot; href=&quot;http://…/:8080/GTLAB/examples/MyProxyExample.jsf&quot;> </Content> </Module> Save this as MyProxy.xml and place in a Web accessible place.

Other Gadgets Providers Tomcat + GTLAB Gadgets Grid and Web Services (TeraGrid, OSG, etc) Other Gadgets Providers Social Network Services (Orkut, LinkedIn,etc) RSS Feed, Cloud, etc Services Gadget containers aggregate content from multiple providers . Content is aggregated on the client by the user. Nearly any web application can be a simple gadget (as Iframes)

Click “Add stuff” and then “Add feed or gadget”. Type the URL of your gadget.

GTLAB Applications as Google Gadgets: MOAB dashboard, remote directory browser, and proxy management.

Tomcat + Portlets and Container Grid and Web Services (TeraGrid, OSG, etc) Grid and Web Services (TeraGrid, OSG, etc) Grid and Web Services (TeraGrid, OSG, etc) HTML/HTTP SOAP/HTTP Common science gateway architecture. Aggregation is in the portlet container. Users have limited selections of components.

Making GTLAB Gadgets into Portlets Do all development in jsf_standalone directory This is where your template tags will be placed When you are ready to convert: Download and build the OGCE portal cd GTLAB/transition mvn clean process-resources cd GTLAB/portlets mvn clean install You will need to make small edits to portlet.xml, group.xml, and layout.xml in GTLAB/portlet/src/main/webapp/WEB-INF/ That’s it.

Do all development in jsf_standalone directory

This is where your template tags will be placed

When you are ready to convert:

Download and build the OGCE portal

cd GTLAB/transition

mvn clean process-resources

cd GTLAB/portlets

mvn clean install

You will need to make small edits to portlet.xml, group.xml, and layout.xml in GTLAB/portlet/src/main/webapp/WEB-INF/

That’s it.

Various GTLAB applications deployed as portlets: Remote directory browsing, proxy management, and LoadLeveler queues.

Getting Started with GTLAB

GTLAB Directory Structure

Build GTLAB Unpack or checkout code Cd GTLAB All commands are executed from here. Edit properties at the top of pom.xml. Change IP Change project.home if you unpack someplace besides $HOME. Run “mvn clean install” <properties> <portal.server.ip> 156.56.104.143 </portal.server.ip> <host.base.url> http://${portal.server.ip}:8080/ </host.base.url> <project.home> ${env.HOME}/GTLAB </project.home> <tomcat.version> apache-tomcat-5.5.27 </tomcat.version> <catalina.home> ${project.home}/portal_deploy/${tomcat.version}/ </catalina.home> <dot.globus.home> ${env.HOME}/.globus/ </dot.globus.home> </properties>

Unpack or checkout code

Cd GTLAB

All commands are executed from here.

Edit properties at the top of pom.xml.

Change IP

Change project.home if you unpack someplace besides $HOME.

Run “mvn clean install”

Run Examples From GTLAB, start tomcat with ./startup.sh. From GTLAB, stop Tomcat with ./shutdown.sh Point browser to http://localhost:8080/GTLAB Start with MyProxy Example

From GTLAB, start tomcat with ./startup.sh.

From GTLAB, stop Tomcat with ./shutdown.sh

Point browser to http://localhost:8080/GTLAB

Start with MyProxy Example

Next Steps Play with examples. These are really bare bones. Make something interesting. Make a Google gadget. Mix and match tags in a pipeline to make a new application. Use the dependency tag. Note you can mix and match JSF and JSP if you are not familiar with JSF. Try making a new tag. Explained next.

Play with examples.

These are really bare bones. Make something interesting.

Make a Google gadget.

Mix and match tags in a pipeline to make a new application.

Use the dependency tag.

Note you can mix and match JSF and JSP if you are not familiar with JSF.

Try making a new tag.

Explained next.

Making New Tags and Examples

Making a New JSF Page from Tags I recommend starting from the examples . jsf_standalone/src/main/webapp/examples “ Build” the examples with mvn –o clean install –f jsf_standalone/pom.xml The “-o” option is used to build offline . Will also avoid unnecessary Maven repository updates. The “-f” specifies only build this specific module. I recommend not futzing with the deployed versions under portal_deploy. A computer is a state machine. State must be reproducible.

I recommend starting from the examples .

jsf_standalone/src/main/webapp/examples

“ Build” the examples with

mvn –o clean install –f jsf_standalone/pom.xml

The “-o” option is used to build offline . Will also avoid unnecessary Maven repository updates.

The “-f” specifies only build this specific module.

I recommend not futzing with the deployed versions under portal_deploy.

A computer is a state machine. State must be reproducible.

Making a New Tag Run the following command from GTLAB: mvn clean process-resources -Dtag.name=test -Dprojectname=Test -f templateTag/pom.xml Add -Ddest.dir=/tmp for a dry run. Replace “test” with the name of your tag. Replace “Test” with the name for your Bean. This will make 4 files TestBean.java , TestTag.java, TestBeanFactory.java , UITest.java Edits also 3 config files gtlab-factory.xml, managed-beans.xml , components.xml This will compile but to implement something useful, you will need to edit the highlighted files.

Run the following command from GTLAB:

mvn clean process-resources -Dtag.name=test -Dprojectname=Test -f templateTag/pom.xml

Add -Ddest.dir=/tmp for a dry run.

Replace “test” with the name of your tag.

Replace “Test” with the name for your Bean.

This will make 4 files

TestBean.java , TestTag.java, TestBeanFactory.java , UITest.java

Edits also 3 config files

gtlab-factory.xml, managed-beans.xml , components.xml

This will compile but to implement something useful, you will need to edit the highlighted files.

Implementing a Tag The place to start is TestBean.java (or whatever you used for –Dprojectname=…). This includes several inherited methods that can be implemented. Most important is submit() . Use the try/catch block. This is where the action is. If you want to hook tags into chains, implement getOutput() and setInput() . Also take a look at the other beans.

The place to start is TestBean.java (or whatever you used for –Dprojectname=…).

This includes several inherited methods that can be implemented.

Most important is submit() . Use the try/catch block. This is where the action is.

If you want to hook tags into chains, implement getOutput() and setInput() .

Also take a look at the other beans.

Suggested Tags: A Wish List What can you do in your bean? Anything server-side Java can do. Some suggestions: Implement a tag client to a remote Web service. Amazon has some interesting ones…. Implement an RSS/Atom feed client to Twitter, your blog, Facebook, etc. Combine the feeds as a mash-up . Connect to a database with JDBC. Implement a JMS publisher or subscriber. Use Google Java APIs to interact with Blogger, Calendar, and YouTube. Try interacting with Facebook .

What can you do in your bean? Anything server-side Java can do.

Some suggestions:

Implement a tag client to a remote Web service. Amazon has some interesting ones….

Implement an RSS/Atom feed client to Twitter, your blog, Facebook, etc.

Combine the feeds as a mash-up .

Connect to a database with JDBC.

Implement a JMS publisher or subscriber.

Use Google Java APIs to interact with Blogger, Calendar, and YouTube.

Try interacting with Facebook .

INCA Tags: An Extended Example

Example: Make an INCA Dashboard INCA is used as the testing framework for the TeraGrid. You can get the latest INCA test results from the URL http://inca.teragrid.org/inca/XML/kit-status-v1/portal_summary See Sangmi’s notes at http://sangpall.blogspot.com/2009/03/being-nice-client-of-inca-service.html You’ll get back some fine looking XML. Let’s make a little dashboard out of this.

INCA is used as the testing framework for the TeraGrid.

You can get the latest INCA test results from the URL

http://inca.teragrid.org/inca/XML/kit-status-v1/portal_summary

See Sangmi’s notes at http://sangpall.blogspot.com/2009/03/being-nice-client-of-inca-service.html

You’ll get back some fine looking XML.

Let’s make a little dashboard out of this.

First, Create a New, Empty Tag mvn clean process-resources -Dtag.name=incastatus -Dprojectname=IncaStatus -f templateTag/pom.xml

mvn clean process-resources

-Dtag.name=incastatus

-Dprojectname=IncaStatus

-f templateTag/pom.xml

Second, Implement the Bean Main thing is to implement IncaStatusBean’s submit() method. Code is hard to put in a PPT slide See http://ogce.svn.sourceforge.net/viewvc/ogce/GTLAB/jsf_standalone/src/main/java/ogce/gsf/gridbeans/IncaStatusBean.java

Main thing is to implement IncaStatusBean’s submit() method.

Code is hard to put in a PPT slide

See http://ogce.svn.sourceforge.net/viewvc/ogce/GTLAB/jsf_standalone/src/main/java/ogce/gsf/gridbeans/IncaStatusBean.java

Third, Implement a JSF Page Actual tag is shown below. Full example is http://ogce.svn.sourceforge.net/viewvc/ogce/GTLAB/jsf_standalone/src/main/webapp/examples/IncaStatusExample.jsp?view=log It includes some boilerplate (can be automated). <o:incastatus id=”mytest&quot; resource=&quot;task1-resources” myfaces=&quot;true&quot;/>

Actual tag is shown below.

Full example is http://ogce.svn.sourceforge.net/viewvc/ogce/GTLAB/jsf_standalone/src/main/webapp/examples/IncaStatusExample.jsp?view=log

It includes some boilerplate (can be automated).

Here’s the resulting page. You can make into a portlet in a couple of steps. You can also tie to other tags in a pipeline.

Next Steps: Make a Pipeline Implement the setInput() and getOutput() methods. Pipeline it with the Resource Description and Resource Prediction Service tags. Integrate QBETS tags Currently unwritten Integrate with Job Submission Find Running Hosts (INCA or RDS) Find Best Host for YourJob (RPS) Find Host with Shortest Wait Time (QBETS) Submit Job

Implement the setInput() and getOutput() methods.

Pipeline it with the Resource Description and Resource Prediction Service tags.

Integrate QBETS tags

Currently unwritten

Integrate with Job Submission

More Stuff about GTLAB Supports linear tag dependencies . Removed dependence on COG graph processing (still use the COG) We can support graphs of any tag you care to write. Only support pipelines currently, not full DAGs Redesigned to simplify writing new tags . Extensive use of factories. You no longer have to change core code to add a tag Templates for 4 Java files and 3 XML config files now can be done in one (mvn) step Typically you will only need to edit one Java file.

Supports linear tag dependencies .

Removed dependence on COG graph processing (still use the COG)

We can support graphs of any tag you care to write.

Only support pipelines currently, not full DAGs

Redesigned to simplify writing new tags .

Extensive use of factories.

You no longer have to change core code to add a tag

Templates for 4 Java files and 3 XML config files now can be done in one (mvn) step

Typically you will only need to edit one Java file.