ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®

UKLUG 2012 – Cardiff, Wales
Background Tasks
An XPager's Guide to Process Server-Side Jobs on IBM® Domino®
Serdar Basegmez | Developi / OpenNTF
Frank van der Linden | e-office / OpenNTF

Agenda
 Introduction
 DOTS
 Basics
 Running, Developing and Deploying DOTS
 Programmability Tips and Tricks
 Asynchronous Processing
 Developing Jobs
 Alternative Approaches
 Tips and Tricks
 Q&A

Serdar Basegmez
•IBM Collaboration Solutions Champion (2011-2013)
•Owner of Developi Information Systems (Istanbul, Turkey)
•Founder/Co-leader of LUGTR – Turkish Lotus User Group
•Bilingual Blogger at LotusNotus.com (Turkish/English)
•OpenNTF Guy in Turkey
• Contributor in XSnippets and CollaborationToday.info
• Member Director at OpenNTF Board
•IBM Notes/Domino / SBT Design Partner
•Expertise on XPages, IBM Domino, IBM Connections, etc.

Frank van der Linden
•IBM Collaboration Solutions Champion (2013)
•XPages developer at e-office (Netherlands)
•Founder/Co-leader of XPages Meetup (Netherlands)
•Blogger at domino-weblog.com
•OpenNTF
• Contributor in XSnippets and CollaborationToday.info
• Member Director at OpenNTF Board
•IBM Notes/Domino / SBT Design Partner
•Expertise on XPages, IBM Domino, IBM Connections, etc.

Introduction

What’s the problem?
•XPages is great!
• We all know it...
•But...
• XPages are based on interaction.
• Users demand, server replies back.
•What if…
• Processing takes too long?
• You need housekeeping?
• You have scheduled reports?
• You need to send reminders?
• Your application needs batch processing?
• You require unattended tasks?

Background Processing is Possible!
Unattended Tasks
• Domino OSGi Tasklet Services
• Alternative to Domino Agents
Asynchronous Processing
• Threads and Jobs
• Multi-threaded Applications

Domino OSGi Tasklet Services

Meet DOTS – your new best friend
•Domino OSGi Tasklet Services: Next Generation Agents for Domino
•DOTS provides ability to develop and run OSGi level server Tasklets for
Domino
• Easy deployment.
• Tooling with Eclipse IDE and its Java and plugins editors and Java debugger.
• Leverage existing OSGi assets (within the company or open source).
• Tasklets can be run:
• Manually (local and remote calls)
• Scheduled or on server start
•OpenNTF project by IBM since April, 2011 (replaced JAVADDIN project)
•Included (but not supported) in IBM Domino 9 Social Edition

What is a Tasklet?
•Component to process a step in your business logic
• Lightweight addin task.
• DOTS Tasklets are simply Java methods within OSGi bundles.
• Declared as OSGi extension point
•Tasklet is functionally less capable than a Server Task
Tasks DOTS Tasklets
Loaded once, run continuously Loaded once, run when needed
Manages own threads DOTS manages threading
Should be native (or encapsulated) Runs inside DOTS container
Unmanaged security Managed security
Hard to develop Easy to develop

DOTS vs. Agents
•DOTS Tasklets outperforms over Java Agents!
AMGR
launches a
Thread
JVM
Initializes
Prepare
Java and JNI
interfaces
Load Agent
Bytecode
Run!
Run!
Java Agent:
DOTS tasklet: Everything is ready to run
> tell amgr run "testXPagescrash.nsf" 'LongJobAgent'
09.11.2012 19:38:39 JVM: Java Virtual Machine initialized.
09.11.2012 19:38:39 AMgr: Start executing agent 'LongJobAgent' in 'testXPagescrash.nsf'
09.11.2012 19:38:39 Agent Manager: Agent printing: 181349
09.11.2012 19:41:02 Agent Manager: Agent printing: 2227
09.11.2012 19:41:02 Agent Manager: Agent printing: Finished in 143 secs... -
09.11.2012 19:41:02 AMgr: Agent 'LongJobAgent' in 'testXPagescrash.nsf' completed execution
> load dots
> Listening for transport dt_socket at address: 8001
09.11.2012 19:42:40 Domino OSGi Tasklet Container started ( profile DOTS )
> 181349
> 2227
09.11.2012 19:43:22 [DOTS] (annotated) Finished in 41 secs...

DOTS vs. Agents
•Amgr is old and complicated
• Security model problems
• Limited functionality (like issues on class loading, reflections)
•Range
• Agents are database-dependent
• DOTS runs server-wide
•More flexible scheduling
• Precise timings like 45 seconds
• Programmatic control over scheduling
•Multi-threading
• DOTS tasklet might perform multi-threaded operations for more efficiency

Examples on DOTS
•IBM SmartCloud™ for Social Business (a.k.a. LotusLive)
• Some part of back-end processing has been implemented using DOTS
•CollaborationToday.info
• Feed Aggregation is implemented using DOTS
•Other possible uses...
• Workflow Engine implementations
• Social Media Monitoring
• Database Monitoring
• Application Architecture Assessment
• Big Data
• Search and Discovery

Runtime Environment
•IBM Domino 9.0 Social Edition provides DOTS Runtime
• via Embedded Experience Add-on
•For older versions,
• DOTS should be installed into IBM Domino server
• Refer to BP207 slides from IBM Connect 2013

Development Environment
•DOTS Tasklets are simply Java methods
• Designated classes as OSGi bundles.
• Deployed as Eclipse plug-ins.
•Eclipse IDE is needed for development and debugging.
• DDE can be used with lots of unsupported configuration changes.
• “Unsupported”  No need for an adventure 
•Refer to BP207 slides from IBM Connect 2013

Deployment Methods
•DOTS tasklets are packaged as a «Plug-in»
•Plug-ins can be deployed into OSGi by...
• Referencing PDE tool to the Eclipse Run Configuration
• For testing & development...
• Putting JAR file into OSGi plug-ins folder,
• Beg your Admin for file-level access to the server!
• Referencing an update site
• Using DOTS Profiles, any update site can be used (NSF, HTTP…)

Demo
Photo: http://www.flickr.com/photos/mjtmail/

Deep Dive into DOTS

You Like Annotation?
•DOTS also provides annotated declarations within Java classes
• Remember, you still need to create an extension point in «plugin.xml»
@RunOnStart
public void runOnStart( IProgressMonitor monitor ){
logMessage("Annotated onStart method");
}
@Run( id="manual")
public void runManual( String[] args, IProgressMonitor monitor ){
logMessage("Annotated run method with id=manual");
}
@RunOnStart
@RunEvery( every=60, unit=RunUnit.second )
@HungPossibleAfter( timeInMinutes=1 )
public void runEvery60seconds( IProgressMonitor monitor ){
logMessage( "Called from annotated method every 60 seconds");
}
<plugin>
<extension point="com.ibm.dots.task">
<task class="com.ic13.samples.helloworld.AnnotatedTasklet"
id="someid" />
</extension>
</plugin>

A Powerful Feature: DOTS Profiles
•A DOTS profile is a specific DOTS container.
• Each profile defines «Tasklet Container» that has different settings for;
• Remote Controller, Security Configuration, Specific plug-in container
•DOTS support multiple profiles for different needs.
• Tasklet used occasionally, for testing or with special security
•Profiles can be stored in XML files or Notes Databases
• It can be any Notes Database like names.nsf.
•Every profile has a unique name (Message Queue Name – mqName)
• Profiles accept separate tell commands, like «tell profileName run XYZ»
• DOTS loaded with a default profile named as «DOTS»

Programmability Tips: Logging
•Logging
• Try not to use «System.out.println». (No output to log.nsf)
• Instead, use «logMessage(...)» and «logException(...)»
•ServerConsole helps.
• If your code works outside the tasklet, you don’t have «logMessage(...)»
• Use «com.ibm.dots.task.ServerConsole» class for console output.
•You know “XPages OpenLog Logger” project?
• It utilize OpenLog database for your Java code.
• Find Paul S.Withers and buy a beer for him.

Programmability Tips: Parameters
•Parameterization
• Manual tasks can get parameters through Console or Plug-in
manifest (plugin.xml)
• «tell dots run XYZ param1 param2».
• «args[n]», «getNthArgument(n)» or «getKeyedArgument(key)» can be used.
• For other tasks, you have multiple options.
• «notes.ini» settings
• Argument Resolver (refer to DOTS samples)
• Profile parameterization extension via DXL (refer to DOTS
readme.pdf)

Programmability Tips: Parameters
•Profile Configuration Document:

Remote Controller for DOTS
•Remote Controller
• DOTS might accept remote commands from other applications
(e.g. XPages apps)
• Remote commands are accepted from localhost only.
• Listener should be configured from Profile document.
Source: DOTS Readme File

Progress Monitoring for DOTS
•Progress Monitoring
• «IProgressMonitor monitor» argument passed on tasklet methods.
• Very useful for tasklets running long time.
• Progress monitor can be used to...
• Inform DOTS task manager about how much tasklet has completed.
• Aware of situation that might break execution.
Vector<?> views = db.getViews();
monitor.beginTask( "List the views", views.size() );  Init with # of steps
for ( Object view : views ){
if ( monitor.isCanceled() ){
break;  Break if canceled.
}
logMessage("t" + ((View)view).getName() );
try {
Thread.sleep( 1000 );
} catch (InterruptedException e) {
e.printStackTrace();
}
monitor.worked( 1 );  One step completed
}

Threads and DOTS
Source: Lotusphere 2012 – SHOW112 session by David Taieb, Paul Fiore, Elizabeth Sawyer
Three Two
different threads

Threads and DOTS
•Implications on running:
• One thread for manual tasklets?
• The first tasklet you call will run,
• The second one will run at the sametime
• One thread for scheduled tasklets?
• Only one scheduled tasklet run at any time.
• Conflicting tasklet will be deferred once.
• What about RunOnStart Tasklets?
• More than one RunOnStart tasklets will run at the same time!

Threads and DOTS
•Best Practices for multiple long-running tasklets
• Need some attention for synchronization issues
• Use Progress Monitoring for cancellations
• Too many scheduled tasklets  Unpredictable scheduling
• You can use multiple profiles for different purposes
• Each profiles has its own service threads!
• Scheduled tasklets might fall into deadlocks!
• Deadlocks are not resolved automatically.
• while(true) is your enemy…
• Some Java methods are black holes.
• e.g. HttpURLConnection has infinite timeout by default.

Threads and DOTS
•In terms of instances:
• Threads run tasklets with reflection.
• Scheduled and manual tasklets differ a bit:
• Manual (and RunOnStart) Tasklets:
• Thread creates an instance from the tasklet class
• Calls the method and destroys the instance.
• Each time you run a manual tasklet, it’s a different Java object!
• Scheduled Tasklets:
• Thread creates an instance for the first time,
• Saves the instance for the next time.
• Each time it runs a scheduled tasklet, it’s the same Java object!

Threads and DOTS
•Instantiation may be a problem in some cases:
BundleBundle.start  Initialize the Queue
Bundle.stop  Save the Queue
QueueManager
(singleton object)
readNextFeed()
refreshQueue()
readFeed(id)
Tasklet Class
@RunEvery( every=2, unit=RunUnit.minute )
Sched1()  qm.readNextFeed()
@RunEvery( every=60, unit=RunUnit.minute )
Sched2()  qm.refreshQueue()
@Run( id="refreshQueue" )
Manual1(args)  qm.refreshQueue()
@Run( id="refreshFeed" )
Manual21(args)  qm.readFeed(args[0])

Problems of DOTS
•DOTS runs in a different OSGi level than HTTP.
• So there is almost no connection between DOTS and XPages
• You can start DOTS tasks from XPages, using remote controller socket
• Refer to Readme documentation for configuration and a basic example
• Some possibilities
• Communicate over Notes Document
• Using REST API to send/receive information between DOTS and XPages
•OSGi has different architecture for Class Loading
• Some Open Source APIs might have issues (e.g. Rome project)
• You need to have some level of OSGi knowledge in case
•Out of Support Contract for now.
• DOTS tasklet functionality is for internal use in Domino 9

Asynchronous Processing for XPages
•The Problem
• XSP Engine provides a single-threaded operation for users.
•Implications
• User has to wait for completion of long-running tasks
• Worse, timeout will cause trouble…
• Sometimes, it’s meaningless for user to wait
• Throw into the basket and forget
• Low performance
• Varies for different cases
•Solution
• Launch a parallel thread for a non-blocking asynchronous task

•Basic Problem
• We have two needs to run a Java code for Domino
• A Session object
• A [Notes] context
• These are provided for us in XPages, DOTS, Agents, etc.
• … for a particular thread…
• There is a planned life cycle for every threads
•Fortunately, we can create a context
• It’s like dancing on a thin ice…
Photo: http://www.flickr.com/photos/ell-r-brown/

Formal Approach
•ThreadSessionExecutor has been introduced in 8.5.3
• Simpler, cleaner
• However, it #FAILS for Domino 9.
• Expecting a fix for 9.0.1

An Eclipse Job
Job class
Modification #1

An Eclipse Job
Run
Method

An Eclipse Job
•Running any code out of these boundaries will fail
• Threads cannot work out of Notes Context…
A static
method to
start the job

Other Approaches
• Stephan H. Wissel posted a thread management bean for XPages
• http://www.wissel.net/blog/d6plinks/SHWL-99U64Q
•Provides
• An abstract class
• Managed Bean for Background Jobs
• Very simple to use
• No need to design separate Job classes for every task
• Can be easily embedded into applications

Other Approaches
• Nathan T.Freeman has ported more classes for Threading
• Within OpenNTF Domino API project (OpenNTF API is optional)
• https://github.com/OpenNTF/...BackgroundRunnable.java
•Provides an abstract class
•Better for long-running threads
•Session for different users (including server)
•For DOTS and XPages
•Will be extended for Callable and Future interfaces

Multi-threading in DOTS
•Similar idea can be applied for DOTS tasklets
•Why?
• Improved Performance
• A job can be divided atomic parts (vertically or horizontally) and run
asynchronously
• Better Functionality
• Developing your own scheduling system
• Shifting Paradigm
• Design a tasklet that acts as a service provider (e.g. FTP Server)

Multi-threading in DOTS
•Method:
• A “RunOnStart” tasklet, continously running.
• A Thread / Job manager adapted
• Session creation is easier, just like a Java Application
• Use NotesThread and NotesFactory
• Test, test, test…
•For sophisticated examples:
• “DOTS” Source code
• “DOTS Extended” project source code

Important Points
•Jobs and Threads will run on behalf of the authenticated user
•Use Java responsibly…
• TRY-CATCH is your best friend.
• Recycle always… (except Session…)
• Use Logging
•If something doesn’t work right,
• Restart HTTP task, Clean the project
•Keep Job and Thread objects clean
• Perform the real task in your own classes.
• Pass the session object, throw and catch exception in case
•Always perform harsh tests before production

Some Important Resources
•IC13 - BP207 Session by Serdar Basegmez & Bruce Elgort
• http://bit.ly/DOTS2013
•OpenNTF Project: Threads and Jobs
• http://www.openntf.org/internal/home.nsf/project.xsp?action=openDocument&na
me=Threads%20and%20Jobs
•Java concurrency (multi-threading) – tutorial by Lars Vogel
• http://www.vogella.com/articles/JavaConcurrency/article.html
•Eclipse Jobs API
• http://www.eclipse.org/articles/Article-Concurrency/jobs-api.html
•Stephan H. Wissel - NotesSessions, XPages and Threads
• http://www.wissel.net/blog/d6plinks/SHWL-99U64Q
•CollaborationToday.info FeedMonster Module – on Github
• https://github.com/OpenNTF/collaborationtoday/tree/master/DOTSFeedMonster

Q&A

Thank you!
Twitter: @flinden68
Blog: http://www.domino-weblog.nl
Twitter: @sbasegmez
Blog: http://lotusnotus.com

ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®

More Related Content

What's hot

Viewers also liked

Similar to ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®

More from Serdar Basegmez

Recently uploaded

ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®