Show104 buried treasure

SHOW104 Buried treasure: Finding
the Hidden Gold in Lotus Notes Data
Mark Myers | London Developer Coop
Julian Robichaux | panagenda

© 2013 IBM Corporation

Agenda

Overview

Working with Feeds

Integrating with IBM Connections

ODBC, JDBC, and Reporting

Generating Files on the Domino Server

Caveats, Considerations, and Data Scrubbing

Where to Get More Information


Who Are Mark & Julian?

 Julian
– Java/Eclipse application developer at panagenda (panagenda.com)
– Developer since before Justin Bieber was born
– Notes/Domino since version 4.1
– Speaker at 7x Lotuspheres, various LUGs and View conferences
– Twitter: @jrobichaux, Blog: nsftools.com

 Mark
– Member of the London Developer Co-op (londc.com)
– Developer from a support background
– 12+ years on Domino, 15+ years in IT
– Speaker at 2x Lotuspheres, 3x UKLUGs, 1x ILUG
– Twitter: @stickfight, Skype: Stickfight, Blog: stickfight.co.uk


Why Are We Here?

 You have years and years of data stored in your Notes databases
– Valuable
– Critical?
– Historically important

 Often, that data needs to be accessed
from other places
– Reports
– Intranets/portals
– Non-IBM systems

 What are your options?

 As it turns out, you have a lot of options…


Tools We Used

 IBM® Lotus® Notes® version 8.5.3

 IBM Lotus Domino® version 8.5.3

 Microsoft® Windows® 7

 IBM Connections V4.0

 Various other software tools mentioned
throughout this presentation
 Most code and techniques we talk about should be applicable to other versions of
Notes/Domino/Windows too


Old and New

 This presentation is an eclectic mix of Old and New, borne out of many many
years of Domino programming experience
 Solutions are based on Ease, Speed and Reliability rather than the latest coding
fashion
 If a solution seems old (more than 2 year old technology) it is because:
– Some implementations we see are still being done wrong (or not done at all)
– Some people are still on old versions of software
– Sometimes the “old” way is still the best way to do things
– Everyone is at a different experience level; maybe you’ve never seen some of this before
– This stuff still works


About the Sample Database

 This year’s sample dataset is… Sea Ducks!

The Atlantic Flyway Sea Duck Survey, conducted by the
U.S. Fish and Wildlife Service, was established in 1991
to record sea duck numbers using near shore (within 700
m of shore) habitats from Cape Breton, Nova Scotia to
Jacksonville, Florida.

https://migbirdapps.fws.gov/mbdc/databases/afsos/disclaimerafsos.html


About the Sample Database

 Public domain data and photos from the US Fish and Wildlife service
– See the “About this Database” page in the sample database for information, disclaimers,
and links

 A little over 10,000 lines of data (i.e. – individual documents in a view)

 Various means of grouping and data retrieval
– By year
– By state
– By duck type

 Some documents also have image files stored in
rich text fields
 So… potentially similar to data you already have


Agenda

 Overview

 Working with Feeds

 Integrating with IBM Connections

 ODBC, JDBC, and Reporting

 Generating Files on the Domino Server

 Caveats, Considerations, and Data Scrubbing

 Where to Get More Information


What’s a Feed?

 Structured data
– 0-to-N items

 Parseable format

 Retrievable via a link or some other automated method

 Ideally (but not necessarily) semantic
– Semantic markup assigns “meaning” to data

http://en.wikipedia.org/wiki/Microformat


JSON vs. XML


ReadViewEntries

 Replace ?OpenView with ?ReadViewEntries in the URL of any Domino view on the
web to see the contents as XML
– Since Domino 7


ReadViewEntries

 Some optional parameters:
– CollapseView / ExpandView
– NavigateReverse=1 – lists documents in reverse order, starting with the last document
– Start=n – indicates which document number to start with
– StartKey= / EndKey= – indicates which key in a sorted view to start and/or end with
– RestrictToCategory= – indicates which category to display in a single-category view

 Count=nn
– Very important if you want to get ALL the docs in a view
– Default is 30, or whatever is specified by “Default lines per view page” on the Domino
Web Engine tab of the server (or web site) document
– Default maximum is 1000, or whatever is specified by “Maximum lines per view page”

 OutputFormat=JSON
– Displays data as JSON instead of XML

 A few other parameters too: check Domino Designer help


ReadViewEntries JSON Format


ReadViewEntries XML Format


Notes on Using ReadViewEntries

 Maximum View Count settings on the server doc might prevent you from seeing all
the data in the view
 You can save the query and refresh the data on demand

 Categorized views get messy, flat views are better

 If you need to login, make sure the server is using Basic Authentication (at least
for that URL)
– Not strictly necessary, but much easier from a programmatic standpoint


Overriding Session Authentication

 If your Domino server uses session authentication (most do), there is an option to
override this and use basic authentication for specific URLs
 Since Domino 7
– http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/com.ibm.help.domino.admin.
doc/DOC/H_OVERRIDING_SESSION_AUTHENTICATION_8847_STEPS.html

 Must be using Web Site documents for web server access
– “Load Internet configurations from ServerInternet Sites documents” enabled on Basics
tab of server document
– http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/com.ibm.help.domino.admin.
doc/DOC/H_R5_WEB_SITE_TO_RNEXT_OVER.html



 Open the “Web Site” document for your server



• The URL pattern can use *
for wildcards (? is treated as
a literal character).
• If you don’t a wildcard at the
beginning and end, the rule
might not work the way you
expect.



 Optional parameter on the Web Site document to auto-generate a session cookie
when Basic Authentication is used in an override rule


Classic Feed: ICalendar

 Still the best format for dealing with event based data

 Near universally supported

 Can be done without 3rd party libraries



We just need a view.


Classic feed: ICalendar

Most ICalendar clients tend
to be picky on carriage
returns so you need to be
careful



ICalendar date time is in the
format yyyyMMddTHHmmssZ


Classic feed: ICalendar

Then insert the view in a form
with the text that wraps the
ICalendar Vevents, this text
details time zones and such
for the calendar as a whole


Classic feed: Web Services

The De facto standard for inter-system communication
 Still seen as a silver bullet by many managers

 See a new web service as an opportunity to re-define how your data is seen

 Complex if done badly, simple with a few tricks or the right software


Classic Feeds: Web Services

 Best Tools
– Lotus Domino server!

– Liquid XML Studio ( http://www.liquid-technologies.com/xml-studio.aspx )
• Brill visual designer
• Powerful but easy to use
• Costs a lot from a stingy developer’s point a lot (but worth it)

– Soap UI ( http://www.soapui.org )
• The simplest way for humans to interact with and test raw web services
• Free for the Lite version


Classic Feeds: Web Services


Custom Feeds: Decisions to Make

 Data format
– XML, JSON, CSV, something else…

 Data structure
– RSS, Atom, something else…
– Standard structures are widely supported
– Custom structures allow more flexibility

 Static or dynamic
– Caching and performance issues
– User-specific information?

 Do you control the clients?
– If not, it might be hard to make changes later


Atom and RSS

 Standard XML formats for document-based data

 RSS is simpler to produce and parse
– Fewer options
– Easy to understand
– Great for syndication of news, blogs, and similar

 Atom is more flexible
– Easily extensible for data customization
– Read AND write data, if you need to publish
– Still primarily for documents, rather than raw spreadsheet-type data

 Almost any software that calls itself a Feed Reader can consume either type of
feed with no problem


Feed Library vs. Build-Your-Own

 XML and RSS/Atom is easy to break
– Character encoding issues
– Illegal characters
– Unclosed tags
– Improperly formatted dates
– Missing required elements

 If you write “raw” XML to a text string for output,
test often and add plenty of checks for poorly formatted data
 Feed libraries take care of the heavy lifting for you
– Built-in validation
– No need to study the specification documents
– Probably slower and more memory-intensive
– Usually several megabytes of required code libraries either stored in the database or
copied to the server


Example: RSS Feed Generator Database

 Template included with Domino 7

 Easy to use for quick feeds:
– Create a new database on the server using the “RSS Generator” template
– Fill out a form to point to a view
– Point the users to the database URL

 Reuse the code in your own applications



 Unfortunately, the <enclosure> item has a hard-coded data type, so it’s only good
for docs where all the attachments are the same type


Example: Java Agent Atom Feed

 A few different Java feed parsing libraries to choose from:
– ROME – http://rometools.org
– Apache Abdera – http://abdera.apache.org
– Apache Wink – http://incubator.apache.org/wink

 All require several additional libraries
– Apache Commons libraries
– Java StAX support
– Etc.

 Handle HTTP connections as well as parsing
– Authentication too

 An excellent test client:
– DEV HTTP Client (Chrome Plugin)
https://chrome.google.com/webstore/detail/aejoelao
ggembcahagimdiliamlcdmfm


Example: Java Agent Get Atom Feed


Example: Java Agent Submit Back to Atom Feed

 Most Atom services (particularly IBM) give examples of what XML they want rather
than examples of the code you should use, so we will do the same
 This is a IBM Connections Blog entry in raw XML:

<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom">
<author><name>I Am Funny</name></author>
<title type="text">Great Duck Joke</title>
<content type="text">Q: What has webbed feet and fangs? A: Count Duckula</content>
</entry>



An amazingly useful line when you are debugging or
testing as it outputs the XML so you can check it against
what you should be sending



Getting this URL is often the hardest part of dealing with
Atom


Example: Useful Extra Lines
Add “registerTrustManager”
and you can connect to
SSL Ports

Nice simple login, you just need the user name, password
and the root domain you are logging on to


Agents vs. XAgents

 Java agents are not efficient when using large JAR files attached to agents or
script libraries
– They are reloaded into memory each time the agent runs
– Beware of memory problems

 Copying the JAR files to the jvm/lib/ext directory on the Domino server keeps them
in memory
– If your admin allows it
– Have to do this with every server the code might run on

 An alternative is XAgents: XPages written so they run as agents
– No UI, except for print statements (“write”, actually)
– JARs and Java code attached directly to the database
– Much better caching of JAR files
– http://www.wissel.net/blog/d6plinks/shwl-7mgfbn
– http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm


Caveats

 Malformed JSON and XML

 Character encoding issues

 Multi-value fields can be tricky

 Memory issues when using large Java libraries
– Or poorly written Java code

 Make sure you use the right Content-Type HTTP header when you send data
– JSON should be “application/json”
– XML should be “application/xml” (preferred) or “text/xml”


Integrating with IBM connections

 2 basic forms of integration
– Submitting existing data directly into Connections (as demonstrated by the previous
Atom examples)

– Read data from Domino and displaying it in Connections via a widget


Connections Caveats

 This is not a Connections session

 The methods shown in this session are one of many methods of integration rather
than specific best practices
– Niklas Heidloff (http://heidloff.net) and Mark Leusink (http://linqed.eu) have some good
examples
– These methods use as few toolkits such as Social Business toolkit/Social business SDK,
social enabler as possible.

 Real life has taught that the best integration environment to build is one where you
can get all data if required (Gather all data using service accounts) then compile a
result and display to the user.
 See the related sessions at the end of this presentations for more details and the
best practice sessions.


Integrating with IBM Connections

Before we start we need to get a
database ready to call Java from
XPages

Move to Package Explorer



In The WEB-INF directory

Right click and create a new
folder



Call it “src” in lower case



Next add this folder
to the projects build
path

Right click on the
root of the project
and select
“Properties”


On the “Source” tab
click “Add folder”
and add the folder
you have created


You will see the
Folder has
moved

Right click and
select “New” →
“Other”

Then Select
“Java” →
“Class”



Create the new Java Class



OK, Java class created, lets get it
some libraries



If you are not allowed
to use the EXT
directory on the server,
the you will need to
create a new “lib” folder



Update (or ask your administrator to) the java.policy file on your server to contain
the following

grant {

permission java.lang.RuntimePermission "getClassLoader";

};


Now we will add the
Integrating with IBM Connections Atom client code
function

You need to pass the
session into this
function as it can’t get
its own

If you need to you can
login here (as
described previously)



Create a New Script
Library



Set it as a Normal “Server JavaScript” Library



Create a SSJS function to call your Java and pass the results back

Pass the “sessionAsSigner” here to give yourself administrator rights


Integrating with IBM Connections – Pulling it together

Create a Custom
Control



Give it a Name



Now you can add a repeat control that calls the SSJS function, which in turn
calls the Java

The Rest of the surrounding boring code is in the demonstration db



Create a New
Xpage Library



Add the
duckstates_cc
custom control

Now that you have an
XPage, you need to
present it to
Connections as a
Open Social widget
the easiest way of
doing this is with a
simple domino page.



Features like
“dynamic-height” are
harder to get working
using this method
(rather than a
embedding the widget
in connections) but if
you have a static sized
widget it is the
simplest.



And add it to your
Connections
installation


SQL Queries of your Notes Data?

 Lotus Notes is NOT a relational database
– You knew that, right?

 IBM has a NotesSQL driver
– Free download, easy install
– Treats Notes Views like SQL tables
– Has been updated for Notes 8.5


From the Notes Client: NotesSQL

 Free tool from IBM

 Windows-only (make sure you have the latest version on Windows 7)

 Set up on each user’s machine (install plus DSNs)

 Notes client must be installed (version 6.0 or higher)

 Easy integration with reporting tools like Crystal Reports, MS Excel, and MS
Access


Setting up NotesSQL

 Download from
http://www.ibm.com/developerworks/lotus/
products/notesdomino/notessql
– Help files are in C:NotesSQLdocs after
install

 After install, the NotesSQL directory and
the Notes program directory MUST be in
the Windows PATH
– You get vague error messages if they’re not

 Create an ODBC DSN to point to a
database


Example of Connecting with MS Excel


Optimizing Your Data

 You will almost certainly want to create one or more “reporting” views
– Many columns of data
– No categorization
– Minimal sorting for better lookup performance
– Reasonable column names (check the “Programmatic Access” name for computed
column data)

 If you’re reporting on rich text data, modify the DSN to allow more than 512
characters for rich text fields
 Read the help docs on performance


NotesSQL Caveats

 Problems with MS Excel 2003
– Sometimes hangs while accessing or saving data
– Access 2003 works fine though
– Excel 2007+ also works fine

 Problems with OpenOffice 2.x
– View columns with text values are blank
– OpenOffice 3 works, but has problems with very large views

 Make sure users are using the correct views for lookups
– View names are not always obvious in existing databases with lots of views

 If users are sharing reports/spreadsheets, the NotesSQL DSN must be set up
exactly the same for each user


From the Server: Domino JDBC

 OpenNTF project written by Philippe Riand

 No Notes client required!

 Access Notes view data directly from the Domino server using JDBC


How It Works

 Runs as a server task
– Uses DOTS (another OpenNTF project) to allow Java to run as a server task
– Domino 8.5.3 or higher

 Define which views to expose as tables using an XML file in the database

 Creates a SQLLite “virtual table” for each view

 JDBC driver on the client reads the data from SQLLite tables on the server

Notes SQLLite JDBC
Views Tables Driver


Setup: Install DOTS (Windows example)

 Download from OpenNTF
http://openntf.org/p/OSGI+Tasklet+Service+for+IBM+Lotus+Domino
 Unzip to c:dots

 Copy these files from “c:dotsbuildwin32" to the Domino program directory:
dotsExtMgr.dll, dotsNSFHook.dll, ndots.exe
– Or buildwin64, or buildlinux64, or whatever

 Create the following folders in the Domino program directory: osgi-
dots/shared/eclipse/plugins and osgi-dots/rcp/eclipse/plugins
 Copy these files from “c:dotsbuild" to the Dominoosgi-dots directory: launcher.jar,
com.ibm.notes.java.api_XX.jar, and dotssec.jar
 Copy c:buildcom.ibm.dots_XX.jar to osgi-dots/shared/eclipse/plugins


Setup: Install DOTS

 Download Eclipse 3.7.x (Indigo) from
http://www.eclipse.org/downloads/packages/release/indigo/sr2
– Get the “Report Developers” version, because we’ll be using BIRT later

 Unzip Eclipse
– Use 7-Zip, not the native Windows unzip program

 Copy required org.eclipse.* files from Eclipse to osgi-dots/rcp/eclipse/plugins
– These MUST be from Eclipse 3.6.x or 3.7.x. Other versions of Eclipse might not work,
and the versions already in the osgi directory on the Domino server definitely will not
work

org.eclipse.update.configurator org.eclipse.equinox.common
org.eclipse.osgi org.eclipse.core.runtime.compatibility.auth
org.eclipse.equinox.preferences org.eclipse.core.jobs
org.eclipse.equinox.registry org.eclipse.core.runtime
org.eclipse.equinox.app org.eclipse.core.contenttype


Setup: Install DOTS

 Add the following lines to notes.ini on the Domino server:
– NSF_HOOKS=dotsNSFHook
– EXTMGR_ADDINS=dotsExtMgr

 Start the Domino server

 Type "load dots" at the server
console
 You should get a "Domino OSGi
Tasklet Container started"
message
– If you copied the samples.jar file,
you will also get lots of messages
every 60 seconds

 To stop the DOTS service, type "tell dots quit"


Setup: Install JDBC

 Download from OpenNTF:
http://openntf.org/p/JDBC+Access+for+IBM+Lotus+Domino
 Unzip to c:DomJDBC

 Copy the following folders from {domino}/osgi/shared/eclipse/plugins to
{domino}/osgi-dots/shared/eclipse/plugins :
– com.ibm.commons.jdbc_XX
– com.ibm.commons_XX

 Copy the following file from {domino}/osgi/rcp/eclipse/plugins to
{domino}osgi-dots/shared/eclipse/plugins :
– com.ibm.notes.java.api_XX.jar
– If you already have the java.api file from the DOTS install, just remove the existing one
and replace it with the one from Domino


Setup: Install JDBC

 Copy all the plugins from c:DomJDBCupdateSite-dots.zip to
{domino}/osgi-dots/shared/eclipse/plugins
 Copy com.ibm.xsp.extlib.relational.domsql_1.0.0.jar from c:DomJDBCupdateSite-
xpages.zip to {domino}/osgi-dots/shared/eclipse/plugins
 Restart DOTS

 You should get a "Initializing DomSQL RMI server for Domino" message


Setup: Install JDBC

 The default JDBC port is 8089, and you need to make sure your firewall allows
ndots.exe to use that port
– You can change the port number by adding DomSQL_Port=XXXX in notes.ini


Configuring Your Databases

 Notes databases on the Domino server must be explicitly set up to allow JDBC
access to views
– Each database set up individually
– You can set up a single view, or all views, or a group of views

 Security
– Only expose the views you want to expose
– Database ACL is respected

 As with ODBC, you might want to create special “reporting” views



 Open the database in Domino Designer

 Add the "Package Explorer" view if you don't
already have it (Window - Show Eclipse
Views - Other - Java - Package Explorer)
 Expand the database in Package Explorer
and go to Web Content - WEB-INF



 Create a new folder under WEB-INF
called "jdbc“
– Right-click - New - Other - General - Folder

 Create a new file in the new "jdbc" folder
called "views.domsql“
– Right-click - New - Other - General – File
– It doesn’t specifically need to be called
“views”, it can be anything with a .domsql
extension
– You can have multiple .domsql files



 To expose all views via JDBC, just use this in your .domsql file:

 To expose specific views, rename views, override column properties:



 NOTE: DomSQL uses the "Programmatic Use" name for columns in the view. You
might want to override this either in the view itself or using a <column> definition in
the .domsql file.


Example: Connect using SquirrelSQL

 Many programs are available to access data via JDBC

 One nice (free) one is SquirrelSQL
– Java, LGPL
– Plugins
– SQL code completion
– Export SQL query results to CSV, Excel,
XML format
– http://squirrel-sql.sourceforge.net


Show104 buried treasure

More Related Content

What's hot

Similar to Show104 buried treasure

More from Mark Myers