Session De13 from Engage User Group Conference 2022 with Thilo Volprich "In the jungle, no application walks alone. Sooner or later, applications will need to interact with other internal and external applications. Luckily, Notes/Domino is a powerful platform to access data from other systems and provides wide range of possibilities to interact with different APIs. This session focuses on how to architect various integration scenarios to implement secure and robust solutions for your Notes/Domino platform. Whether you use LotusScript or Java, come to this session to see the best practices with real-world use cases." (25/May/2022)

1. #engageug
DE13
The Superpower of Integrating External APIs
for Notes and Domino Apps
Serdar Basegmez, Developi
Thilo Volprich, Team Technology
ENGAGE 2022

2. #engageug
Who is speaking to you today?
▪ CEO of Team Technology GmbH
▪ IBM Champion 2018 - 2019
▪ HCL Ambassador 2019-2021
▪ Loving Domino
▪ Doing that stuff since 2003
Thilo Volprich
▪ Developer/Half-blooded Admin
▪ Developi UK
▪ OpenNTF Board
▪ Notes/Domino since 1999
▪ IBM Champion Alumni (2011-2018)
▪ HCL Ambassador (2020-2022)
Serdar Basegmez

3. #engageug
Our session today
๏What is in our session
๏ A lot :-)
๏ Collective experience with customer projects
๏ Our (very personal) recommendations
๏ Some small code examples that can be easily used
๏ A list of helpers (software and libraries)
๏ Space for your questions and exchange of experience (all together)
๏What is not in our session
๏ Solutions to all your problems
๏ Only a selection of interfaces. The topic is far too big

4. #engageug
What is an API
An application programming interface (API)
is an interface or communication
protocol between a client and a server intended
to simplify the building of client-side software.
We focus on consuming APIs from other systems!
(Source: en.Wikipedia.org).
„
„

5. #engageug
Security
SECURITY

6. #engageug
Authentication Planning
๏Questions to ask…
๏ Process
๏ Who knows the password, who sign in, who changes, how often?
๏ Method
๏ Username/Password, API key, OpenID/OAUTH, Obscurity?
๏ LAN? WAN? Internet? Public/Private Cloud?
๏ Potential eavesdroppers? Security precautions?
๏ What else?
๏ Any regulations to comply?
๏ Test-QA-Dev-Production staging?

7. #engageug
Authentication Planning
๏API Authentication at a glance…
๏ No Secret
๏ Authentication not needed for some reason
๏ Shared Secret (Credentials)
๏ We know a “thing” (password, api key, etc.)
๏ Generated Secret (Tokens)
๏ We know a “thing” to acquire a token
๏ Delegated Access (Tokens)
๏ Someone knows a “thing” to provide us a token
Where to
keep these
“thing”s?

8. #engageug
Keep in mind!
Forget about security
if credentials are
crappy!
Source: https://www.flickr.com/photos/30478819@N08/29613520138

9. #engageug
Best practise around security
๏Always run your code on the server
๏Clients and Users should not store/use the keys
๏ Beware of …
๏ LotusScript Debug
๏ Document Properties
๏ Malicious code to access data
๏Never trust your local network. Always use secure
channels (Use SSL!)

10. #engageug
Storing Credentials / Keys
๏Create a profile form, e.g. frm.fancyAPI
๏ Add a readers field
๏ Add yourself (Admin) and the Signer of the database
๏Create an agent, e.g. ag.fancyAPI
๏ Set security settings
๏ Run on behalf of [Database signer] (select the signer)
๏Run the code
๏ Run your Agent or script
๏ Call ag_fancyAPI
๏ Within ag_fancyAPI use
๏ db.GetProfileDocCollection(“frm.fancyAPI”).getfirstdocument

11. #engageug
Best practise around security
๏Additionally, you can use encrypted field
๏ Every ID file has Public/Private keys
๏ Including servers!
๏ So, native support for encryption!
๏ Example: “WebSSOConfig” (Public Addressbook)
๏The lifecycle of credentials
๏ Plan ahead!
๏ Do they expire? How often? Do they need changing?
๏Please, RTFM!

12. #engageug
Domino
How to secure your Domino server
Essentials

13. #engageug
Easy but helpful settings
Notes.ini
๏ Disable SSLv3 if you are still pre 9.0.1 FP9
๏ Disable_SSLv3=1
๏ Disable TLSv1
๏ SSL_DISABLE_TLS_10=1
๏ HTTPDisableServerHeader=1
๏ HSTS (HTTP Strict Transport Security)
๏ Protects against protocol downgrade attacks
๏ Declares browsers should only interact using HTTPS
๏ It’s actually a HTTP Response Header set via notes.ini

14. #engageug
HTTP Response Headers
HTTP Response Headers: the most important once:
๏ Strict-Transport-Security
๏ X-Frame-Options
๏ X-Content-Type-Options
๏ Content-Security-Policy
Set them for every hostname or IP your server is reachable from the internet.

15. #engageug
Strict-Transport-Security
Strict Transport Security
๏ Force clients only to connect via HTTPS
๏ Set completely in notes.ini
๏ Enabled by default since 9.0.1 FP3 IF2
๏ but: to short:
๏ HTTP_HSTS_MAX_AGE= 17280000
๏ and: does not include subdomains
๏ HTTP_HSTS_INCLUDE_SUBDOMAINS=1

16. #engageug
X-Content-Type-Options
X-Content-Type Options
๏ Reduces exposure to drive-by downloads and the risks of user uploaded
content that, with clever naming, could be treated as a different content-
type, like an executable.
๏ X-Content-Type-Options nosniff
๏ Domino 9: need to be set
๏ Domino 10: active by default
๏ can be disabled by notes.ini parameter starting with 10.0.1 FP4
๏ HTTP_DISABLE_X_CONTENT_TYPE_OPTIONS_NOSNIFF=1.
๏ Domino 11: active by default
๏ same as above

17. #engageug
Content-Security-Policy
Content-Security-Policy
๏ More complex only needed if you provide input fields
๏ Defines approved sources of content that the browser may load
๏ Can be an effective countermeasure to Cross Site Scripting (XSS) attacks
๏ You can whitelist approved sources
๏ Details about the options:
๏ https://scotthelme.co.uk/content-security-policy-an-introduction/

18. #engageug
Domino Administration
Domino Administration basics
๏ Update to the latest version of domino to be more secure!
๏ Do the minimum on Domino
๏ Disable Port 80 or at least redirect it to 443
๏ Hide your server type
๏ It’s 2022: use at least TLS 1.2
๏ Protect your users/customers with HTTP-Headers
๏ More security requires a Reverse Proxy
๏ They are built for that
See all details here. Source: https://en.rnug.ru/wp-content/uploads/2021/01/rnug-letsmakeyourdominowebserverrocksolid.pdf

19. #engageug
Integration Methods

20. #engageug
API Integration at a Glance
๏User-Initiated
๏ Notes Client / Web / Mobile
๏ User Interaction
๏ Implementation:
๏ Notes Client Actions
๏ Agents (LS / Java)
๏ XPages App (Java)
๏ OSGi Plugins (Java)
๏Unattended
๏ Scheduled / Event-triggered
๏ No user interaction / Background
๏ Implementation:
๏ Agents (LS / Java)
๏ Agent → XPages (Java)
๏ DOTS (Java)
๏ XOTS (Java)

21. #engageug
Language Selection
๏Java
๏ XPages
๏ Managed Beans, SSJS
๏ XOTS
๏ Plugins
๏ HTTP, DOTS
๏ Agents
๏ Java Agents
๏LotusScript
๏ Notes Client
๏ Actions, Events, etc.
๏ Agents
๏ Client-Side or Server-Side
๏JavaScript (limited)
๏ Web Browser
๏ SSJS
๏ App. Dev. Pack (?)

22. #engageug
Java Agents
๏Java agents? Really Why?
๏ We still need them!
๏ Unattended run (scheduled agents)
๏ RunOnServer
๏Try to avoid, if possible
๏ Compatibility problems
๏ Memory Leak with JAR files
๏ Tip: Put your JAR files into /jvm/lib/ext
๏ (Very) Low Performance

23. #engageug
Scheduling XPages Code
๏Why schedule XPages code?
๏ Agents are far from ideal
๏ Reuse the same codebase
๏ Modern(ish) development environment
๏How?
๏ Write your code
๏ /mydb.nsf/myAgent.xsp
๏ Create a scheduled agent
๏ Call “https://myserver.com/mydb.nsf/myAgent.xsp”

24. #engageug
DOTS
๏DOTS: Domino OSGi Tasklet Services
๏ Server Tasklets for Domino
๏ Tasklet: A lightweight server task
๏ Run background tasks in a lightweight scalable container
๏ Separate JVM and OSGi Container
๏ Various triggers
๏ Manual (console commands / socket trigger)
๏ Scheduled or on server start
๏ Triggered (Hooks through data events)

25. #engageug
DOTS
๏Some History
๏ 2010 - OpenNTF project contributed by IBM
๏ 2013 - Added to IBM Domino 9 Social Edition
๏ OpenSocial add-on
๏ Out of support (Internal Use Only)
๏ 2018 - Removed in Domino 10
๏ 2021 - Came back into Domino 12
๏ Installed with the Core product
๏ Support: Windows 64 and Linux 64
๏ Upgraded to Eclipse OSGi 4.6.2 (Neon 2)

26. #engageug
Lotus Script / Agents
๏Lotus Script / Agents
๏ Can do a lot (domino is able to work with other systems)
๏ Read / Write files
๏ Read / Write to SQL databases
๏ Read / Write SOAP Requests
๏ Read / Write REST APIs
๏ Remote control of other programs
๏ Can be invoked by
๏ Time: Scheduled Agent
๏ User: Action Button
๏ User: Events (eg opening a document)
๏Conclusion
๏ Usually works well. Implemented using non modern technology.

27. #engageug
Real World Examples
REAL WORLD EXAMPLES

28. #engageug
Real World Examples
๏Scheduling XPages Code
๏ Hybrid Application
๏ 80% XPages, 20% Notes
๏ Documents to be sent a remote service over REST API
๏ New content and critical changes => Upload immediately
๏ Other changes => Queue and Upload next hour
๏ Upload code complications
๏ Java SDK supplied, fails in Java agents
๏ Upload should be scheduled
๏ Single document upload triggered by XPages and/or Notes Client
๏All possible complications in a single project…

29. #engageug
Real World Examples
๏Scheduling XPages Code
๏ Upload business logic
๏ Java classes embedded in managed beans
๏ Utilise SDK and Apache HttpComponents
๏ XPages code
๏ Multimode: Upload specific document or process queue
๏ Java Agents
๏ Notes client can trigger Java agent to upload single document
๏ Scheduled Java agent runs every hour to trigger upload
๏ Security
๏ XPages code should only be triggered by a Java Agent
๏ Precautions: IP limitations, authentication, temporary backend document, etc.

30. #engageug
Real World Examples
๏Uploading Data via FTP
๏ Problem:
๏ Newsletters and Campaign Communication
๏ Forced migration to a cloud-based e-mail marketing service
๏ Upload text files to a Secure FTP server
๏ Transient Customer data needed daily and weekly
๏ Custom Campaign data to be uploaded instantly
๏ Queries and Uploads take too much time and resource
๏ Security is the top-priority!
๏ Solution:
๏ DOTS Tasklets

31. #engageug
Real World Examples
๏Uploading Data via FTP
๏ Tasklet can handle long-running upload process
๏ Watching queue every minute
๏ Very small footprint for queue monitoring
๏ Reuse Java code already developed before
User builds a target query
for the campaign
Predefined target lists for
newsletters
UploadJob Queue
DOTS Tasklet
(Scheduled / Manual)
•Fetch next UploadJob
•Run query
•Convert to CSV file
•Compress
•Upload to SCP Server
Upload Job #1
Upload Job #2
…
Upload Job #N

32. #engageug
Real world examples
๏Expense Workflow - Accounting App Integration
๏ Very common scenario
๏ Expense form
๏ Reads and caches Project codes, Customers, etc.
๏ Writes back approved expense amounts
๏ Various Solutions
๏ SQL calls to Stored Procedures
๏ LotusScript using a DLL for integration
๏ Web Services (SOAP, REST, etc)
๏ Text file exchange

33. #engageug
Real world
SQLInterface class

34. #engageug
Real world
SQLInterface class: Init

35. #engageug
Real world
SQLInterface class: run sql

36. #engageug
Real world
SQLInterface class: demo of usage

37. #engageug
I am confused!?* What should I use?
๏Whenever you can, use an REST JSON API
๏ Instead of interacting with databases directly (LSX, ODBC)
๏ Instead of file transfers (Export / Import)
๏ And yes, instead of using HEI
๏ Use JAVA. Lotus Script implementation is far behind!
๏But keep in mind
๏ APIs can change very frequently (e.g. Microsoft M365)
๏ You don’t have control on details
๏ You don’t know the details (but most of the time, you are not interested
in)

38. #engageug
Common tasks
Common tasks

39. #engageug
The question of life, the universe and all the rest
There are only two hard things in Computer
Science:
- Cache invalidation and
- Naming things
(Source: Phil Karlton (Former Netscape developper))
„
„

40. #engageug
Caching: How to
๏Caching: Main challenges
๏ When to provide
๏ When you request the same data multiple times
๏ The data you receive do not change very often or you know exactly when
๏ Time required to obtain the data is high
๏ YES, all of them!
๏ When to clean (cache invalidation)
๏ As early as possible without loosing the advantages
๏ Time based
๏ Data change recognised

41. #engageug
Caching: Javascript Example

42. #engageug
Java Libraries
๏Utilise Libraries
๏ Practical Development
๏ OpenNTF Domino API
๏ Apache Commons
๏ Network Connections (REST APIs)
๏ Performance, compatibility, simplicity, protection from implementation
changes
๏ HttpComponents (Apache Commons)
๏ Misc.
๏ POI (Java API for Microsoft Documents)

43. #engageug
OSGi Plugins
๏Make use of OSGi Plugins
๏ Larger integration projects
๏ Multiple applications/business processes
๏ Broader functionality (UI modules, server-wide caching etc.)
๏ Easier implementations for various scenarios
๏ e.g. OAuth authentication
๏ Libraries
๏ XPages run in a secure container → Compatibility problems
๏ Jackson, Gson, some JDBC-wrappers, etc.

44. #engageug
Logging
๏Use modern logging
๏ Log Levels
๏ Less is more, more is not enough…
๏ Consider stages
๏ Dev/Test and Production
๏ Tip: notes.ini
๏ Try OpenLog by Julian Robichaux
๏ Lotusscript: OpenLog
๏ XPages: XPages OpenLog Logger (Paul Withers)
๏ Java (XPages, DOTS, OSGi): XLogback (Serdar Basegmez)

45. #engageug
JSON Processing using JAVA
๏What to process? How to process?
๏ Create/Consume JSON String
๏ Maps, Arrays, etc.
๏ IBM Commons, preloaded in Domino
๏ JSON ↔ Java Object conversions
๏ Direct Conversion between POJO and JSON
๏ Gson, Jackson, etc.
๏ JSON Streaming
๏ Read/Write large files/network streams
๏ Optimised CPU/Memory usage
๏ Gson, Jackson, etc.

46. #engageug
JSON Processing
๏Java Options
๏ IBM Commons JSON Library
๏ Included in Domino
๏ Encapsulated Maps and Lists
๏ Gson (Google)
๏ Mid-weight, easy to use
๏ Object mapping, POJO conversion, streaming, etc.
๏ Jackson (FasterXML)
๏ More features (JAX-RS provider, language bindings, etc.)
๏ Tip: Gson and Jackson use Reflection
๏ Encapsulate into plugin
๏ Disable security (java.pol file)

47. #engageug
JSON Processing using Javascript
๏How to process?
๏ Basic functionality of Javascript
๏ Very fast
๏ Very easy
๏ JSON ↔ JS Object conversions
๏ JSON.stringify
๏ JSON.parse

48. #engageug
JSON Processing using Lotus Script
๏How to process?
๏ Two main classes to work with
๏ NotesJSONNavigator (reading)
๏ NotesJSONObject (writing)
๏ JSON ↔ JS Object conversions
๏ You have to run through the object
๏ No direct access
๏ Avoid to use it, if you work with large objects
๏ Buggy Has some difficulties
๏ Slow Needs time for processing and traversing
๏ Alternative (if you really have to, maybe)
๏ ls.snapps.JSONReader from OpenNTF

49. #engageug
Java Tricks
๏JavaOptionsFile
๏ Adding JVM Options using notes.ini
๏ Setting TLS protocols (Important for Java Agents)
๏ Additional debugging, tweak third party libraries
๏ Client and Server
๏ Alternative Form: JavaUserOptionsFile
๏ Only custom options (after “-D”)
๏ DOTS uses a different JVM
๏ DOTS_JavaOptionsFile

50. #engageug
Java Tricks
๏SSL and TLS issues
๏ Issues with HTTPS connections in Java agents
๏ Before Domino 9.0.1FP7,
๏ Use JavaOptionsFile to force TLS1.2
๏ Technote KB0028812
๏ SSL Handshake Errors
๏ Certification Authority information outdated
๏ Add trusted certificates to CACERTS
๏ Technote KB0035853
๏ Much older versions
๏ Try your luck with Bouncy Castle

51. #engageug
Litte helpers
TOOLS / UTILITIES

52. #engageug
Dev / Testing helper software / apps
๏Postman
๏PAW
๏SOAPUI
๏CURL
๏JSONLINT

53. #engageug
Security Assessment / test websites
Check your server
๏ SSL Labs (www.ssllabs.com)
๏ Grade change in January 2020 if you provide TLS 1.0/1.1

54. #engageug
Security Assessment / test websites
๏ https://securityheaders.com/
๏ Not every header is for every site
๏ Check your site functionality!
HTTP Response Headers can break your site

55. Any questions?