OpenNTF Webinar Series: DQL with John Curtis September 2020

OPENNTF WEBINARS
September Webinar - What's New with Domino Query
Language with John Curtis from HCL

AGENDA
• Welcome – Howard Greenberg
• John Curtis
• Questions – Graham Acres

ASKING QUESTIONS
• First Question – Will this be recorded?
• Yes, view on YouTube!!!
• https://www.youtube.com/user/OpenNTF
• Use the Questions Pane in GoToWebinar
• We will get to your questions at the end of
the webinar
• The speakers will respond to your questions
verbally
• (not in the Questions pane)
• Please keep all questions related to the
topics that our speakers are discussing!!!
• Unrelated Question => post at:
• http://openntf.slack.com/

THANKS TO THE OPENNTF SPONSORS
• HCL made a significant contribution to help our
organization
• Funds these webinars!
• Contests like Hackathons
• Running the organization
• Prominic donates all IT related services
• Cloud Hosting for OpenNTF
• Infrastructure management for HCL Domino and Atlassian
Servers
• System Administration for day-to-day operation

THIS IS OUR COMMUNITY
• Join us and get involved!
• We are all volunteers
• No effort is too small
• If your idea is bigger than you can do on your own, we
can connect you to a team to work on it
• Test or help or modify an existing project
• Write guides or documentation
• Add reviews on projects / stars on Snippets

NEXT WEBINAR
• October 22, 2020
• Michael Smith – A Flexible View Control for XPages
• Signup at https://openntf.org/webinars

UPCOMING EVENTS
• Let’s Connect Virtual Event – September 29-30
• https://letsconnect.world/agenda/
• Collabsphere – October 27-29
• https://collabsphere.org/ug/cs2020.nsf/index.html
• DNUG – monthly online events
• https://dnug.de/en/dnug47online-2/

8 | Copyright © 2019 HCL Technologies Limited | www.hcltech.com8 | Technology for the Next Decade, Today Copyright © 2019 HCL Technologies Limited | www.hcltechsw.com
What's New with Domino Query
Language (and Domino Appdev)
John Curtis
Software Architect, Project Lead, Domino Development
HCL Digital Solutions
@john_d_curtis_
http://www.jdcurtis.blog
john_curtis@pnp-hcl.com

Copyright © 2020 HCL Technologies Limited | www.hcltechsw.com
DQL Evolution
October 2018
December 2018
1H-2021
March 2020
• Consolidation of Domino search and
results processing
• Constant improvement in
• Performance
• Capability
• Multiple venues and
programming models
December 2019
DQL V1
FT Support
Inboard catalog
FT Support completed
Statistics
Performance
Results processing (sort, federation, joins)
Formula Language inclusion

v11.0.1 DQL Features
Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com
• Complete full text index utilization – enable numeric
and date terms to take advantage of FT indexes
▪ 5-200X improvement in speed
▪ DQL plan and costing executes FT terms first, then views,
then NSF scans
▪ Seamless – existing DQL syntax will work automatically on
terms and across Boolean operators
▪ Caveats/limitations:
➢ Single-occurrence fields only (Domino does NOT FT index multiple
field values of numbers and dates)
➢ No TIMEs in the datetime values, only dates

v11.0.1 DQL Features
• DQL search strategy by term type
Term type Full Text Search View Search NSF Scan
field = ‘text’? No Yes Yes
field contains (‘Text’) Yes No No
field = 123122 Yes * Yes Yes
field > ‘text’ No Yes Yes
field < 123122 Yes * Yes Yes
field = @dt(‘2020-09-17’) Yes * Yes Yes
field > @dt(‘2020-09-17’) Yes * Yes Yes
field = @dt(‘2020-09-17T11:00:00.00’) No Yes Yes
* = single occurrence only (FT search does NOT handle ranges or lists)

v11.0.1 DQL Features (continued)
• DQL statistics
▪ Avg, max, min by term type
and overall
▪ “show stat query*”
• Enhanced Boolean
optimization
▪ Deeper results injection
▪ Reduces view and especially
NSF scan processing for
much faster query times

v12 Early Access September 2020 drop DQL Features
• DQL client/server
▪ DominoQuery working against remote databases (not
supported till now)
▪ Transaction appears in the “show trans” list from the
Domino console
▪ Full support for substitution variables, entire DQL syntax

v12 Early Access September 2020 drop DQL Features (continued)
• Embedded Formula Language
▪ Syntax
▪ @FL or @formula (case insensitive)
▪ Embedded syntax enclosed in single quotes
@fl(‘@length > 2320)
@FORMULA(‘@left(fld1;3) = "abl" or @left(fld2;2) = "ab“’)
▪ Provides
▪ Intra-document terms (between different fields)
▪ Full expression support
▪ HUGE library of existing functionality

v12 Early Access September 2020 drop DQL Features (continued)
• Embedded Formula Language (continued)
▪ Support for substitution variables
@fl(‘@length’) > ?lengthvar
@formula(‘@left(fld1;3)’) = ?fld1var or @FL(‘@left(fld2;2)’) = ?fld2var
▪ Complete compatibility with the rest of DQL
▪ All formula language terms satisfied via NSF scanning
▪ All ANDed/ORed terms satisfied with a single pass
▪ Current implementation
▪ 256-byte maximum Formula Language syntax allowed
▪ No substitution variable support for an entire FL string

DEMO

v12 Work in progress
• Results processing
▪ Extensive handling of document collections
➢ NOT just DQL – ALL document collections
▪ Across multiple databases and database schemas
▪ Flexible computed values everywhere
▪ Sorting (including categorization)
▪ Combine rules (joins, lookups, etc.)
▪ Aggregate function support
▪ Impossible to avoid complexity

• Results processing (continued) – the details
▪ (Notes)QueryResultProcessor
➢ New backend (Java/Lotusscript) class
➢ Supplied lists of data and other classes working together
➢ Builds on known classes never combined or used this way before
➢ Extensive capabilities, multiple output formats and objects
➢ Low code? Not so much, but let’s build one to see

• (Notes)QueryResultProcessor (continued) – methods
▪ AddSortColumn(programmatic name, column title, sort order,
hidden, categorized)
▪ Akin to creating a view column on the fly
▪ In fact, one of the output types IS a view
▪ May add additional attributes but those shown should be
familiar from use in Designer
▪ REMEMBER the programmatic name – it will appear again in
other calls
▪ If NOT overridden, will be used as a field name to fetch

▪ AddDocumentCollection((Notes)DocumentCollection doccol,
collection name)
… or …
▪ AddDominoQuery ((Notes)DominoQuery, collection name)
▪ (Notes)DocumentCollection can be created against any
database using any means (db.search, ftsearch,
getalldocumentsbykey or DQL)
▪ DominoQuery option will also execute DQL for you
▪ Remember collection name – will appear later

▪ AddFormula(document collection name, programmatic column
name, formula)
➢ Used to override the programmatic name with a formula to
compute and fetch values for processing by the results engine
➢ Can use one call to override the formula for multiple document
collections using wildcards
Addsortcolumn (“name”, “Name Column Title”, “ascending” … ) …
AddDocumentCollection(doccol, “doccol1”) .. e.g. from db.search - database 1
AddDocumentCollection(doccol2, “doccol2) .. e.g. from ftsearch - database 2
AddFormula(“doccol*”, “name”, “@Propercase(NameField)”)

▪ AddCombineRule (left programmatic name, rule, right
programmatic name)
➢ Prior to sort or view create, combine data using the sort columns
(with applied formulas) using the specified rule (or method)
➢ “Join” is the most familiar rule type, also keyword lookup ++
▪ SetOutputType (type, viewname, streamtype)
➢ View (with name) or stream (with streamtype)
▪ Execute (run the result processor)

• (Notes)QueryResultProcessor how it works
Sort
doccol1
DQL1
ftdoccol
Dbsearch
doccol
DQL2
Combine Fetch
View
Specified
output
Input
(Notes)DocumentCollections
Optional join/lookup
Quick fetch
including DQL executes
(uses formula overrides
Domino sort service
(with streaming optimization)
or

• (Notes)QueryResultProcessor additional information
▪ NEW - Sort package is decoupled from view building
➢ In stream mode, we don’t need to wait for a completed view
build
▪ Separate column formulas for each DocumentCollection
▪ Views created can be used and discarded or retained
➢ Given input DocumentCollections, views cannot be refreshed
➢ Document security problem (multi-db documents have multiple
readers and authors lists)
▪ Join strategies – sort/merge and nested loop using views or
FT indexes

Early Access September 2020 drop non-DQL Feature
• Domino Transaction Support in Lotusscript/Java
▪ (Notes)Database object has 3 new methods
➢ TransactionBegin – starts a unit of work
➢ TransactionCommit – completes a unit of work
➢ TransactionRollback – aborts everything since TransactionBegin
▪ Automatic rollback happens when
➢ Database object is closed or goes out of scope
➢ Agent is terminated
▪ Large, uncommitted transactions can cause log file full (crash)
▪ Transactions cannot nest (one db, one level, one at a time)

v12 Work in progress – non-DQL
• Field length limit relief
▪ Aha idea - https://domino-ideas.hcltechsw.com/ideas/DDXP-I-5
▪ Traditional limits:
➢ 32k limitation on field size
➢ 62k limitation on document summaries
• Other problems:
➢ Better error messages when limits are hit
➢ Support in views and folders for larger fields
➢ Designer support (code size)
➢ Other Domino limitations – like 32k Lotusscript arrays

• Field length limit relief (continued)
• (load) compact <db> -ls (or -largesummary) on
➢ First delivered ~ 9.01FP8
➢ New max sizes:
✓ All fields (text lists or number or time ranges) – 64K
✓ Document summary size - 16MB (had been 62k)
➢ Requires compact to ODS 52
• Have you tried this?

▪ Multiple core API and persistent data structure changes
▪ Will require a new ODS and large summaries enabled
▪ Only the max summary size (16MB) will remain –
➢ That is, the SUM of the lengths of all summary fields
➢ No size limit within the 16MB for individual summary fields
▪ Nonsummary fields can be up to 4GB (not a typo)
▪ Retain a hard limit on view-indexable fields (64K)

• “Field is too large (32K) or View's column & selection formulas
are too large”
➢ Catch-all, multiple venues and error states throw the error
➢ Often is the SUM of field lengths generating the error – no one
field is responsible
• Effort for v12 – separate the states into 2 types and improve –
➢ Overall summary size – provide a document summary “map” to
show sizes only
➢ Individual fields – include field name in the message

OpenNTF Webinar Series: DQL with John Curtis September 2020

More Related Content

What's hot

Similar to OpenNTF Webinar Series: DQL with John Curtis September 2020

Recently uploaded

OpenNTF Webinar Series: DQL with John Curtis September 2020