CollabSphere 2021 - DEV114 - The Nuts and Bolts of CI/CD With a Large XPages And REST App
•Download as PPTX, PDF•
0 likes•992 views
Though Domino makes working with build servers and CI/CD pipelines difficult, it is possible to do so even with complex applications. This session will discuss the specifics of using several OpenNTF projects - NSF ODP Tooling, the Jakarta XPages Runtime, and XPages Jakarta EE Support - as well as open-source technologies such as Docker to build, test, and deploy Java-based Domino applications for testing and staging. This builds on previous sessions about the NSF ODP Tooling and Maven generally.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to CollabSphere 2021 - DEV114 - The Nuts and Bolts of CI/CD With a Large XPages And REST App
Similar to CollabSphere 2021 - DEV114 - The Nuts and Bolts of CI/CD With a Large XPages And REST App (20)
More from Jesse Gallagher
Recently uploaded
Recently uploaded (20)
CollabSphere 2021 - DEV114 - The Nuts and Bolts of CI/CD With a Large XPages And REST App
- 1. THE NUTS AND BOLTS OF CI/CD WITH A LARGE XPAGES AND REST APP IT'S... TECHNICALLY DOABLE
- 2. JESSE GALLAGHER CTO - I KNOW SOME GUYS IP MANAGER - OPENNTF HTTPS://FROSTILLIC.US @GIDGERBY
- 4. The Moving Parts Core set of OSGi plugins XPages Library JAX-RS–based REST services A handful of compile-time tests NPM-build–based JavaScript applications Some delivered from OSGi One delivered via NSF A set of NSFs Some XPages apps Most Notes client apps Most not properly run with source control A set of JEE webapps The main XPages app One of the JS apps Testcontainers-based tests
- 5. The Toolchain Maven maven-bundle-plugin (used to use Tycho) frontend-maven-plugin NSF ODP Tooling Eclipse (Usually) NPM (via frontend-maven- plugin) XPages Jakarta EE Support Open Liberty XPages Runtime project Testcontainers (Docker)
- 6. Opening Caveat Every explanation in this presentation comes with an implicit "...but it gets more complicated in practice"
- 7. MAVEN
- 8. Brief Aside: Maven It’s not (inherently) scary! Just look how cute that owl is Maven is a tool used for build automation It’s primarily used for Java projects, but is adaptable to many things via plugins A “pom.xml” file is a clear giveaway of Maven’s use We use it here because it allows for cleanly- reproducible builds across systems, without environment-specific details
- 9. Tycho vs. MBP Historically, we used Tycho to manage the OSGi build It presents an OSGi environment very similar to Domino It can strongly enforce expectations and dependencies Good support for OSGi-based test suites with Notes environments I switched to maven-bundle-plugin for added flexibility It makes it much easier to use normal Maven dependencies ...but it requires a lot more knowledge of the "gotchas" of the Domino stack I still hand-tweak OSGi options, and the compile step doesn't validate this There are other options (bnd-maven-plugin), but I don't know them enough yet https://frostillic.us/blog/posts/2019/8/22/converting-tycho-projects-to-maven-bundle-plugin-initial-phase
- 10. OpenAPI Generation <plugin> <groupId>io.openapitools.swagger</groupId> <artifactId>swagger-maven-plugin</artifactId> <version>2.1.5</version> <executions> <execution> <id>generate-openapi</id> <goals> <goal>generate</goal> </goals> </execution> </executions> </plugin> We use swagger-maven-plugin and swagger-annotations to generate an openapi.yaml file during the build
- 12. Runtime Independence The core parts of the app target three main environments: XPages on Domino (OSGi) Servlets on Domino (OSGi) .war-based Jakarta EE (not OSGi) Can't make a lot of assumptions!
- 13. RuntimeEnvironment Idiom We use a RuntimeEnvironment type to handle differences Contains methods like getSession(), getServletRequest(), etc. Each target runtime has an implementation that does the dirty work Core code calls RuntimeEnvironment.getSession() and works anywhere It gets fiddly! Error handling, detecting the right runtime, etc.
- 14. Balancing Dependencies maven-bundle-plugin makes third-party dependencies easier ...but Domino still makes it hard Some dependencies need to be re-packaged for Domino's OSGi environment Some are safest to avoid due to Domino's polluted classpath (Guava)
- 15. RuntimeEnvironment Links https://frostillic.us/blog/posts/2020/6/18/the-runtimeenvironment-idiom
- 16. NPM-BASED BUILDS IN MAVEN
- 17. frontend-maven-plugin frontend-maven-plugin allows you to run various frontend-dev tools (npm, yarn, webpack, etc.) based on configuration in a Maven pom We have the JS apps in a separate directory away from Java The build config specifies node+npm versions and the target directory One project copies the results into an OSGi plugin, another copies them into an NSF during build
- 18. Example Use Install Local Node + NPM `npm install` `npm run build` <execution> <id>install node and npm</id> <goals> <goal>install-node-and-npm</goal> </goals> <phase>generate-resources</phase> </execution> <execution> <id>actionItems install</id> <goals> <goal>npm</goal> </goals> <phase>generate-resources</phase> <configuration> <workingDirectory>${buildDir}</workingDir ectory> </configuration> </execution> <execution> <id>actionItems build</id> <goals> <goal>npm</goal> </goals> <phase>generate-resources</phase> <configuration> <workingDirectory>${buildDir}</workingDir ectory> <arguments>run build</arguments> </configuration> </execution>
- 20. NSF ODP TOOLING
- 21. NSF ODP Tooling The glue for working with NSFs: Compiles ODPs to NSF Updates ODPs for server-side changes outside Git Provides DXL and XSP autocomplete in Eclipse
- 22. Compilation Compiles ODPs into NSFs Uses fresh build of the OSGi plugins to ensure compatibility Runs in a local Equinox environment similar to Domino Sets $TemplateBuild fields
- 23. Updating Other-Developer ODPs <build> <plugins> <plugin> <groupId>org.openntf.maven</groupId> <artifactId>nsfodp-maven-plugin</artifactId> <configuration> <databasePath>Some/Server!!db/app.nsf</databasePath> </configuration> </plugin> </plugins> </build> Configure ODP projects for the canonical source:
- 24. Updating Other-Developer ODPs #!/usr/bin/env bash set -e REPO_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/.." find $REPO_DIR/osgi-plugin/nsfs/ -name pom.xml -not -path */nsf-dashboard/pom.xml -not -path */nsf-plans/pom.xml -print0 | xargs -0 -n 1 sh -c 'mvn org.openntf.maven:nsfodp-maven- plugin:3.5.0:generate-odp -X -f $0 || exit 255' Update ODPs for projects other than those properly done in Git:
- 25. XML Autocomplete Provides "good enough" XML autocomplete for XSP Covers core and custom controls, but not third-party library controls No preview pane, but eh... DXL autocomplete uses the provided- with-Domino schema files
- 26. NSF ODP Links https://frostillic.us/blog/posts/2018/3/5/301B92A0DF1F2CAF85258247007 BD28F https://frostillic.us/blog/posts/2020/8/27/nsf-odp-tooling-setting-up-jenkins- builds https://www.openntf.org/main.nsf/project.xsp?r=project/NSF%20ODP%20 Tooling https://github.com/OpenNTF/org.openntf.nsfodp Last year's session: https://youtu.be/7MeCMz0F-vM
- 27. XPAGES JAKARTA EE SUPPORT
- 28. XPages Jakarta EE Support Acts as a "platform update" for the app Brings in, in particular, EL 3 and JAX-RS 2.1, much newer than the built-in variants Also brings in CDI, which is used fully in the webapps and REST services Like managed beans, but much better Not required for the rest of this, but it sure is nice
- 29. Example JAX-RS Resource @Path("/serverInfo") @PermitAll public class ServerInfoResource { @Inject private ServerInfo serverInfo; @GET @Produces(MediaType.APPLICATION_JSON) @Tag(name = TAG_EXTERNALPORTAL) @Operation(summary = "Get information about server-specific resources", description = "...") public ServerInfo get() { return serverInfo; } }
- 30. XPages Jakarta EE Links https://frostillic.us/blog/posts/2018/6/3/6AC99D0B866A92AA852582A 1006C2FA6 https://www.openntf.org/main.nsf/project.xsp?r=project/XPages%20Ja karta%20EE%20Support https://github.com/OpenNTF/org.openntf.xsp.jakartaee
- 32. XPages Runtime Open-Source project that allows running XPages in a normal Servlet container I've only run it in Liberty, but others should work Turns out all the stuff from pre-OSGi is still in there Requires some tweaks with third-party libraries, and code has to not assume it's in OSGi This enables CI deployment servers and full automated tests
- 33. XPages Runtime Some shim code handles bootstrapping ODA, etc. JAX-RS services use Liberty's built-in support The webapp projects use symlinks to point to the ODP for XPages, Java, etc.
- 34. XPages Runtime - Tweaks Make no assumptions about OSGi The RuntimeEnvironment idiom goes a long way The IBM Commons ExtensionManager accounts for this Duplicate any plugin.xml extensions in META-INF/services With a JEE server, no JAX-RS runtime (Wink, RestEasy, etc.) is needed On Liberty, don't enable JSF or JSP features (conflict with XPages)
- 35. XPages Runtime - Tweaks @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { NotesThread.sinitThread(); Set<Long> handles = new HashSet<>(); if(!Factory.isInitialized()) { Factory.initThread(new Factory.ThreadConfig(new Fixes[0], AutoMime.WRAP_32K, true)); } if(sessions.get() == null) { sessions.set(createSession((HttpServletRequest)request, handles)); sessionsAsSigner.set(createNativeSession()); sessionsFull.set(createNativeSession()); if(StringUtil.isEmpty(expectedDbPath)) { throw new IllegalStateException(ENV_DBPATH + " environment variable should be set to the Domino API path to the context database"); } try { databases.set(sessions.get().getDatabase(expectedDbPath)); } catch(Throwable e) { } if(!checkAccess(databases.get())) { databases.set(null); } } // ... } Use a @WebFilter to provide context sessions/database (There's a lot more to this in practice)
- 36. XPages Runtime - Tweaks @Override public Object resolveVariable(FacesContext facesContext, String varName) throws EvaluationException { switch(StringUtil.toString(varName)) { case "session": case "sessionAsSigner": case "sessionAsSignerFullAccess": return ODAFilter.sessions.get(); case "database": return ODAFilter.databases.get(); } // ... } Use a VariableResolver to find custom session/database
- 39. Dockerized XPages Apps We start with the open-liberty:full-java8-openj9 image Then we bring in the Notes runtime parts of the official HCL Domino Docker images Then we bring in the built application WAR It's already been adapted to read the context server/DB from the environment Et voilà ! This is how I deploy CI servers with Jenkins nowadays
- 40. Dockerfile # Configure the runtime image FROM --platform=linux/amd64 open-liberty:full-java8-openj9 # Bring in the Liberty app and configuration COPY --chown=default:users webapp.war /apps/ COPY config/* /config/ # Bring in the Domino runtime COPY --from=domino-docker:V1101_03212020prod /opt/hcl/domino/notes/11000100/linux /opt/hcl/domino/notes/latest/linux COPY --from=domino-docker:V1101_03212020prod /local/notesdata /local/notesdata # Bring in our Domino config and assign the data directory to the default user USER root COPY --chown=default:users notesdata/* /local/notesdata/ RUN mkdir -p /local/notesdata/IBM_TECHNICAL_SUPPORT RUN chown -R default:users /local/notesdata USER default ENV LD_LIBRARY_PATH "/opt/hcl/domino/notes/latest/linux" ENV NotesINI "/local/notesdata/notes.ini" ENV Notes_ExecDirectory "/opt/hcl/domino/notes/latest/linux" ENV Directory "/local/notesdata" ENV PATH="${PATH}:/opt/hcl/domino/notes/latest/linux:/opt/hcl/domino/notes/latest/linux/res/C" ENV InitDominoRuntime="1" # DB context env var is set when launching the container and read by the app EXPOSE 8080 8443
- 41. Containerized XPages Apps Links https://frostillic.us/blog/posts/2020/6/28/weekend-domino-apps-in- docker-experimentation https://frostillic.us/blog/posts/2020/8/13/executing-a-complicated-osgi- nsf-surefire-npm-build-with-docker https://github.com/jesse-gallagher/domino-docker-war-example
- 42. TESTCONTAINERS
- 43. Testcontainers Testcontainers lets us run the full webapp (XPages + REST) with true headless browsers Requires Docker, a containerized app, and patience I've only gone so far with it - with work, this could run full automated UI tests of XPages and JavaScript apps
- 44. Testcontainers - Maven Process Use maven-resources-plugin to copy a Dockerfile, built WAR, and Notes runtime files to scratch space Use dockerfile-maven-plugin to build the app image `<pullNewerImage>false</pullNewerImage>` to avoid trying to resolve official Domino images from Docker Hub Use maven-failsafe-plugin to run all `it.*` classes Set environment `TESTCONTAINERS_RYUK_DISABLED=true` to avoid trouble I hit when running on Jenkins and instead manage lifecycles manually
- 45. Testcontainers - Example Code driver.get(getContainerRootUrl() + "index.xsp"); assertEquals("Expected Page Title", driver.getTitle()); // Test all CSS and JS to make sure there's no 404s URI rootUri = URI.create(getRootUrl()); driver.findElements(By.xpath("//link[@rel='stylesheet']")) .stream() .map(link -> link.getAttribute("href")) .map(href -> rootUri.resolve(href)) .map(this::decontainerize) // etc. .forEach(uri -> checkUrlWorks(uri, client)); driver.findElements(By.xpath("//script[@href]")) // etc.
- 47. QUESTIONS