Sling is an established web application framework, with a multitude of core features and extensions. It has a very productive inner loop, with OSGi bundle deployment, JCR content editing and live configuration updates. The less-told story is how an application should be assembled, configured, deployed, and monitored. In this talk we will present the main approaches for bootstrapping, deploying, updating, and monitoring Sling-based applications, based on Open Source tools and libraries. The participants will gain a better understanding of the options available for managing their own Sling-based application and will be able to minimise the effort needed to manage such an application.

1. EUROPE'S LEADING AEM DEVELOPER CONFERENCE
25TH – 27TH SEPTEMBER 2023
Sling Applications - a DevOps perspective
Robert Munteanu, Adobe

2. About me
2

3. Outline
Define
Deploy
Monitor
Maintain
3

4. [Define] OSGi Feature Model
{
"bundles":[
{
"id":"org.owasp.encoder:encoder:1.2.3",
"start-order":"5"
},
{
"id":"commons-codec:commons-codec:1.16.0",
"start-order":"5"
}
]
}
4

5. [Define] Maven tooling
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for pospai 1.0-SNAPSHOT:
[INFO]
[INFO] pospai ............................................. SUCCESS [ 0.157 s]
[INFO] pospai - Core ...................................... SUCCESS [ 2.074 s]
[INFO] pospai - UI Users .................................. SUCCESS [ 0.446 s]
[INFO] pospai - UI Apps ................................... SUCCESS [ 0.088 s]
[INFO] pospai - Launcher .................................. SUCCESS [ 1.819 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
5

6. [Define] Reference blueprints
<aggregate>
<filesInclude>*.json</filesInclude>
<includeArtifact>
<groupId>org.apache.sling</groupId>
<artifactId>org.apache.sling.starter</artifactId>
<version>12</version>
</includeArtifact>
</aggregate>
Compact
Low-effort start of upgrade
Lifecycle tied to upstream release
Opaque
All or-nothing for certain features
6

7. [Define] Replicate blueprints
src/main/features/
├── app
│ ├── pospai.json
│ └── pospai-repoinit.txt
└── platform
├── base.json
├── base-repoinit.txt
├── boot.json
├── caconfig.json
└── caconfig-repoinit.txt
(snip)
Fine-grained control, incremental updates possible
More effort in keeping up-to-date
7

8. [Define] Secrets management
"org.apache.sling.extensions.oidc_rp.impl.OidcConnectionImpl~google": {
"name": "google",
"baseUrl": "https://accounts.google.com",
"clientId": "$[secret:google/clientId]",
"clientSecret": "$[secret:google/clientSecret]",
"scopes": ["openid"]
}
8

9. [Build] Application assembly
<plugin>
<groupId>org.apache.sling</groupId>
<artifactId>slingfeature-maven-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<aggregates>
<aggregate>
<classifier>app</classifier>
<filesInclude>platform/**/*.json</filesInclude>
<filesInclude>app/*.json</filesInclude>
</aggregate>
</aggregates>
<!-- snip -->
</configuration>
</plugin>
9

10. [Build] Application analysis
<plugin>
<groupId>org.apache.sling</groupId>
<artifactId>slingfeature-maven-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<scans>
<scan>
<includeClassifier>app</includeClassifier>
</scan>
</scans>
<!-- snip -->
</configuration>
</plugin>
10

11. [Build] Smoke ITs
<plugin>
<artifactId>docker-maven-plugin</artifactId>
<configuration>
<!-- snip -->
<run>
<ports>http.port:8080</ports>
<wait>
<http>
<url>http://localhost:${http.port}↳
/system/health.txt?tags=startup</url>
<status>200</status>
</http>
</wait>
</run>
</configuration>
</plugin>
11

12. [Build] Health checks
"configurations": {
"org.apache.felix.hc.generalchecks.BundlesStartedCheck": {
"hc.tags": ["startup"], "useCriticalForInactive": true
},
"org.apache.sling.jcr.contentloader.hc.BundleContentLoadedCheck": {
"hc.tags": ["startup"],
},
"org.apache.felix.hc.generalchecks.FrameworkStartCheck": {
"hc.tags": ["startup"], "targetStartLevel:Integer":"30"
},
"org.apache.felix.hc.generalchecks.ServicesCheck": {
"hc.tags": ["startup"],
"services.list": [
"org.apache.sling.jcr.api.SlingRepository"
]
}
}
12

13. [Build] Security checks (Maven)
$ mvn org.sonatype.ossindex.maven:ossindex-maven-plugin:audit
[ERROR] Failed to execute goal ossindex-maven-plugin:audit (default-cli)
on project org.apache.sling.starter: Detected 1 vulnerable components:
[ERROR] xerces:xercesImpl:jar:2.6.2:test;
https://ossindex.sonatype.org/(...)
[ERROR] * [CVE-2009-2625] CWE-400: ('Resource Exhaustion') (5.0);
https://ossindex.sonatype.org/(...)
[ERROR] * [CVE-2012-0881] CWE-399 (7.5);
https://ossindex.sonatype.org/(...)
[ERROR] * [CVE-2013-4002] CWE-400: ('Resource Exhaustion') (7.1);
https://ossindex.sonatype.org/(...)
[ERROR] * [CVE-2022-23437] CWE-835: ('Infinite Loop') (6.5);
https://ossindex.sonatype.org/(...)
[ERROR] * [CVE-2017-10355] CWE-833: Deadlock (5.9);
https://ossindex.sonatype.org/(...)
13

14. [Build ] Security checks (Container)
$ trivy image --severity HIGH,CRITICAL sample/app:snapshot
sample/app:snapshot (debian 11.3)
Total: 46 (HIGH: 36, CRITICAL: 10)
┌──────────────┬────────────────┬──────────┬─────────────────────┬───────────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Status │ Installed Version │ Fixed Version │ Title │
├──────────────┼────────────────┼──────────┼─────────────────────┼───────────────────┼──────────────────────────────────────────────────────────────┤
│ bash │ CVE-2022-3715 │ affected │ 5.1-2+b3 │ │ a heap-buffer-overflow in valid_parameter_transform │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-3715 │
├──────────────┼────────────────┼──────────┼─────────────────────┼───────────────────┼──────────────────────────────────────────────────────────────┤
│ dpkg │ CVE-2022-1664 │ fixed │ 1.20.9 │ 1.20.10 │ Dpkg::Source::Archive in dpkg, the Debian package management │
│ │ │ │ │ │ system, b ... │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-1664 │
└──────────────┴────────────────┴──────────┴─────────────────────┴───────────────────┴──────────────────────────────────────────────────────────────┘
# (snip)
Java (jar)
Total: 1 (HIGH: 1, CRITICAL: 0)
┌─────────────────────────────────────────┬───────────────┬────────┬───────────────────┬───────────────┬───────────────────────────────────────────┐
│ Library │ Vulnerability │ Status │ Installed Version │ Fixed Version │ Title │
├─────────────────────────────────────────┼───────────────┼────────┼───────────────────┼───────────────┼───────────────────────────────────────────┤
│ com.google.guava:guava (guava-15.0.jar) │ CVE-2023-2976 │ fixed │ 15.0 │ 32.0.0 │ insecure temporary directory creation │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-2976 │
└─────────────────────────────────────────┴───────────────┴────────┴───────────────────┴───────────────┴───────────────────────────────────────────┘
14

15. [Deploy] RPM packaging
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>rpm-maven-plugin</artifactId>
<configuration>
<name>pospai</name>
<mappings>
<mapping>
<directory>/usr/share/pospai/lib</directory>
<dependency/>
</mapping>
<mapping>
<directory>/usr/share/pospai/repo</directory>
<sources>
<source>
<location>../launcher/target/repository</location>
</source>
</sources>
</mapping>
</mappings>
</configuration>
</plugin> 15

16. [Deploy] Container image
FROM docker.io/apache/sling:snapshot AS starter
FROM docker.io/eclipse-temurin:17
ENV EXTRA_JAVA_OPTS="-Dorg.apache.felix.configadmin.plugin.interpolation.secretsdir=/etc/pospai/secrets"
EXPOSE 8080
RUN groupadd --system sling &&
useradd --no-log-init --system --gid sling sling &&
mkdir /opt/sling &&
mkdir /opt/sling/bin &&
mkdir /opt/sling/org.apache.sling.feature.launcher &&
mkdir /opt/sling/launcher &&
mkdir /opt/sling/artifacts &&
mkdir /opt/sling/agents &&
chown -R sling:sling /opt/sling/launcher
VOLUME /opt/sling/launcher
# continued on the next slide
16

17. [Deploy] Container image (2)
# continued from the next slide
COPY --from=starter /opt/sling/bin /opt/sling/bin
COPY target/dependency/org.apache.sling.feature.launcher /opt/sling/org.apache.sling.feature.launcher
COPY target/artifacts/ /opt/sling/artifacts/
# ensure all files are readable by the sling user
# for some reason some jar files are 0600 while most are 0644
RUN find /opt/sling/artifacts -type f -perm 0600 | xargs --no-run-if-empty chmod 0644
USER sling:sling
WORKDIR /opt/sling
ENTRYPOINT [ "/opt/sling/bin/launch.sh" ]
CMD ["oak_tar"]
17

18. [Deploy] Kubernetes
$ flux tree ks app-pospai
Kustomization/flux-system/app-pospai
├── Namespace/app-pospai
├── Secret/app-pospai/docker.io
├── Service/app-pospai/pospai
├── Deployment/app-pospai/pospai
├── PersistentVolumeClaim/app-pospai/pospai-data
└── Ingress/app-pospai/ingress
18

19. [Deploy] Kubernetes scaling
$ kubectl get pvc
NAME STATUS CAPACITY ACCESS MODES
pospai-data Bound 1Gi RWO
$ kubectl get deploy pospai -o json | jq '.spec.replicas'
1
$ kubectl get deploy pospai -o json | jq ' .spec.strategy.type'
"Recreate"
19

20. [Monitor] Logging
{
"configurations":{
"org.apache.sling.commons.log.LogManager":{
"org.apache.sling.commons.log.file":""
}
}
}
20

21. [Monitor] Logging
21

22. [Monitor] Metrics
{
"bundles": [
{
"id": "org.apache.sling/org.apache.sling.commons.metrics.prometheus/1.0",
"start-order": 20
}
]
}
$ curl --silent http://localhost:8080/metrics | grep -E '^(sling|oak|jvm)' | wc -l
486
22

23. [Monitor] Oak performance metrics
oak_QUERY_DURATION / oak_QUERY_DURATION_index...
oak_COMMIT_QUEUE_SIZE
oak_INDEX_SIZE(...)
oak_SESSION_COUNT
23

24. [Monitor] Oak availability metrics
oak_LOGIN_ERRORS_total
oak_LOCAL_INDEX_DIR_SIZE
oak_SEGMENT_REPO_SIZE
24

25. [Monitor] Sling performance metrics
sling_commons_scheduler_running_jobs
sling_org_apache_sling_resourceresolver_numberOfVanityPaths
sling_org_apache_sling_resourceresolver_numberOfAliases
25

26. [Monitor] Sling availability metrics
sling_discovery_oak_local_cluster_instances
sling_event_jobs_failed_count
sling_org_apache_sling_resourceresolver_unclosedResourceResolvers
26

27. [Monitor] Metrics
27

28. [Monitor] Metrics
28

29. [Monitor] Alerting
29

30. [Monitor] Tracing
30

31. [Maintain ] Renovate PR - message
31

32. [Maintain ] Renovate PR - message
32

33. [Maintain ] Renovate Config - apps
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [ "config:base", "regexManagers:mavenPropertyVersions" ],
"packageRules": [
{
"matchPackagePrefixes": [ "org.apache.jackrabbit:" ],
"groupName": "Apache Jackrabbit and Jackrabbit Oak",
"allowedVersions": "/^[0-9]+.[0-9]*[02468]+.[0-9]+$/"
},
{
"matchPackagePatterns": [ "guava" ],
"enabled": false
}
]
}
33

34. [Maintain ] Renovate Config - bundles
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [ "config:base" ],
"packageRules": [
{
"matchManagers": [
"maven"
],
"matchDepTypes": [
"provided"
],
"enabled": false
}
]
}
34

35. Demo
Demo
Demo
Demo
Demo
35

36. Resources
https://github.com/apache/sling-org-apache-sling-commons-metrics-prometheus/
https://github.com/renovatebot/renovate/
https://github.com/apache/sling-project-archetype/pull/12
https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/9469
https://github.com/rombert/pospai
36