Have you ever felt that AEM is playing tricks on you? I bet you have. This is one of those stories - the more you dig in, the less you understand. That was a classic headless setup - AEM’s role was to expose content via REST-like JSON API so various applications can consume the data. Things were running smoothly, but one day a misrendered JSON shows up - a set of mandatory properties just disappeared from the object. The case got thoroughly investigated, however no one could even reproduce it. Nothing has changed at JCR level, there was no deployment in the meantime and when you visit the exact same URL all the data are correct. “Oh, that must have been a one-off incident” someone said. The ticket gets closed and life goes on. A week after similar issue got reported - a different JSON object is broken this time, but at least you can reproduce it. Unfortunately, an hour later the problem magically goes away. Time passes by and a slightly different variant of the problem surfaces in production - you keep requesting affected URL and the response alternates between completely valid JSON and its broken form. Your team hops on a call to get to the bottom of the problem, but in a matter of minutes it just vanishes without a trace again. Interested in what happened and where we ended up? That’s what the talk’s going to be about.

1. EUROPE'S LEADING AEM DEVELOPER CONFERENCE
25th – 27th SEPTEMBER 2023
The curious case of misrendered JSON
Jakub Wądołowski, diva-e (@jwadolowski)

2. Let’s talk failure
https://flic.kr/p/88bWKw

3. Project context
▪ e-commerce platform
▪ AEMaaCS as headless content API (JSON)*
▪ Sling Model Exporter
▪ Hyperlinked API (driven by content structure)
▪ Custom SPA Editor (WYSIWYG)
https://flic.kr/p/xJi1En

4. How it’s made?
https://flic.kr/p/r7Syak

10. JSON structure
https://flic.kr/p/ANM3jp

16. June 2021
“Approx. 29% of AEM requests end
with misrendered JSON and 200 status
code”
https://flic.kr/p/9bUbH3

18. Missing JSON data reasons
▪ Content issues
▪ Deployment / caching issue
▪ AEM downtime / maintenance
https://flic.kr/p/bJoDYk

19. Troubleshooting phase
▪ Not reproducible on local AEM
▪ Debug loggers / headers
▪ Page (re)activation solves the problem
▪ Cache bypassing tricks
▪ Ongoing monitoring
https://flic.kr/p/c5xUxS

20. “If you torture the data long enough, it
will confess to anything”
Ronald H. Coase

21. Analysis results
▪ Incomplete AEMaaCS logs (Loki 2.3 issue)
▪ No relevant log entries
▪ New endpoints with invalid JSON
▪ Bypassed cache == issue is gone*
▪ Warmup service goes first*
https://flic.kr/p/dgfRgD

22. AEMaaCS startup and warmup service
▪ AEMaaCS /systemready probe
▪ Warmup service
▪ goal: pre-populate dispatcher cache
▪ requests the most popular URLs (last 24h)
▪ Host header is taken into account
https://flic.kr/p/xGJnBw

23. Unreliable API == angry developers
https://flic.kr/p/czHNpq

24. Disable caching!
https://flic.kr/p/oaMedF

25. New hope
▪ SKYOPS-16686 (~mid July 2021)
▪ dead end
▪ SKYOPS-17857 (~mid Aug 2021)
▪ delivered early Sep 2021
https://flic.kr/p/8JhwbU

26. Over the finish line
▪ AEMaaCS 2021.8.5755 includes the fix
▪ Dispatcher caching got re-enabled
▪ No JSON issues for 2 weeks - let’s celebrate!
https://flic.kr/p/5bdFXt

27. Whack-a-mole time begins!
https://flic.kr/p/7r9TAS

30. Asset data vanished
▪ “skyline-service-warmup” goes first again
▪ Affected 1 out of 2 publish instances
▪ Duration: ~6 hours
▪ (Re)activation fixes everything
https://flic.kr/p/aH4q2

31. Race conditions
▪ Publish isn’t fully ready when warmup starts
▪ Dynamic Media fault?
▪ Does “Accept-Encoding” matter?
▪ How /systemready works?
https://flic.kr/p/xL4C1D

32. Why?
https://flic.kr/p/8vT9yB

34. The naive loop
• (Re)start local AEM instance
• AEM state snapshots
• /systemready
• OSGi bundles/components/services
• misbehaving JSON
https://flic.kr/p/deCibc

35. AEM startup stages
▪ #1
▪ /systemready 503
▪ JSON 404
▪ #2
▪ /systemready 503
▪ JSON 200 + invalid data
▪ #3
▪ /systemready 503
▪ JSON 200 + correct data
▪ #4
▪ /systemready 200
▪ JSON 200 + correct data
https://flic.kr/p/cmoAhY

37. Healthchecks
▪ Let’s create a custom one!
▪ Extend the built-in healthcheck config
https://flic.kr/p/2me55Vo

38. org.apache.felix.hc.generalchecks.DsComponentsCheck~myproject-systemready.cfg.json

39. Warning!
▪ Nonexistent component on the list
▪ Mind refactored / renamed components
https://flic.kr/p/5nmVJy

40. December 2021 / January 2022
▪ Component list fine-tuning
▪ It’s finally over!
https://flic.kr/p/qPu4DZ

41. May/June 2022
▪ Guess who’s back?
https://flic.kr/p/9eMnKj

43. Development gap
▪ New code == new healthcheck entries
https://flic.kr/p/abg7JF

44. September 2022
▪ Unexpected JSON cutoff
https://flic.kr/p/2kZcf4k

46. Next steps
▪ SLING-11569 (released in Sling 2.11.0)
▪ AEMaaCS 2022.12.10488 (Sling 2.12.2)
https://flic.kr/p/LA2Ta

47. Remember asset issues?
▪ Invalid asset links (DAM instead of DM)
https://flic.kr/p/2jxjahF

50. Asset link hurdles
▪ DM is not enabled everywhere
▪ Enabled DM implies blocked DAM access (404)
▪ /conf/global/settings/dms7enabled - bad idea!
▪ Per asset DM detection (dam:scene7File)
▪ Dedicated option for DAM fallback
https://flic.kr/p/CyXpjV

51. The end!?
▪ Kind of…
https://flic.kr/p/CyXpjV

52. June 2023
▪ JSON cutoff strikes back
▪ This time in Single Page Editor (AEM Author)
▪ Beware of AEMaaCS auto-upgrades
https://flic.kr/p/dypRQZ

53. Thank you!