The document discusses various patterns for scaling a CQ5 implementation. It describes scaling patterns for high volume delivery, high frequency input feeds, many editors, high processing input feeds, high volume input feeds, geo-distributed editors, many DAM assets, and geo-distributed disaster recovery. For each pattern, it provides descriptions of use cases and limiting factors, and proposes solutions such as sharding, separating processing instances, batching operations, and leveraging caching and compression.

1. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Michael Marth | Senior Engineering Manager | @michaelmarth
Scaling CQ5

2. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 2
Michael
§ Engineering manager in R&D
§ Responsible for Granite (repo, CQSE, etc)
§ Many years in the field
§ Wears shorts today

3. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Escape the soup coma, let’s make this a workshop session
3
Soup Coma

4. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Scalability?
§ Performance vs Scalability
§ Performance: “it takes X secs to do Y”
§ Scalability: “it takes X secs to do Y simultaneously Z times”
§ Yet, performance can help with scalability
§ This talk
§ is about horizontal scalability (vertical scaling is trivial)
§ is about pre-Oak scalability patterns
§ The content will soon be documented in docs.day.com
4

5. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Scaling problems are good problems to have (you are in demand!)
5
Text
opportunities

6. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
CQ5 Scaling Patterns
1. High Volume and High Performance Delivery
2. High Frequency Input Feed
3. Many Editors
4. High Processing Input Feed
5. High Volume Input Feed
6.Geo-distributed Editors
7. Many DAM assets
8.Geo-distributed disaster recovery
6

7. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Volume and High Performance Delivery - Description
§ Use Case:
§ High traffic site (100m impressions/d)
§ Examples: adobe.com
§ Limiting factor
§ CPU on publish
7

8. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Volume and High Performance Delivery - Solution Pattern
8
§ Leverage dispatcher caching as much as possible
§ in latest dispatcher: single-page dispatcher flush and scripted flushing, use to cache/flush
content in dispatcher
§ SSI and/or client-side for personalized content
§ Selectors for query caching
§ CDN with short TTL
P
P Disp
Disp
Load
Balancer
round robin
CDN Client

9. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Volume and High Performance Delivery
§ Notes
§ Related to rendering performance, see also
§ CQ performance patterns (use CQ timing component, prefer tree walking over JCR queries,
use ClientLibraryManager to concat and minify JS, etc, see [1])
§ Generic performance patterns (reduce requests with e.g. css sprites, gzip responses, put JS
calls at bottom of HTML, etc, see [2])
§ Anti-Pattern
§ Adding publishers before leveraging caching
[1] http://dev.day.com/docs/en/cq/current/deploying/performance.html
[2] http://shop.oreilly.com/product/9780596529307.do
9

10. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Frequency Input Feed - Description
§ Use Case: news feed import (moderate amounts, but constant updates)
§ Limiting factor
§ Dispatcher cache invalidation
§ Therefore actual limiting factor is CPU on publish
10

11. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Frequency Input Feed - Solution Pattern 1
11
§ Set up content structure so that other pages do not get invalidated on dispatcher
cache
§ if possible: highly volatile content e.g. in /etc
§ with latest dispatcher: single-page flush possible
§ Separate replication queue (so that main queue is not blocked)
A
editing
import
process
Backend
P
replication
Disp
flush

12. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Frequency Input Feed - Solution Pattern 2
§ Set up content structure so that other pages do not get invalidated on dispatcher
cache
§ as previous pattern
§ Import directly into Publish (no replication necessary)
A
import
process
Backend
P Disp
flush
import
process
Backend

13. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Frequency Input Feed
§ Questions to ask
§ Human filtering/processing needed? Then imports should be on author and
replicated.
§ If no: is the use case OK with different states on publish?
§ if yes: no replication needed, then pattern 2 is preferable
13

14. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Many Editors - Description
§ Use Case:
§ News or media portal
§ >50 editors editing content concurrently
§ Limiting factor
§ Depends on what do the editors actually do:
§ Heavyweight editing, e.g. MSM rollouts, starting WFs: repository- or CPU-bound
§ Lightweight editing: CPU bound
14

15. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Many Editors - Solution Pattern 1
15
§ Sharding: split up different web sites / parts of web sites onto separate author
instances
§ Publish instances are shared
A
site 3
editing
P
A
site 2
editing
P
A
site 1
editing
P
replication

16. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Many Editors - Solution Pattern 2
16
§ Sharding: split up different web sites into separate author instances, but replicate into
one main author, e.g. for shared workflow processes
§ Practical if the shards do not need to share content.
§ Cross-replication can be done, but will be hard to keep consistent
§ Publish instances are shared
A
site 3
editing
A
site 2
editing
P
A
site 1
editing
replication
A

17. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Many Editors
§ Questions to ask
§ What do the editors actually do?
§ Notes
§ Author dispatcher helps to reduce CPU load on author instances
§ Author cluster instead of sharding will mitigate the problem if CPU-bound
17

18. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Processing Input Feed - Description
18
§ Use Case:
§ DAM import of images
§ 1000 images at once
§ happens regularly
§ other editors are editing content at the same time
§ Limiting factor
§ CPU, memory on author

19. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Processing Input Feed - Solution Pattern 1
19
§ Separate processing instances from human editing instances
§ Offload 1 Workflow step, e.g. thumbnail generation from PSDs
§ There can be more than 1 processing instance
§ Replicate back and forth in packages if possible
§ CQ5.6.1: share DS between instances and replicate without binary, offloading framework
A
Processing
Author
Processing
Author
editing
workflows
running
offload CPU-intensive
WF steps
by replication
workflow steps
running, replicating results back

20. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Processing Input Feed - Solution Pattern 2
20
§ Separate pre-processing instances for uploading
§ There can be more than 1 pre-processing instance
§ CQ5.6.1: share DS between instances and replicate without binary
A
Processing
Author
editing
workflows
running
replication
image
upload

21. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Processing Input Feed
21
§ Notes
§ Author cluster can help mitigate the problem, but editors must edit content on slave
§ Throttling WFs or execution during night can help mitigate the problem
§ If the import is limited by CPU needed image conversion consider using ImageMagick
rather than Java

22. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Volume Input Feed - Description
22
§ Use Case:
§ Product data import
§ 1 million products, 10000 modifications/day
§ Limiting factor
§ Writing to the repository
§ reads are also blocked
§ Potentially (to a lesser degree) in case repository scans are needed to create diffs:
§ CPU for calculating diffs
§ Repository read caches get flushed

23. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Volume Input Feed - Solution Pattern
23
§ Separate import instance to process imports, partition if possible
§ only useful if import requires significant CPU (e.g. no diff delivered)
§ Replicate to author
§ Replicate as package
§ CQ5.6.1: share DS between instances and replicate without binary
§ Replication to publish as package if possible
A
Import
Processing
Author
editing
import into
repository
replication of diff
import
processBackend
P
replication
of diff

24. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
High Volume Input Feed
24
§ Questions to ask
§ Can the import be throttled? Most problems get much less severe.
§ Do all changes get on publish?
§ Notes
§ Use batch saves (1000 nodes) on import (reduces overhead in indexing, etc and speeds up
the import overall)
§ Import as nt:unstructured rather cq:Page if possible
§ If not: switch off heavy listeners (e.g. ContentSync) or use the JcrObservationThrottle
§ Anti-Pattern
§ Usage of network disc (usually have high latency)
§ Replicating to publish through same replication queue as editorial content
§ Might heavily delay important editorial changes from publication
§ Using PageManager (cannot operate in batch mode, unless you use 5.6.1 where that option
was added)

25. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Geo-distributed Editors
25
§ Use Case:
§ Editors located in different geos (US, EMEA, APAC)
§ Limiting factor
§ Bandwidth between editor location and author server location

26. § Use Dispatcher in front of Author
§ Guiding principle: limit traffic between Dispatcher and editor location.
§ gzip traffic
§ Use Client Library Manager to minimize traffic
§ minify, concat and gzip all client libraries
§ Cache all responses that are not under /content in
§ Editor’s browser cache
§ Potentially also dispatcher cache
© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Geo-distributed Editors - Solution Pattern
26
ADisp
cache
cache
minimize requests
gzip responses

27. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Geo-distributed Editors
§ Notes
§ In extreme cases consider writing templates that treat author renditions differently from
publish renditions (especially reducing the number of necessary requests, e.g. by dropping
requests to tracking servers, external CSS, etc)
§ Or use Scaffolding for editing
27

28. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Many DAM Assets
§ Use Case:
§ Many assets (>5Mio) in DAM
§ Limiting factor
§ Disc space
28

29. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Many DAM Assets - Solution Pattern
29
§ Split physical storage of data store and repository tar files
§ tar files need disc with very low latency
§ for data store high latency is acceptable
§ Locate data store on cheap discs remotely (NAS, S3)
§ Share data store between instances
§ In 5.6.1: use binary-less replication in case of shared DS to minimize network traffic
Repository
tar DS
local S3

30. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Many DAM Assets
§ Notes
§ In case of shared DS: the DS garbage collection needs to be run on an instance that keeps
references to all assets in DS
§ In 5.6.1: huge performance improvements (~10x or more) for DS GC when the persistence is
tar-based
30

31. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Geo-distributed disaster recovery
31
§ Use Case:
§ Data centers located in different geos
§ One DC shall act as failover for author
§ Limiting factor
§ Latency between DCs (in very low latency cases CRX clustering could be used)

32. § Use file level tools like rysnc to create replicas in 2nd DC
§ Hourly: sync data store
§ This is usually the most time consuming part
§ Sync can be performed anytime, due to add-only data store architecture
§ Nightly:
§ Create incremental backup into filesystem on 1st DC to get consistent state of files
§ Rsync backup to 2nd DC. For that period CQ on 2nd DC must not be running.
© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Geo-distributed disaster recovery - Solution Pattern
32

33. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.