SlideShare a Scribd company logo
Scaling AEM
Michael Marth | Senior Engineering Manager

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Scalability?
§

Performance vs Scalability
§
§

§
§
§

Performance: “it takes X secs to do Y”
Scalability: “it takes X secs to do Y simultaneously Z times”

Yet, performance can help with scalability
This talk is about horizontal scalability (vertical scaling is trivial)
The content is documented at
http://dev.day.com/docs/en/cq/current/deploying/scaling.html

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

2
Preliminary Notes
§

This talk is about pre-Oak scalability patterns, i.e. the CQ5.x line
§

§

with AEM 6 these patterns will change

CRX2 Cluster is mostly a technology for reliability, not for scalability
§

Slight exception is read scalability

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

3
Scaling AEM
1. High Volume and High Performance Delivery
2. High Frequency Input Feed
3. High Processing Input Feed
4. High Volume Input Feed
5. Many Editors
6. Geo-distributed Editors
7. Many DAM assets
8. Geo-distributed disaster recovery

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

4
High Volume and High Performance Delivery - Description
§

Use Case:
§

High traffic site (100m impressions/d)

§

Examples: adobe.com

§

Limiting factor
§

CPU on publish

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

5
High Volume and High Performance Delivery - Solution Pattern
§

Leverage dispatcher caching as much as possible
§

§

SSI and/or client-side for personalized content

§

§

in latest dispatcher: single-page dispatcher flush and scripted flushing, use to cache/flush
content in dispatcher

Selectors for query caching

CDN with short TTL

P

Disp

round robin

Load
Balancer

P

Disp

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

6

CDN

Client
High Volume and High Performance Delivery
§

Notes
§

Related to rendering performance, see also
§

§

§

CQ performance patterns (use CQ timing component, prefer tree walking over JCR queries,
use ClientLibraryManager to concat and minify JS, etc, see [1])
Generic performance patterns (reduce requests with e.g. css sprites, gzip responses, put JS
calls at bottom of HTML, etc, see [2])

Anti-Pattern
§

Adding publishers before leveraging caching

[1] http://dev.day.com/docs/en/cq/current/deploying/performance.html
[2] http://shop.oreilly.com/product/9780596529307.do
© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

7
High Frequency Input Feed - Description
§

Use Case: news feed import (moderate amounts, but constant updates)

§

Limiting factor
§

Dispatcher cache invalidation
§

Therefore actual limiting factor is CPU on publish

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

8
High Frequency Input Feed - Solution Pattern 1
§

Set up content structure so that other pages do not get invalidated on dispatcher
cache
§
§

§

if possible: highly volatile content e.g. in /etc
with latest dispatcher: single-page flush possible

Separate replication queue (so that main queue is not blocked)

Backend

import
process

editing

A

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

replication

P

9

flush

Disp
High Frequency Input Feed - Solution Pattern 2
§

Set up content structure so that other pages do not get invalidated on dispatcher
cache
§

§

as previous pattern

Import directly into Publish (no replication necessary)

Backend

Backend

import
process

A

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

import
process

P

flush

Disp
High Frequency Input Feed
§

Questions to ask
§

Human filtering/processing needed? Then imports should be on author and
replicated.
§

If no: is the use case OK with different states on publish?
§

if yes: no replication needed, then pattern 2 is preferable

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

11
High Processing Input Feed - Description
§

Use Case:
§

DAM import of images
§
§

happens regularly

§

§

1000 images at once

other editors are editing content at the same time

Limiting factor
§

CPU, memory on author

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

12
High Processing Input Feed - Solution Pattern 1
§

Separate processing instances from human editing instances
§

Offload 1 Workflow step, e.g. thumbnail generation from PSDs

§

There can be more than 1 processing instance

§

Replicate back and forth in packages if possible
§

CQ5.6.1: share DS between instances and replicate without binary, offloading framework

Processing
Author

Processing
Author

workflow steps
running, replicating results back

offload CPU-intensive
WF steps
by replication
editing

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

A

workflows
running

13
High Processing Input Feed - Solution Pattern 2
§

Separate pre-processing instances for uploading
§

There can be more than 1 pre-processing instance
§

CQ5.6.1: share DS between instances and replicate without binary

workflows
running
image
upload

Processing
Author

replication

editing

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

A

14
High Processing Input Feed
§

Notes
§

Author cluster can help mitigate the problem, but editors must edit content on slave

§

Throttling WFs or execution during night can help mitigate the problem

§

If the import is limited by CPU needed image conversion consider using ImageMagick
rather than Java

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

15
High Volume Input Feed - Description
§

Use Case:
§

Product data import
§

§

1 million products, 10000 modifications/day

Limiting factor
§

Writing to the repository
§

§

reads are also blocked

Potentially (to a lesser degree) in case repository scans are needed to create diffs:
§

CPU for calculating diffs

§

Repository read caches get flushed

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

16
High Volume Input Feed - Solution Pattern
§

Separate import instance to process imports, partition if possible
§

§

only useful if import requires significant CPU (e.g. no diff delivered)

Replicate to author
§

Replicate as package
§

§

CQ5.6.1: share DS between instances and replicate without binary

Replication to publish as package if possible
import into
repository
Backend

import
process

Import
Processing
Author

replication of diff

editing

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

A

17

replication
of diff

P
High Volume Input Feed
§

Questions to ask
§

§

Can the import be throttled? Most problems get much less severe.

Notes
§

Use batch saves (1000 nodes) on import (reduces overhead in indexing, etc and
speeds up the import overall)

§

Import as nt:unstructured rather cq:Page if possible
§

§

If not: switch off heavy listeners (e.g. ContentSync) or use the JcrObservationThrottle

Anti-Pattern
§

Usage of network disc (usually have high latency)

§

Replicating to publish through same replication queue as editorial content
§

§

Might heavily delay important editorial changes from publication

Using PageManager (cannot operate in batch mode, unless you use 5.6.1 where that
option was added)

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

18
Many Editors - Description
§

Use Case:
§
§

§

News or media portal
>50 editors editing content concurrently

Limiting factor
§

Depends on what do the editors actually do, eg. MSM rollouts, starting WFs, but
usually CPU-bound

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

19
Many Editors - Solution Pattern 1
§

Sharding: split up different web sites / parts of web sites onto separate author
instances

§

Publish instances are shared

editing

A

replication

site 1

editing

A

P

site 2

editing

A

P

site 3

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

P

20
Many Editors - Solution Pattern 2
§

Sharding: split up different web sites into separate author instances, but replicate into
one main author, e.g. for shared workflow processes
§
§

§

Practical if the shards do not need to share content.
Cross-replication can be done, but will be hard to keep consistent

Publish instances are shared
A

editing

site 1

A

editing

replication

site 2

A

editing

site 3

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

21

A

P
Many Editors
§

Questions to ask
§

§

What do the editors actually do?

Notes
§

Author dispatcher helps to reduce CPU load on author instances

§

Author CRX cluster instead of sharding will mitigate the problem if CPU-bound

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

22
Geo-distributed Editors
§

Use Case:
§

§

Editors located in different geos (US, EMEA, APAC)

Limiting factor
§

Bandwidth between editor location and author server location

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

23
Geo-distributed Editors - Solution Pattern
§

Use Dispatcher in front of Author

§

Guiding principle: limit traffic between Dispatcher and editor location.
§

gzip traffic

§

Use Client Library Manager to minimize traffic
§

§

minify, concat and gzip all client libraries

Cache all responses that are not under /content in
§

Editor’s browser cache

§

Potentially also dispatcher cache

cache

cache
minimize requests
gzip responses

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Disp

24

A
Geo-distributed Editors
§

Notes
§

In extreme cases consider writing templates that treat author renditions differently from
publish renditions (especially reducing the number of necessary requests, e.g. by dropping
requests to tracking servers, external CSS, etc)

§

Or use Scaffolding for editing

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

25
Many DAM Assets
§

Use Case:
§

§

Many assets (>5Mio) in DAM

Limiting factor
§

Disc space

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

26
Many DAM Assets - Solution Pattern
§

Split physical storage of data store and repository tar files
§
§

for data store high latency is acceptable

§

§

tar files need disc with very low latency

Locate data store on cheap discs remotely (NAS, S3)

Share data store between instances
§

In 5.6.1: use binary-less replication in case of shared DS to minimize network traffic

Repository

tar

local
© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

DS

S3
27
Many DAM Assets
§

Notes
§

In case of shared DS: the DS garbage collection needs to be run on an instance that keeps
references to all assets in DS

§

In 5.6.1: huge performance improvements (~10x or more) for DS GC when the persistence is
tar-based

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

28
Geo-distributed disaster recovery
§

Use Case:
§
§

§

Data centers located in different geos
One DC shall act as failover for author

Limiting factor
§

Latency between DCs (in very low latency cases CRX clustering could be used)

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

29
Geo-distributed disaster recovery - Solution Pattern
§

Use file level tools like rysnc to create replicas in 2nd DC

§

Hourly: sync data store
§
§

§

This is usually the most time consuming part
Sync can be performed anytime, due to add-only data store architecture

Nightly:
§

Create incremental backup into filesystem on 1st DC to get consistent state of files

§

Rsync backup to 2nd DC. For that period CQ on 2nd DC must not be running.

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

30
© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

More Related Content

What's hot

Effectively Deploying MongoDB on AEM
Effectively Deploying MongoDB on AEMEffectively Deploying MongoDB on AEM
Effectively Deploying MongoDB on AEM
Norberto Leite
 
CIRCUIT 2015 - Monitoring AEM
CIRCUIT 2015 - Monitoring AEMCIRCUIT 2015 - Monitoring AEM
CIRCUIT 2015 - Monitoring AEM
ICF CIRCUIT
 
Webinar: Adobe Experience Manager Clustering Made Easy on MongoDB
Webinar: Adobe Experience Manager Clustering Made Easy on MongoDB Webinar: Adobe Experience Manager Clustering Made Easy on MongoDB
Webinar: Adobe Experience Manager Clustering Made Easy on MongoDB
MongoDB
 
Adobe Managed Services: Complicated Cloud Deployments
Adobe Managed Services: Complicated Cloud DeploymentsAdobe Managed Services: Complicated Cloud Deployments
Adobe Managed Services: Complicated Cloud Deployments
Adam Pazik
 
AEM + MongoDB: How to Scale and Operate Large Digital Asset Management Systems
AEM + MongoDB: How to Scale and Operate Large Digital Asset Management SystemsAEM + MongoDB: How to Scale and Operate Large Digital Asset Management Systems
AEM + MongoDB: How to Scale and Operate Large Digital Asset Management Systems
MongoDB
 
Accelerate your ColdFusion Applications using Caching
Accelerate your ColdFusion Applications using CachingAccelerate your ColdFusion Applications using Caching
Accelerate your ColdFusion Applications using Caching
ColdFusionConference
 
Aem maintenance
Aem maintenanceAem maintenance
Aem maintenance
Ashokkumar T A
 
Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWS Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWS
Tom Laszewski
 
Aem offline content
Aem offline contentAem offline content
Aem offline content
Ashokkumar T A
 
EVOLVE'13 | Enhance | Permission Sensitive Caching | Paul McMahon & Jason Rap...
EVOLVE'13 | Enhance | Permission Sensitive Caching | Paul McMahon & Jason Rap...EVOLVE'13 | Enhance | Permission Sensitive Caching | Paul McMahon & Jason Rap...
EVOLVE'13 | Enhance | Permission Sensitive Caching | Paul McMahon & Jason Rap...
Evolve The Adobe Digital Marketing Community
 
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
Scale ColdFusion with Terracotta Distributed Caching for EhchacheScale ColdFusion with Terracotta Distributed Caching for Ehchache
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
ColdFusionConference
 
AEM - A Collection of developer friendly tools
AEM - A Collection of developer friendly toolsAEM - A Collection of developer friendly tools
AEM - A Collection of developer friendly tools
Ashokkumar T A
 
Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures
Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructuresMaster Chef class: learn how to quickly cook delightful CQ/AEM infrastructures
Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures
François Le Droff
 
CIRCUIT 2015 - AEM Infrastructure Automation with Chef Cookbooks
CIRCUIT 2015 - AEM Infrastructure Automation with Chef CookbooksCIRCUIT 2015 - AEM Infrastructure Automation with Chef Cookbooks
CIRCUIT 2015 - AEM Infrastructure Automation with Chef Cookbooks
ICF CIRCUIT
 
Running High Availability Websites with Acquia and AWS
Running High Availability Websites with Acquia and AWSRunning High Availability Websites with Acquia and AWS
Running High Availability Websites with Acquia and AWS
Acquia
 
Scaling CQ5
Scaling CQ5Scaling CQ5
Scaling CQ5
connectwebex
 
Scaling Drupal & Deployment in AWS
Scaling Drupal & Deployment in AWSScaling Drupal & Deployment in AWS
Scaling Drupal & Deployment in AWS
永对 陈
 
Apache Performance Tuning: Scaling Out
Apache Performance Tuning: Scaling OutApache Performance Tuning: Scaling Out
Apache Performance Tuning: Scaling Out
Sander Temme
 
Oracle on AWS partner webinar series
Oracle on AWS partner webinar series Oracle on AWS partner webinar series
Oracle on AWS partner webinar series
Tom Laszewski
 
AWS Summit Sydney 2014 | AWSome Data Protection with Veeam - Session Sponsore...
AWS Summit Sydney 2014 | AWSome Data Protection with Veeam - Session Sponsore...AWS Summit Sydney 2014 | AWSome Data Protection with Veeam - Session Sponsore...
AWS Summit Sydney 2014 | AWSome Data Protection with Veeam - Session Sponsore...
Amazon Web Services
 

What's hot (20)

Effectively Deploying MongoDB on AEM
Effectively Deploying MongoDB on AEMEffectively Deploying MongoDB on AEM
Effectively Deploying MongoDB on AEM
 
CIRCUIT 2015 - Monitoring AEM
CIRCUIT 2015 - Monitoring AEMCIRCUIT 2015 - Monitoring AEM
CIRCUIT 2015 - Monitoring AEM
 
Webinar: Adobe Experience Manager Clustering Made Easy on MongoDB
Webinar: Adobe Experience Manager Clustering Made Easy on MongoDB Webinar: Adobe Experience Manager Clustering Made Easy on MongoDB
Webinar: Adobe Experience Manager Clustering Made Easy on MongoDB
 
Adobe Managed Services: Complicated Cloud Deployments
Adobe Managed Services: Complicated Cloud DeploymentsAdobe Managed Services: Complicated Cloud Deployments
Adobe Managed Services: Complicated Cloud Deployments
 
AEM + MongoDB: How to Scale and Operate Large Digital Asset Management Systems
AEM + MongoDB: How to Scale and Operate Large Digital Asset Management SystemsAEM + MongoDB: How to Scale and Operate Large Digital Asset Management Systems
AEM + MongoDB: How to Scale and Operate Large Digital Asset Management Systems
 
Accelerate your ColdFusion Applications using Caching
Accelerate your ColdFusion Applications using CachingAccelerate your ColdFusion Applications using Caching
Accelerate your ColdFusion Applications using Caching
 
Aem maintenance
Aem maintenanceAem maintenance
Aem maintenance
 
Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWS Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWS
 
Aem offline content
Aem offline contentAem offline content
Aem offline content
 
EVOLVE'13 | Enhance | Permission Sensitive Caching | Paul McMahon & Jason Rap...
EVOLVE'13 | Enhance | Permission Sensitive Caching | Paul McMahon & Jason Rap...EVOLVE'13 | Enhance | Permission Sensitive Caching | Paul McMahon & Jason Rap...
EVOLVE'13 | Enhance | Permission Sensitive Caching | Paul McMahon & Jason Rap...
 
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
Scale ColdFusion with Terracotta Distributed Caching for EhchacheScale ColdFusion with Terracotta Distributed Caching for Ehchache
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
 
AEM - A Collection of developer friendly tools
AEM - A Collection of developer friendly toolsAEM - A Collection of developer friendly tools
AEM - A Collection of developer friendly tools
 
Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures
Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructuresMaster Chef class: learn how to quickly cook delightful CQ/AEM infrastructures
Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures
 
CIRCUIT 2015 - AEM Infrastructure Automation with Chef Cookbooks
CIRCUIT 2015 - AEM Infrastructure Automation with Chef CookbooksCIRCUIT 2015 - AEM Infrastructure Automation with Chef Cookbooks
CIRCUIT 2015 - AEM Infrastructure Automation with Chef Cookbooks
 
Running High Availability Websites with Acquia and AWS
Running High Availability Websites with Acquia and AWSRunning High Availability Websites with Acquia and AWS
Running High Availability Websites with Acquia and AWS
 
Scaling CQ5
Scaling CQ5Scaling CQ5
Scaling CQ5
 
Scaling Drupal & Deployment in AWS
Scaling Drupal & Deployment in AWSScaling Drupal & Deployment in AWS
Scaling Drupal & Deployment in AWS
 
Apache Performance Tuning: Scaling Out
Apache Performance Tuning: Scaling OutApache Performance Tuning: Scaling Out
Apache Performance Tuning: Scaling Out
 
Oracle on AWS partner webinar series
Oracle on AWS partner webinar series Oracle on AWS partner webinar series
Oracle on AWS partner webinar series
 
AWS Summit Sydney 2014 | AWSome Data Protection with Veeam - Session Sponsore...
AWS Summit Sydney 2014 | AWSome Data Protection with Veeam - Session Sponsore...AWS Summit Sydney 2014 | AWSome Data Protection with Veeam - Session Sponsore...
AWS Summit Sydney 2014 | AWSome Data Protection with Veeam - Session Sponsore...
 

Viewers also liked

12 hot features to engage and save time with aem 6.2
12 hot features to engage and save time with aem 6.212 hot features to engage and save time with aem 6.2
12 hot features to engage and save time with aem 6.2
Tricode (part of Dept)
 
Masculinoadulto
MasculinoadultoMasculinoadulto
Masculinoadulto
copa18dofortedefutsal
 
Aula trans.calor e massa
Aula trans.calor e massaAula trans.calor e massa
Aula trans.calor e massa
JOÃO DIAS
 
Do you know Taiwan questions
Do you know Taiwan questions Do you know Taiwan questions
Do you know Taiwan questions
Joanne Chen
 
Eurozone Einzelhandel
Eurozone EinzelhandelEurozone Einzelhandel
Eurozone EinzelhandelGapa News
 
Alma el
Alma elAlma el
Primera práctica
Primera prácticaPrimera práctica
Primera práctica
rosavizarreta
 
Content strategy for the content experience wave
Content strategy for the content experience waveContent strategy for the content experience wave
Content strategy for the content experience wave
Zoran Nikolovski
 
FELIZ NATAL 2011
FELIZ NATAL 2011FELIZ NATAL 2011
FELIZ NATAL 2011
becresforte
 
Mision
MisionMision
Mision
andreitapao
 
BM#3
BM#3BM#3
Verbete - PEPS
Verbete - PEPSVerbete - PEPS
PACE-IT: CE 1.5 - LinkedIn
PACE-IT: CE 1.5 - LinkedInPACE-IT: CE 1.5 - LinkedIn
PACE-IT: CE 1.5 - LinkedIn
PACE - IT at Edmonds Community College
 
Manual de-hidrc3a1ulica-bc3a1sica-ifba
Manual de-hidrc3a1ulica-bc3a1sica-ifbaManual de-hidrc3a1ulica-bc3a1sica-ifba
Manual de-hidrc3a1ulica-bc3a1sica-ifba
Juliana Maia Rosa Ferreira
 
Google Analytics - Certification
Google Analytics - CertificationGoogle Analytics - Certification
Google Analytics - Certification
Ranga Vinod kumar
 
Termodinâmica
TermodinâmicaTermodinâmica
Termodinâmica
Ricardo Akerman
 
Treasure Hunt Activity (方位句型)
Treasure Hunt Activity (方位句型)Treasure Hunt Activity (方位句型)
Treasure Hunt Activity (方位句型)
Ping Wu
 
Culture study unit 11 worksheet
Culture study unit 11 worksheetCulture study unit 11 worksheet
Culture study unit 11 worksheet
Joanne Chen
 

Viewers also liked (19)

12 hot features to engage and save time with aem 6.2
12 hot features to engage and save time with aem 6.212 hot features to engage and save time with aem 6.2
12 hot features to engage and save time with aem 6.2
 
Masculinoadulto
MasculinoadultoMasculinoadulto
Masculinoadulto
 
Aula trans.calor e massa
Aula trans.calor e massaAula trans.calor e massa
Aula trans.calor e massa
 
Do you know Taiwan questions
Do you know Taiwan questions Do you know Taiwan questions
Do you know Taiwan questions
 
Eurozone Einzelhandel
Eurozone EinzelhandelEurozone Einzelhandel
Eurozone Einzelhandel
 
Alma el
Alma elAlma el
Alma el
 
Primera práctica
Primera prácticaPrimera práctica
Primera práctica
 
Content strategy for the content experience wave
Content strategy for the content experience waveContent strategy for the content experience wave
Content strategy for the content experience wave
 
FELIZ NATAL 2011
FELIZ NATAL 2011FELIZ NATAL 2011
FELIZ NATAL 2011
 
Mision
MisionMision
Mision
 
BM#3
BM#3BM#3
BM#3
 
Verbete - PEPS
Verbete - PEPSVerbete - PEPS
Verbete - PEPS
 
POLYMERS.2
POLYMERS.2POLYMERS.2
POLYMERS.2
 
PACE-IT: CE 1.5 - LinkedIn
PACE-IT: CE 1.5 - LinkedInPACE-IT: CE 1.5 - LinkedIn
PACE-IT: CE 1.5 - LinkedIn
 
Manual de-hidrc3a1ulica-bc3a1sica-ifba
Manual de-hidrc3a1ulica-bc3a1sica-ifbaManual de-hidrc3a1ulica-bc3a1sica-ifba
Manual de-hidrc3a1ulica-bc3a1sica-ifba
 
Google Analytics - Certification
Google Analytics - CertificationGoogle Analytics - Certification
Google Analytics - Certification
 
Termodinâmica
TermodinâmicaTermodinâmica
Termodinâmica
 
Treasure Hunt Activity (方位句型)
Treasure Hunt Activity (方位句型)Treasure Hunt Activity (方位句型)
Treasure Hunt Activity (方位句型)
 
Culture study unit 11 worksheet
Culture study unit 11 worksheetCulture study unit 11 worksheet
Culture study unit 11 worksheet
 

Similar to Scaling AEM (CQ5) Gem Session

Aem asset optimizations & best practices
Aem asset optimizations & best practicesAem asset optimizations & best practices
Aem asset optimizations & best practices
Kanika Gera
 
Redis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs TalksRedis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs Talks
Redis Labs
 
Veeam webinar - Deduplication best practices
Veeam webinar - Deduplication best practicesVeeam webinar - Deduplication best practices
Veeam webinar - Deduplication best practices
Joep Piscaer
 
Orlando DNN Usergroup Pres 12/06/11
Orlando DNN Usergroup Pres 12/06/11Orlando DNN Usergroup Pres 12/06/11
Orlando DNN Usergroup Pres 12/06/11
Jess Coburn
 
Docker vs. Kubernetes vs. Serverless
Docker vs. Kubernetes vs. ServerlessDocker vs. Kubernetes vs. Serverless
Docker vs. Kubernetes vs. Serverless
LogicworksNY
 
Automating a PostgreSQL High Availability Architecture with Ansible
Automating a PostgreSQL High Availability Architecture with AnsibleAutomating a PostgreSQL High Availability Architecture with Ansible
Automating a PostgreSQL High Availability Architecture with Ansible
EDB
 
NoSQL and ACID
NoSQL and ACIDNoSQL and ACID
NoSQL and ACID
FoundationDB
 
ActiveMQ Performance Tuning
ActiveMQ Performance TuningActiveMQ Performance Tuning
ActiveMQ Performance Tuning
Christian Posta
 
Running Oracle EBS in the cloud (DOAG TECH17 edition)
Running Oracle EBS in the cloud (DOAG TECH17 edition)Running Oracle EBS in the cloud (DOAG TECH17 edition)
Running Oracle EBS in the cloud (DOAG TECH17 edition)
Andrejs Prokopjevs
 
Scaling PHP apps
Scaling PHP appsScaling PHP apps
Scaling PHP apps
Matteo Moretti
 
Emc sql server 2012 overview
Emc sql server 2012 overviewEmc sql server 2012 overview
Emc sql server 2012 overview
solarisyougood
 
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Bhupesh Bansal
 
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedInHadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
Hadoop User Group
 
Turbocharging php applications with zend server (workshop)
Turbocharging php applications with zend server (workshop)Turbocharging php applications with zend server (workshop)
Turbocharging php applications with zend server (workshop)
Eric Ritchie
 
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community
 
OVH Lab - Enterprise Cloud Databases
OVH Lab - Enterprise Cloud DatabasesOVH Lab - Enterprise Cloud Databases
OVH Lab - Enterprise Cloud Databases
OVHcloud
 
High Performance Drupal
High Performance DrupalHigh Performance Drupal
High Performance Drupal
Jeff Geerling
 
Containerizing GPU Applications with Docker for Scaling to the Cloud
Containerizing GPU Applications with Docker for Scaling to the CloudContainerizing GPU Applications with Docker for Scaling to the Cloud
Containerizing GPU Applications with Docker for Scaling to the Cloud
Subbu Rama
 
Data Scotland 2019: You can run SQL Server on AWS
Data Scotland 2019: You can run SQL Server on AWSData Scotland 2019: You can run SQL Server on AWS
Data Scotland 2019: You can run SQL Server on AWS
John McCormack
 
Connect2013 id506 hadr ideas for social business
Connect2013 id506 hadr ideas for social businessConnect2013 id506 hadr ideas for social business
Connect2013 id506 hadr ideas for social business
Luis Guirigay
 

Similar to Scaling AEM (CQ5) Gem Session (20)

Aem asset optimizations & best practices
Aem asset optimizations & best practicesAem asset optimizations & best practices
Aem asset optimizations & best practices
 
Redis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs TalksRedis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs Talks
 
Veeam webinar - Deduplication best practices
Veeam webinar - Deduplication best practicesVeeam webinar - Deduplication best practices
Veeam webinar - Deduplication best practices
 
Orlando DNN Usergroup Pres 12/06/11
Orlando DNN Usergroup Pres 12/06/11Orlando DNN Usergroup Pres 12/06/11
Orlando DNN Usergroup Pres 12/06/11
 
Docker vs. Kubernetes vs. Serverless
Docker vs. Kubernetes vs. ServerlessDocker vs. Kubernetes vs. Serverless
Docker vs. Kubernetes vs. Serverless
 
Automating a PostgreSQL High Availability Architecture with Ansible
Automating a PostgreSQL High Availability Architecture with AnsibleAutomating a PostgreSQL High Availability Architecture with Ansible
Automating a PostgreSQL High Availability Architecture with Ansible
 
NoSQL and ACID
NoSQL and ACIDNoSQL and ACID
NoSQL and ACID
 
ActiveMQ Performance Tuning
ActiveMQ Performance TuningActiveMQ Performance Tuning
ActiveMQ Performance Tuning
 
Running Oracle EBS in the cloud (DOAG TECH17 edition)
Running Oracle EBS in the cloud (DOAG TECH17 edition)Running Oracle EBS in the cloud (DOAG TECH17 edition)
Running Oracle EBS in the cloud (DOAG TECH17 edition)
 
Scaling PHP apps
Scaling PHP appsScaling PHP apps
Scaling PHP apps
 
Emc sql server 2012 overview
Emc sql server 2012 overviewEmc sql server 2012 overview
Emc sql server 2012 overview
 
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
 
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedInHadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
 
Turbocharging php applications with zend server (workshop)
Turbocharging php applications with zend server (workshop)Turbocharging php applications with zend server (workshop)
Turbocharging php applications with zend server (workshop)
 
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph
 
OVH Lab - Enterprise Cloud Databases
OVH Lab - Enterprise Cloud DatabasesOVH Lab - Enterprise Cloud Databases
OVH Lab - Enterprise Cloud Databases
 
High Performance Drupal
High Performance DrupalHigh Performance Drupal
High Performance Drupal
 
Containerizing GPU Applications with Docker for Scaling to the Cloud
Containerizing GPU Applications with Docker for Scaling to the CloudContainerizing GPU Applications with Docker for Scaling to the Cloud
Containerizing GPU Applications with Docker for Scaling to the Cloud
 
Data Scotland 2019: You can run SQL Server on AWS
Data Scotland 2019: You can run SQL Server on AWSData Scotland 2019: You can run SQL Server on AWS
Data Scotland 2019: You can run SQL Server on AWS
 
Connect2013 id506 hadr ideas for social business
Connect2013 id506 hadr ideas for social businessConnect2013 id506 hadr ideas for social business
Connect2013 id506 hadr ideas for social business
 

Recently uploaded

HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Jeffrey Haguewood
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
flufftailshop
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
Shinana2
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
Pravash Chandra Das
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
SitimaJohn
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
Dinusha Kumarasiri
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
alexjohnson7307
 

Recently uploaded (20)

HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
 

Scaling AEM (CQ5) Gem Session

  • 1. Scaling AEM Michael Marth | Senior Engineering Manager © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  • 2. Scalability? § Performance vs Scalability § § § § § Performance: “it takes X secs to do Y” Scalability: “it takes X secs to do Y simultaneously Z times” Yet, performance can help with scalability This talk is about horizontal scalability (vertical scaling is trivial) The content is documented at http://dev.day.com/docs/en/cq/current/deploying/scaling.html © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 2
  • 3. Preliminary Notes § This talk is about pre-Oak scalability patterns, i.e. the CQ5.x line § § with AEM 6 these patterns will change CRX2 Cluster is mostly a technology for reliability, not for scalability § Slight exception is read scalability © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 3
  • 4. Scaling AEM 1. High Volume and High Performance Delivery 2. High Frequency Input Feed 3. High Processing Input Feed 4. High Volume Input Feed 5. Many Editors 6. Geo-distributed Editors 7. Many DAM assets 8. Geo-distributed disaster recovery © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 4
  • 5. High Volume and High Performance Delivery - Description § Use Case: § High traffic site (100m impressions/d) § Examples: adobe.com § Limiting factor § CPU on publish © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5
  • 6. High Volume and High Performance Delivery - Solution Pattern § Leverage dispatcher caching as much as possible § § SSI and/or client-side for personalized content § § in latest dispatcher: single-page dispatcher flush and scripted flushing, use to cache/flush content in dispatcher Selectors for query caching CDN with short TTL P Disp round robin Load Balancer P Disp © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 6 CDN Client
  • 7. High Volume and High Performance Delivery § Notes § Related to rendering performance, see also § § § CQ performance patterns (use CQ timing component, prefer tree walking over JCR queries, use ClientLibraryManager to concat and minify JS, etc, see [1]) Generic performance patterns (reduce requests with e.g. css sprites, gzip responses, put JS calls at bottom of HTML, etc, see [2]) Anti-Pattern § Adding publishers before leveraging caching [1] http://dev.day.com/docs/en/cq/current/deploying/performance.html [2] http://shop.oreilly.com/product/9780596529307.do © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 7
  • 8. High Frequency Input Feed - Description § Use Case: news feed import (moderate amounts, but constant updates) § Limiting factor § Dispatcher cache invalidation § Therefore actual limiting factor is CPU on publish © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8
  • 9. High Frequency Input Feed - Solution Pattern 1 § Set up content structure so that other pages do not get invalidated on dispatcher cache § § § if possible: highly volatile content e.g. in /etc with latest dispatcher: single-page flush possible Separate replication queue (so that main queue is not blocked) Backend import process editing A © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. replication P 9 flush Disp
  • 10. High Frequency Input Feed - Solution Pattern 2 § Set up content structure so that other pages do not get invalidated on dispatcher cache § § as previous pattern Import directly into Publish (no replication necessary) Backend Backend import process A © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. import process P flush Disp
  • 11. High Frequency Input Feed § Questions to ask § Human filtering/processing needed? Then imports should be on author and replicated. § If no: is the use case OK with different states on publish? § if yes: no replication needed, then pattern 2 is preferable © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 11
  • 12. High Processing Input Feed - Description § Use Case: § DAM import of images § § happens regularly § § 1000 images at once other editors are editing content at the same time Limiting factor § CPU, memory on author © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 12
  • 13. High Processing Input Feed - Solution Pattern 1 § Separate processing instances from human editing instances § Offload 1 Workflow step, e.g. thumbnail generation from PSDs § There can be more than 1 processing instance § Replicate back and forth in packages if possible § CQ5.6.1: share DS between instances and replicate without binary, offloading framework Processing Author Processing Author workflow steps running, replicating results back offload CPU-intensive WF steps by replication editing © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. A workflows running 13
  • 14. High Processing Input Feed - Solution Pattern 2 § Separate pre-processing instances for uploading § There can be more than 1 pre-processing instance § CQ5.6.1: share DS between instances and replicate without binary workflows running image upload Processing Author replication editing © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. A 14
  • 15. High Processing Input Feed § Notes § Author cluster can help mitigate the problem, but editors must edit content on slave § Throttling WFs or execution during night can help mitigate the problem § If the import is limited by CPU needed image conversion consider using ImageMagick rather than Java © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 15
  • 16. High Volume Input Feed - Description § Use Case: § Product data import § § 1 million products, 10000 modifications/day Limiting factor § Writing to the repository § § reads are also blocked Potentially (to a lesser degree) in case repository scans are needed to create diffs: § CPU for calculating diffs § Repository read caches get flushed © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 16
  • 17. High Volume Input Feed - Solution Pattern § Separate import instance to process imports, partition if possible § § only useful if import requires significant CPU (e.g. no diff delivered) Replicate to author § Replicate as package § § CQ5.6.1: share DS between instances and replicate without binary Replication to publish as package if possible import into repository Backend import process Import Processing Author replication of diff editing © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. A 17 replication of diff P
  • 18. High Volume Input Feed § Questions to ask § § Can the import be throttled? Most problems get much less severe. Notes § Use batch saves (1000 nodes) on import (reduces overhead in indexing, etc and speeds up the import overall) § Import as nt:unstructured rather cq:Page if possible § § If not: switch off heavy listeners (e.g. ContentSync) or use the JcrObservationThrottle Anti-Pattern § Usage of network disc (usually have high latency) § Replicating to publish through same replication queue as editorial content § § Might heavily delay important editorial changes from publication Using PageManager (cannot operate in batch mode, unless you use 5.6.1 where that option was added) © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 18
  • 19. Many Editors - Description § Use Case: § § § News or media portal >50 editors editing content concurrently Limiting factor § Depends on what do the editors actually do, eg. MSM rollouts, starting WFs, but usually CPU-bound © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 19
  • 20. Many Editors - Solution Pattern 1 § Sharding: split up different web sites / parts of web sites onto separate author instances § Publish instances are shared editing A replication site 1 editing A P site 2 editing A P site 3 © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. P 20
  • 21. Many Editors - Solution Pattern 2 § Sharding: split up different web sites into separate author instances, but replicate into one main author, e.g. for shared workflow processes § § § Practical if the shards do not need to share content. Cross-replication can be done, but will be hard to keep consistent Publish instances are shared A editing site 1 A editing replication site 2 A editing site 3 © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 21 A P
  • 22. Many Editors § Questions to ask § § What do the editors actually do? Notes § Author dispatcher helps to reduce CPU load on author instances § Author CRX cluster instead of sharding will mitigate the problem if CPU-bound © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 22
  • 23. Geo-distributed Editors § Use Case: § § Editors located in different geos (US, EMEA, APAC) Limiting factor § Bandwidth between editor location and author server location © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 23
  • 24. Geo-distributed Editors - Solution Pattern § Use Dispatcher in front of Author § Guiding principle: limit traffic between Dispatcher and editor location. § gzip traffic § Use Client Library Manager to minimize traffic § § minify, concat and gzip all client libraries Cache all responses that are not under /content in § Editor’s browser cache § Potentially also dispatcher cache cache cache minimize requests gzip responses © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Disp 24 A
  • 25. Geo-distributed Editors § Notes § In extreme cases consider writing templates that treat author renditions differently from publish renditions (especially reducing the number of necessary requests, e.g. by dropping requests to tracking servers, external CSS, etc) § Or use Scaffolding for editing © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 25
  • 26. Many DAM Assets § Use Case: § § Many assets (>5Mio) in DAM Limiting factor § Disc space © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 26
  • 27. Many DAM Assets - Solution Pattern § Split physical storage of data store and repository tar files § § for data store high latency is acceptable § § tar files need disc with very low latency Locate data store on cheap discs remotely (NAS, S3) Share data store between instances § In 5.6.1: use binary-less replication in case of shared DS to minimize network traffic Repository tar local © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. DS S3 27
  • 28. Many DAM Assets § Notes § In case of shared DS: the DS garbage collection needs to be run on an instance that keeps references to all assets in DS § In 5.6.1: huge performance improvements (~10x or more) for DS GC when the persistence is tar-based © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 28
  • 29. Geo-distributed disaster recovery § Use Case: § § § Data centers located in different geos One DC shall act as failover for author Limiting factor § Latency between DCs (in very low latency cases CRX clustering could be used) © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 29
  • 30. Geo-distributed disaster recovery - Solution Pattern § Use file level tools like rysnc to create replicas in 2nd DC § Hourly: sync data store § § § This is usually the most time consuming part Sync can be performed anytime, due to add-only data store architecture Nightly: § Create incremental backup into filesystem on 1st DC to get consistent state of files § Rsync backup to 2nd DC. For that period CQ on 2nd DC must not be running. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 30
  • 31. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.