CZ
Uploaded byCarsten Ziegeler
PDF, PPTX14,501 views

Adobe AEM - From Eventing to Job Processing

The document discusses the transition from eventing to job processing within Adobe's technology stack, focusing on the use of OSGi Event Admin for event publishing and job management. It outlines the architecture for job distribution, highlighting the components involved and the handling of job consumers and jobs in both clustered and non-clustered setups. It also introduces new features in version 5.6.1, such as job offloading and the integration of workflows for better resource management.

Embed presentation

Download as PDF, PPTX
FROM EVENTING TO JOB PROCESSING Carsten Ziegeler | Adobe Research Switzerland 1
2 •  RnD Team at Adobe Research Switzerland •  Co-founder Adobe Granite •  OSGi Core Platform and Enterprise Expert Groups •  Member of the ASF •  Current PMC Chair of Apache Sling •  Apache Sling, Felix, ACE •  Conference Speaker •  Technical Reviewer •  Article/Book Author cziegele@adobe.com @cziegeler ABOUT
3 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
4 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
5 OSGI EVENT ADMIN Publish / Subscribe Model OSGi Event Admin Component A publish deliver Component X Component Y
6 •  OSGi event is a data object with •  Topic (hierarchical namespace) •  Properties (key-value-pairs) •  Page Event •  Topic: com/day/cq/wcm/core/page •  Properties: path, change type (add/remove/edit) etc. OSGI EVENT ADMIN Publish / Subscribe Model
7 •  Publisher creates event object •  Sends event through EventAdmin service •  Either sync or async delivery •  Subscriber is an OSGi service (EventHandler) •  Service registration properties •  Interested topic(s) •  com/day/cq/wcm/core/* •  Additional filters (optional) •  (type="add“) OSGI EVENT ADMIN Publish / Subscribe Model
8 •  Immediate delivery to available subscribers •  No guarantee of delivery •  No distributed delivery OSGI EVENT ADMIN Publish / Subscribe Model
9 •  Immediate delivery to available subscribers •  No guarantee of delivery •  No distributed delivery OSGI EVENT ADMIN Publish / Subscribe Model DiscoverySling Job Distribution
10 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
11 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution New in 5.6.1
12 INSTALLATION SCENARIOS Clustered CRX Single Instance CRX Instance 1 Instance 2 Instance 3
13 Instance: Unique Id (Sling ID) TOPOLOGIES I Apache Sling Discovery Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3
14 •  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader TOPOLOGIES I Apache Sling Discovery Cluster 99Cluster 35 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Leader
15 •  Topology: Set of clusters TOPOLOGIES I Apache Sling Discovery Cluster 99Cluster 35 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Leader TopologyTopology •  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader
16 TOPOLOGIES I Apache Sling Discovery •  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader •  Topology: Set of clusters Cluster 99 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Topology Cluster 35 Leader
Cluster 99 17 •  Instance •  Sling ID •  Optional: Name and description •  Belongs to a cluster •  Might be the cluster leader •  Additional properties which are distributed •  Extensible through own services (PropertyProvider) •  E.g. data center, region or enabled job topics •  Cluster •  Elects (stable) leader •  Stable instance ordering •  TopologyEventListener: receives events on topology changes TOPOLOGIES II Apache Sling Discovery ID : 42 Instance 3 Topology Leader
18 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
19 •  Job : Guaranteed processing, exactly once •  Exactly one job consumer •  Started by client code, e.g. for replication, workflow... •  Job topic •  Payload is a serializable map JOB HANDLING I Apache Sling Job Handling
20 •  Sling Job Manager handles and distributes jobs •  Delivers job to a job consumer… •  …and waits for response •  Retry and failover •  Notification listeners (fail, retry, success) JOB HANDLING I Apache Sling Job Handling
21 STARTING / PROCESSING A JOB I Apache Sling Job Handling public interface JobConsumer { String PROPERTY_TOPICS = "job.topics"; enum JobResult { OK, FAILED, CANCEL, ASYNC } JobResult process(Job job); } public interface JobManager { Job addJob(String topic, String optionalName, Map<String, Object> properties); … } Starting a job Processing a job Note: Starting/processing of jobs through Event Admin is deprecated but still supported New in 5.6.1
22 @Component @Service(value={JobConsumer.class}) @Property(name=JobConsumer.PROPERTY_TOPICS, value="org/apache/sling/jobs/backup") public class BackupJobConsumer implements JobConsumer { @Override public JobResult process(final Job job) { // do backup return JobResult.OK; } } STARTING / PROCESSING A JOB II Apache Sling Job Handling New in 5.6.1
23 •  New jobs are immediately persisted (resource tree / repository) •  Jobs are “pushed” to the processing instance •  Processing instances use different queues •  Associated with job topic(s) •  Main queue •  0..n custom queues •  For example: replication agent queue or workflow queue JOB HANDLING II Apache Sling Job Handling
24 •  Queue is configurable •  Queue is started on demand in own thread •  And stopped if unused for some time •  Types •  Ordered queue (eg replication) •  Parallel queues: Plain and Topic Round Robin (eg workflow) •  Limit for parallel threads per queue •  Number of retries (-1 = endless) •  Retry delay •  Thread priority JOB QUEUE Apache Sling Job Handling
25 •  Job Manager Configuration = Main Queue Configuration •  Maximum parallel jobs (15) •  Retries (10) •  Retry Delay •  Eventing Thread Pool Configuration •  Used by all queues •  Pool size (35) = Maximum parallel jobs for a single instance ADDITIONAL CONFIGURATIONS Apache Sling Job Handling
26 MONITORING – WEB CONSOLE Apache Sling Job Handling
27 MONITORING – WEB CONSOLE Apache Sling Job Handling
28 •  Each instance determines enabled job topics •  Derived from Job Consumers (new API required) •  Can be whitelisted/blacklisted (in Job Consumer Manager) •  Announced through Topology •  Job Distribution depends on enabled job topics and queue type •  Potential set of instances derived from topology (enabled job topics) •  Ordered : processing on leader only, one job after the other •  Parallel: Round robin distribution on all potential instances •  Local cluster instances have preference •  Failover •  Instance crash: leader redistributes jobs to available instances •  Leader change taken into account •  On enabled job topics changes: potential redistribution JOB DISTRIBUTION Apache Sling Job Handling
29 •  Scalability in AEM: •  DAM Ingestion •  Non-Clustered installation requirement •  The term Offloading: •  In AEM used for all things job distribution and topology in clustered and non- clustered installations, e.g. ‘Offloading Browser’ •  More technically it’s ‘only’ a little add-on in Granite to Sling Job Distribution for handling non-clustered installations WHY OFFLOADING
30 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
31 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
32 SLING JOB DISTRIBUTION Topology Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job Manager Job Consumer Topic: A Job Consumer Topic: B Job Consumer Topic: C
33 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerA Job Consumer Topic: A Job Consumer Topic: B A:2 Job Consumer Topic: CJob Topology
34 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerB Job Consumer Topic: A Job Consumer Topic: B B:3 Job Consumer Topic: CJob Topology
35 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerC Job Consumer Topic: A Job Consumer Topic: B C:4 Job Consumer Topic: CJob ? Topology
36 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerC Job Consumer Topic: A Job Consumer Topic: B C:4 Job Consumer Topic: CJob Offloading Framework Topology
37 •  Detects offloading jobs •  Transport of job and job payload between origin and target instance •  Uses replication for the transport •  No distribution of jobs •  No execution of jobs OFFLOADING FRAMEWORK Overview Instance Sling ID: 1 Job Manager Offloading Job Manager C:4 Instance Sling ID: 4 Job Manager Job Consumer Topic: C Offloading Job Manager C:4 C Job Replication 1 2 3 5 4 6 7 Topology
38 •  Replication agents are created automatically •  Uses naming convention •  Needs manual adjustments (replication user) OFFLOADING FRAMEWORK Replication Worker Sling ID: 4 Author Sling ID: 1 offloading_4 (outgoing agent) offloading_reverse_4 (reverse agent) offloading_outbox (outbox agent)
39 •  Additional properties required •  Offloading job input •  Property: OffloadingJobProperties. INPUT_PAYLOAD (offloading.input.payload) •  Offloading job output •  Property: OffloadingJobProperties. OUTPUT_PAYLOAD (offloading.output.payload) •  Takes comma separated list of paths •  Used to build the replication package •  Job sender needs to set these properties for offloading to work! OFFLOADING FRAMEWORK Job Payload
40 •  Configures Job Consumers •  Configures the topic white/black listing properties of each instance •  What jobs to execute on what instance •  Configures the distribution •  Configuration applies for both, clustered and non-clustered installations OFFLOADING BROWSER UI
41 OFFLOADING BROWSER UI
42 OFFLOADING BROWSER UI
43 OFFLOADING BROWSER UI
44 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
45 •  New JobConsumer •  Class: WorkflowOffloadingJobConsumer •  Topic: com/adobe/granite/workflow/offloading •  Can launch new workflows •  Expects the workflow model on the job payload •  Expects the workflow payload on the job payload •  For use with clustered and non-clustered installations WORKFLOW DISTRIBUTION
46 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified)
47 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Define the job topic
48 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Create service component •  Must register with topic •  Implement new JobConsumer interface
49 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Access job properties (payload) •  Read workflow model and payload from job properties
50 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Workflow specific •  Use workflow API to start workflow for the given model and payload
51 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Use JobResult enumeration to report back the job status
52 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
53 •  Default ingestion workflow: “DAM Update Asset” •  Load is put on the instance where the workflow is started, usually the author •  New ingestion workflow: “DAM Update Asset Offloading” •  Needs to be manually enabled by changing the workflow launcher •  New workflow model with a single step: AssetOffloadingProcess •  Uses WorkflowExternalProcess API •  Creates a new job on topic: com/adobe/granite/workflow/offloading •  Allows distributing the default ingestion workflow •  Load is put on the instance where the job is distributed to •  Can be used to distribute in clustered and non-clustered installations DAM INGESTION
54 @Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step)
55 @Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Create service component •  Implement WorkflowExternalProcess interface •  Reference JobManager service
56 @Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  DAM and Workflow specific •  Resolve to Asset •  Read model from meta data •  Read workflow payload from Asset path
57 public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  ValueMap for job properties •  Put model and payload on job properties •  Used by the JobConsumer
58 public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Build offloading payload properties •  Comma separated list of paths •  Put them on the job payload as well •  Only used for non-clustered distribution
59 public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Create job using JobManager service •  Use topic from job consumer •  Put job payload properties •  Return the jobId as the workflow process id (workflow specific)
60 @Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ …. } public boolean hasFinished(Serializable externalProcessId, ..){ // returns null, if job is finished Job offloadingJob = jobManager.getJobById((String) externalProcessId); return offloadingJob == null; } } DAM INGESTION Create Job (from workflow step) •  Workflow API specific callback •  Process id = jobId, from execute() •  Query job by jobId •  Workflow step finished when job is finished
61 •  Choose a job topic •  Create JobConsumer component and register with topic chosen •  To create a new job use new JobManager.addJob() API with the topic chosen and the job payload •  Add offloading payload to job payload •  Bundle and deploy JobConsumer on topology instances •  Enable/Disable the new topic on the instances, using Offloading Browser DEVELOPMENT - RECIPE
62 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
63 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution org.apache.sling.discovery.api •  Discovers topology •  Cross-Cluster detection •  Foundation for job distribution
64 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution org.apache.sling.event •  Uses Sling Discovery •  New JobConsumer API and job topics •  New JobManager API for creating new distributed jobs •  Distributes jobs based on available job topics and job queue configuration. •  Distributes in the whole topology, including clustered and non- clustered instances •  Can execute cluster local jobs only
65 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution com.adobe.granite.offloading.core •  Builds on top of Sling Distributed Jobs •  Does not perform distribution •  Detects jobs distributed to non-clustered instances •  Transports the jobs and payload to non-clustered instances •  Uses replication for transport •  Does not execute jobs
66 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution com.adobe.granite.workflow.core •  Defines a job consumer to distribute the execution of whole workflows •  Defines topic com/adobe/granite/workflow/ offloading •  Implements WorkflowOffloadingJobConsumer component •  Supports clustered and non clustered installations
67 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution cq-dam-core •  Makes use of com/adobe/granite/workflow/ offloading topic from Workflow Distribution •  New workflow step (external step) that creates a new job on topic com/adobe/granite/workflow/ offloading •  New “DAM Update Asset Offloading” workflow •  Supports clustered and non clustered configurations
68 TAKE AWAY Potential Future Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
69 TAKE AWAY Potential Future Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing New OSGi Specification • Distributed Eventing • Cloud Computing
70 TAKE AWAY Potential Future Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing Job Distribution • Improved load balancing • Pull based distribution
71 TAKE AWAY Potential Future Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing Offloading • As part of job distribution • Even simpler setup
72 TAKE AWAY Questions? Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
73

More Related Content

PPTX
Osgi
byHeena Madan
 
PPTX
AEM and Sling
byLo Ki
 
PDF
Designing APIs with OpenAPI Spec
byAdam Paxton
 
PPTX
Secure your app with keycloak
byGuy Marom
 
PPTX
AWS API Gateway
byMuhammed YALÇIN
 
PPTX
Introduction to Maven
byOnkar Deshpande
 
PDF
Spring Boot
byJaran Flaath
 
PDF
Swagger
byNexThoughts Technologies
 
Osgi
byHeena Madan
 
AEM and Sling
byLo Ki
 
Designing APIs with OpenAPI Spec
byAdam Paxton
 
Secure your app with keycloak
byGuy Marom
 
AWS API Gateway
byMuhammed YALÇIN
 
Introduction to Maven
byOnkar Deshpande
 
Spring Boot
byJaran Flaath
 
Swagger
byNexThoughts Technologies
 

What's hot

PPTX
Spring Security 5
byJesus Perez Franco
 
PPTX
Apache Maven
byRahul Tanwani
 
PPTX
Docker Kubernetes Istio
byAraf Karsh Hamid
 
PPTX
Maven ppt
bynatashasweety7
 
PDF
Understanding Sling Models in AEM
byAccunity Software
 
PDF
Spring Security
byKnoldus Inc.
 
PDF
OpenAPI 3.0, And What It Means for the Future of Swagger
bySmartBear
 
PPTX
Laravel overview
byObinna Akunne
 
PPTX
Fondamentaux d’une API REST
byAbdoulaye Dieng
 
PPTX
Building secure applications with keycloak
byAbhishek Koserwal
 
PPTX
Introducing Swagger
byTony Tam
 
PPTX
OpenID Connect: An Overview
byPat Patterson
 
PPTX
API Docs with OpenAPI 3.0
byFabrizio Ferri-Benedetti
 
PDF
[오픈소스컨설팅] 스카우터 사용자 가이드 2020
byJi-Woong Choi
 
PPTX
AEM (CQ) Dispatcher Security and CDN+Browser Caching
byAndrew Khoury
 
PDF
Open Policy Agent
byTorin Sandall
 
PDF
Distributed tracing using open tracing &amp; jaeger 2
byChandresh Pancholi
 
PDF
지금 핫한 Real-time In-memory Stream Processing 이야기
byTed Won
 
PPTX
Final terraform
byGourav Varma
 
PDF
Spring Boot & Actuators
byVMware Tanzu
 
Spring Security 5
byJesus Perez Franco
 
Apache Maven
byRahul Tanwani
 
Docker Kubernetes Istio
byAraf Karsh Hamid
 
Maven ppt
bynatashasweety7
 
Understanding Sling Models in AEM
byAccunity Software
 
Spring Security
byKnoldus Inc.
 
OpenAPI 3.0, And What It Means for the Future of Swagger
bySmartBear
 
Laravel overview
byObinna Akunne
 
Fondamentaux d’une API REST
byAbdoulaye Dieng
 
Building secure applications with keycloak
byAbhishek Koserwal
 
Introducing Swagger
byTony Tam
 
OpenID Connect: An Overview
byPat Patterson
 
API Docs with OpenAPI 3.0
byFabrizio Ferri-Benedetti
 
[오픈소스컨설팅] 스카우터 사용자 가이드 2020
byJi-Woong Choi
 
AEM (CQ) Dispatcher Security and CDN+Browser Caching
byAndrew Khoury
 
Open Policy Agent
byTorin Sandall
 
Distributed tracing using open tracing &amp; jaeger 2
byChandresh Pancholi
 
지금 핫한 Real-time In-memory Stream Processing 이야기
byTed Won
 
Final terraform
byGourav Varma
 
Spring Boot & Actuators
byVMware Tanzu
 

Viewers also liked

PDF
Apache Sling - Distributed Eventing, Discovery, and Jobs (adaptTo 2013)
byCarsten Ziegeler
 
PDF
AEM Best Practices for Component Development
byGabriel Walt
 
PPT
Adobe Experience Manager - Replication deep dive
bymwmd
 
PDF
Sling Dynamic Include
byTomasz Rękawek
 
PPTX
Smart Interpretation - Fast AEM Modelling - SAGEEP 2017
byTorben Bach
 
PDF
Scala for scripting
byday
 
PDF
Scala4sling
byday
 
PPTX
Bombardier adobe aem msm implementation
byKen Knitter
 
PDF
Use Case: Building OSGi Enterprise Applications (QCon 14)
byCarsten Ziegeler
 
PDF
What's cool in the new and updated OSGi Specs
byCarsten Ziegeler
 
PDF
What's cool in the new and updated OSGi specs
byCarsten Ziegeler
 
PDF
Monitoring OSGi Applications with the Web Console
byCarsten Ziegeler
 
PDF
Distributed Eventing in OSGi
byCarsten Ziegeler
 
PDF
Introducing CQ 5.1
byDavid Nuescheler
 
PPTX
CQ5 Development Setup, Maven Build and Deployment
by6D Global
 
PDF
Hubot + Slack + AEM
byMax Barrass
 
PPTX
Introdcution to Adobe CQ
byRest West
 
PPTX
User interface customization for aem6 circuit
byDamien Antipa
 
PDF
OSGi, Scripting and REST, Building Webapps With Apache Sling
byCarsten Ziegeler
 
PPTX
UI Customization in AEM 6.0
byGilles Knobloch
 
Apache Sling - Distributed Eventing, Discovery, and Jobs (adaptTo 2013)
byCarsten Ziegeler
 
AEM Best Practices for Component Development
byGabriel Walt
 
Adobe Experience Manager - Replication deep dive
bymwmd
 
Sling Dynamic Include
byTomasz Rękawek
 
Smart Interpretation - Fast AEM Modelling - SAGEEP 2017
byTorben Bach
 
Scala for scripting
byday
 
Scala4sling
byday
 
Bombardier adobe aem msm implementation
byKen Knitter
 
Use Case: Building OSGi Enterprise Applications (QCon 14)
byCarsten Ziegeler
 
What's cool in the new and updated OSGi Specs
byCarsten Ziegeler
 
What's cool in the new and updated OSGi specs
byCarsten Ziegeler
 
Monitoring OSGi Applications with the Web Console
byCarsten Ziegeler
 
Distributed Eventing in OSGi
byCarsten Ziegeler
 
Introducing CQ 5.1
byDavid Nuescheler
 
CQ5 Development Setup, Maven Build and Deployment
by6D Global
 
Hubot + Slack + AEM
byMax Barrass
 
Introdcution to Adobe CQ
byRest West
 
User interface customization for aem6 circuit
byDamien Antipa
 
OSGi, Scripting and REST, Building Webapps With Apache Sling
byCarsten Ziegeler
 
UI Customization in AEM 6.0
byGilles Knobloch
 

Similar to Adobe AEM - From Eventing to Job Processing

PDF
EVOLVE'13 | Enhance | Eventing to job Processing | Carsten Zeigler
byEvolve The Adobe Digital Marketing Community
 
PDF
Distributed Eventing in OSGi - Carsten Ziegeler
bymfrancis
 
KEY
Cooking a rabbit pie
byTomas Doran
 
PDF
Think BIG, Spend Small A Look at how WSO2 Can Help Scale Up with Less Cost
byWSO2
 
PDF
Amazon SWF and Gordon
byJonathan Weiss
 
KEY
Modern Continuous Software Delivery
byMartin Logan
 
PPTX
Moving to DevOps the Amazon Way
byAmazon Web Services LATAM
 
PPTX
AWS Initiate - DevOps do Jeito Amazon
byAmazon Web Services LATAM
 
PDF
CIRCUIT 2015 - 10 Things Apache Sling Can Do
byICF CIRCUIT
 
ODP
Scheduler_session
byshivani garg
 
PDF
DevOps: Automate all the things
byMat Mannion
 
PDF
Netflix conductor
byViren Baraiya
 
PDF
AWS Community Day Ireland - Building roads and bridges in the last decade of ...
byHeitor Lessa
 
KEY
Real time system_performance_mon
byTomas Doran
 
PDF
Simple Solutions for Complex Problems - Boulder Meetup
byNATS
 
PDF
Simple Solutions for Complex Problems - Boulder Meetup
byApcera
 
PDF
Serverless best practices plus design principles 20m version
byHeitor Lessa
 
PDF
Genomics on aws-webinar-april2018
byBrendan Bouffler
 
PDF
Journey towards serverless infrastructure
byVille Seppänen
 
PPTX
Meetup gitlab intro in seoul
byGuenjun Yoo
 
EVOLVE'13 | Enhance | Eventing to job Processing | Carsten Zeigler
byEvolve The Adobe Digital Marketing Community
 
Distributed Eventing in OSGi - Carsten Ziegeler
bymfrancis
 
Cooking a rabbit pie
byTomas Doran
 
Think BIG, Spend Small A Look at how WSO2 Can Help Scale Up with Less Cost
byWSO2
 
Amazon SWF and Gordon
byJonathan Weiss
 
Modern Continuous Software Delivery
byMartin Logan
 
Moving to DevOps the Amazon Way
byAmazon Web Services LATAM
 
AWS Initiate - DevOps do Jeito Amazon
byAmazon Web Services LATAM
 
CIRCUIT 2015 - 10 Things Apache Sling Can Do
byICF CIRCUIT
 
Scheduler_session
byshivani garg
 
DevOps: Automate all the things
byMat Mannion
 
Netflix conductor
byViren Baraiya
 
AWS Community Day Ireland - Building roads and bridges in the last decade of ...
byHeitor Lessa
 
Real time system_performance_mon
byTomas Doran
 
Simple Solutions for Complex Problems - Boulder Meetup
byNATS
 
Simple Solutions for Complex Problems - Boulder Meetup
byApcera
 
Serverless best practices plus design principles 20m version
byHeitor Lessa
 
Genomics on aws-webinar-april2018
byBrendan Bouffler
 
Journey towards serverless infrastructure
byVille Seppänen
 
Meetup gitlab intro in seoul
byGuenjun Yoo
 

More from Carsten Ziegeler

PDF
Service oriented web development with OSGi
byCarsten Ziegeler
 
PDF
JCR - Java Content Repositories
byCarsten Ziegeler
 
PDF
Embrace Change - Embrace OSGi
byCarsten Ziegeler
 
PDF
JCR In Action (ApacheCon US 2009)
byCarsten Ziegeler
 
PDF
Apache Sling : JCR, OSGi, Scripting and REST
byCarsten Ziegeler
 
PDF
Embrace OSGi
byCarsten Ziegeler
 
PDF
Apache Sanselan (ApacheCon US 2007 FFT)
byCarsten Ziegeler
 
PDF
Apache iBatis (ApacheCon US 2007)
byCarsten Ziegeler
 
PDF
JCR In Action (ApacheCon US 2007)
byCarsten Ziegeler
 
PDF
Apache Portals Panel (ApacheCon US 2007)
byCarsten Ziegeler
 
PDF
JCR In Action (ApacheCon EU 2008)
byCarsten Ziegeler
 
PDF
Maven SCR Plugin (ApacheCon EU 2008 - FFT)
byCarsten Ziegeler
 
Service oriented web development with OSGi
byCarsten Ziegeler
 
JCR - Java Content Repositories
byCarsten Ziegeler
 
Embrace Change - Embrace OSGi
byCarsten Ziegeler
 
JCR In Action (ApacheCon US 2009)
byCarsten Ziegeler
 
Apache Sling : JCR, OSGi, Scripting and REST
byCarsten Ziegeler
 
Embrace OSGi
byCarsten Ziegeler
 
Apache Sanselan (ApacheCon US 2007 FFT)
byCarsten Ziegeler
 
Apache iBatis (ApacheCon US 2007)
byCarsten Ziegeler
 
JCR In Action (ApacheCon US 2007)
byCarsten Ziegeler
 
Apache Portals Panel (ApacheCon US 2007)
byCarsten Ziegeler
 
JCR In Action (ApacheCon EU 2008)
byCarsten Ziegeler
 
Maven SCR Plugin (ApacheCon EU 2008 - FFT)
byCarsten Ziegeler
 

Recently uploaded

PDF
The Evolving Role of the CEO in the Age of AI
byPipal Tree Services Executive Search Firm
 
PDF
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
PDF
The Necessity of Digital Forensics, the Digital Forensics Process & Laborator...
bychrstnpetwinchester
 
PDF
PCCC25（設立25年記念PCクラスタシンポジウム）：エヌビディア合同会社 テーマ２「NVIDIA BlueField-4 DPU」
byPC Cluster Consortium
 
PDF
Cheryl Hung, Vibe Coding Auth Without Melting Down! isaqb Software Architectu...
byCheryl Hung
 
PDF
[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...
byWintari Yasmin
 
PDF
Accessibility & Inclusion: What Comes Next. Presentation of the Digital Acces...
byAssociazione Digital Days
 
PDF
Mulesoft Meetup Online Portuguese: MCP e IA
byPedro Baroni
 
PDF
[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...
byWintari Yasmin
 
PDF
5 Common Supply Chain Attacks and How They Work | CyberPro Magazine
byCyberPro Magazine
 
PDF
How Much Does It Cost to Build an eCommerce Website in 2025.pdf
byPixlogix Infotech
 
PPTX
UFCD 0797 - SISTEMAS OPERATIVOS_Unidade Completa.pptx
byscribddobruno
 
PDF
Lets Build a Serverless Function with Kiro
byChris Bingham
 
PDF
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
PDF
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
PDF
[DevFest Strasbourg 2025] - NodeJs Can do that !!
bymoassiongbon
 
PDF
Transcript: The partnership effect: Libraries and publishers on collaborating...
byBookNet Canada
 
PDF
[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...
byWintari Yasmin
 
PDF
[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...
bywintari3
 
PPTX
Connecting the unconnectable: Exploring LoRaWAN for IoT
byasasiraarthur
 
The Evolving Role of the CEO in the Age of AI
byPipal Tree Services Executive Search Firm
 
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
The Necessity of Digital Forensics, the Digital Forensics Process & Laborator...
bychrstnpetwinchester
 
PCCC25（設立25年記念PCクラスタシンポジウム）：エヌビディア合同会社 テーマ２「NVIDIA BlueField-4 DPU」
byPC Cluster Consortium
 
Cheryl Hung, Vibe Coding Auth Without Melting Down! isaqb Software Architectu...
byCheryl Hung
 
[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...
byWintari Yasmin
 
Accessibility & Inclusion: What Comes Next. Presentation of the Digital Acces...
byAssociazione Digital Days
 
Mulesoft Meetup Online Portuguese: MCP e IA
byPedro Baroni
 
[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...
byWintari Yasmin
 
5 Common Supply Chain Attacks and How They Work | CyberPro Magazine
byCyberPro Magazine
 
How Much Does It Cost to Build an eCommerce Website in 2025.pdf
byPixlogix Infotech
 
UFCD 0797 - SISTEMAS OPERATIVOS_Unidade Completa.pptx
byscribddobruno
 
Lets Build a Serverless Function with Kiro
byChris Bingham
 
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
[DevFest Strasbourg 2025] - NodeJs Can do that !!
bymoassiongbon
 
Transcript: The partnership effect: Libraries and publishers on collaborating...
byBookNet Canada
 
[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...
byWintari Yasmin
 
[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...
bywintari3
 
Connecting the unconnectable: Exploring LoRaWAN for IoT
byasasiraarthur
 

Adobe AEM - From Eventing to Job Processing

  • 1.
    FROM EVENTING TOJOB PROCESSING Carsten Ziegeler | Adobe Research Switzerland 1
  • 2.
    2 •  RnD Teamat Adobe Research Switzerland •  Co-founder Adobe Granite •  OSGi Core Platform and Enterprise Expert Groups •  Member of the ASF •  Current PMC Chair of Apache Sling •  Apache Sling, Felix, ACE •  Conference Speaker •  Technical Reviewer •  Article/Book Author cziegele@adobe.com @cziegeler ABOUT
  • 3.
    3 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
  • 4.
    4 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
  • 5.
    5 OSGI EVENT ADMIN Publish/ Subscribe Model OSGi Event Admin Component A publish deliver Component X Component Y
  • 6.
    6 •  OSGi eventis a data object with •  Topic (hierarchical namespace) •  Properties (key-value-pairs) •  Page Event •  Topic: com/day/cq/wcm/core/page •  Properties: path, change type (add/remove/edit) etc. OSGI EVENT ADMIN Publish / Subscribe Model
  • 7.
    7 •  Publisher createsevent object •  Sends event through EventAdmin service •  Either sync or async delivery •  Subscriber is an OSGi service (EventHandler) •  Service registration properties •  Interested topic(s) •  com/day/cq/wcm/core/* •  Additional filters (optional) •  (type="add“) OSGI EVENT ADMIN Publish / Subscribe Model
  • 8.
    8 •  Immediate deliveryto available subscribers •  No guarantee of delivery •  No distributed delivery OSGI EVENT ADMIN Publish / Subscribe Model
  • 9.
    9 •  Immediate deliveryto available subscribers •  No guarantee of delivery •  No distributed delivery OSGI EVENT ADMIN Publish / Subscribe Model DiscoverySling Job Distribution
  • 10.
    10 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  • 11.
    11 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution New in 5.6.1
  • 12.
  • 13.
    13 Instance: Unique Id(Sling ID) TOPOLOGIES I Apache Sling Discovery Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3
  • 14.
    14 •  Instance: UniqueId (Sling ID) •  Cluster: Unique Id and leader TOPOLOGIES I Apache Sling Discovery Cluster 99Cluster 35 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Leader
  • 15.
    15 •  Topology: Setof clusters TOPOLOGIES I Apache Sling Discovery Cluster 99Cluster 35 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Leader TopologyTopology •  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader
  • 16.
    16 TOPOLOGIES I Apache SlingDiscovery •  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader •  Topology: Set of clusters Cluster 99 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Topology Cluster 35 Leader
  • 17.
    Cluster 99 17 •  Instance • Sling ID •  Optional: Name and description •  Belongs to a cluster •  Might be the cluster leader •  Additional properties which are distributed •  Extensible through own services (PropertyProvider) •  E.g. data center, region or enabled job topics •  Cluster •  Elects (stable) leader •  Stable instance ordering •  TopologyEventListener: receives events on topology changes TOPOLOGIES II Apache Sling Discovery ID : 42 Instance 3 Topology Leader
  • 18.
    18 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  • 19.
    19 •  Job :Guaranteed processing, exactly once •  Exactly one job consumer •  Started by client code, e.g. for replication, workflow... •  Job topic •  Payload is a serializable map JOB HANDLING I Apache Sling Job Handling
  • 20.
    20 •  Sling JobManager handles and distributes jobs •  Delivers job to a job consumer… •  …and waits for response •  Retry and failover •  Notification listeners (fail, retry, success) JOB HANDLING I Apache Sling Job Handling
  • 21.
    21 STARTING / PROCESSINGA JOB I Apache Sling Job Handling public interface JobConsumer { String PROPERTY_TOPICS = "job.topics"; enum JobResult { OK, FAILED, CANCEL, ASYNC } JobResult process(Job job); } public interface JobManager { Job addJob(String topic, String optionalName, Map<String, Object> properties); … } Starting a job Processing a job Note: Starting/processing of jobs through Event Admin is deprecated but still supported New in 5.6.1
  • 22.
    22 @Component @Service(value={JobConsumer.class}) @Property(name=JobConsumer.PROPERTY_TOPICS, value="org/apache/sling/jobs/backup") public classBackupJobConsumer implements JobConsumer { @Override public JobResult process(final Job job) { // do backup return JobResult.OK; } } STARTING / PROCESSING A JOB II Apache Sling Job Handling New in 5.6.1
  • 23.
    23 •  New jobsare immediately persisted (resource tree / repository) •  Jobs are “pushed” to the processing instance •  Processing instances use different queues •  Associated with job topic(s) •  Main queue •  0..n custom queues •  For example: replication agent queue or workflow queue JOB HANDLING II Apache Sling Job Handling
  • 24.
    24 •  Queue isconfigurable •  Queue is started on demand in own thread •  And stopped if unused for some time •  Types •  Ordered queue (eg replication) •  Parallel queues: Plain and Topic Round Robin (eg workflow) •  Limit for parallel threads per queue •  Number of retries (-1 = endless) •  Retry delay •  Thread priority JOB QUEUE Apache Sling Job Handling
  • 25.
    25 •  Job ManagerConfiguration = Main Queue Configuration •  Maximum parallel jobs (15) •  Retries (10) •  Retry Delay •  Eventing Thread Pool Configuration •  Used by all queues •  Pool size (35) = Maximum parallel jobs for a single instance ADDITIONAL CONFIGURATIONS Apache Sling Job Handling
  • 26.
    26 MONITORING – WEBCONSOLE Apache Sling Job Handling
  • 27.
    27 MONITORING – WEBCONSOLE Apache Sling Job Handling
  • 28.
    28 •  Each instancedetermines enabled job topics •  Derived from Job Consumers (new API required) •  Can be whitelisted/blacklisted (in Job Consumer Manager) •  Announced through Topology •  Job Distribution depends on enabled job topics and queue type •  Potential set of instances derived from topology (enabled job topics) •  Ordered : processing on leader only, one job after the other •  Parallel: Round robin distribution on all potential instances •  Local cluster instances have preference •  Failover •  Instance crash: leader redistributes jobs to available instances •  Leader change taken into account •  On enabled job topics changes: potential redistribution JOB DISTRIBUTION Apache Sling Job Handling
  • 29.
    29 •  Scalability inAEM: •  DAM Ingestion •  Non-Clustered installation requirement •  The term Offloading: •  In AEM used for all things job distribution and topology in clustered and non- clustered installations, e.g. ‘Offloading Browser’ •  More technically it’s ‘only’ a little add-on in Granite to Sling Job Distribution for handling non-clustered installations WHY OFFLOADING
  • 30.
    30 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  • 31.
    31 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  • 32.
    32 SLING JOB DISTRIBUTION Topology Instance SlingID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job Manager Job Consumer Topic: A Job Consumer Topic: B Job Consumer Topic: C
  • 33.
    33 SLING JOB DISTRIBUTION Instance SlingID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerA Job Consumer Topic: A Job Consumer Topic: B A:2 Job Consumer Topic: CJob Topology
  • 34.
    34 SLING JOB DISTRIBUTION Instance SlingID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerB Job Consumer Topic: A Job Consumer Topic: B B:3 Job Consumer Topic: CJob Topology
  • 35.
    35 SLING JOB DISTRIBUTION Instance SlingID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerC Job Consumer Topic: A Job Consumer Topic: B C:4 Job Consumer Topic: CJob ? Topology
  • 36.
    36 SLING JOB DISTRIBUTION Instance SlingID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerC Job Consumer Topic: A Job Consumer Topic: B C:4 Job Consumer Topic: CJob Offloading Framework Topology
  • 37.
    37 •  Detects offloadingjobs •  Transport of job and job payload between origin and target instance •  Uses replication for the transport •  No distribution of jobs •  No execution of jobs OFFLOADING FRAMEWORK Overview Instance Sling ID: 1 Job Manager Offloading Job Manager C:4 Instance Sling ID: 4 Job Manager Job Consumer Topic: C Offloading Job Manager C:4 C Job Replication 1 2 3 5 4 6 7 Topology
  • 38.
    38 •  Replication agentsare created automatically •  Uses naming convention •  Needs manual adjustments (replication user) OFFLOADING FRAMEWORK Replication Worker Sling ID: 4 Author Sling ID: 1 offloading_4 (outgoing agent) offloading_reverse_4 (reverse agent) offloading_outbox (outbox agent)
  • 39.
    39 •  Additional propertiesrequired •  Offloading job input •  Property: OffloadingJobProperties. INPUT_PAYLOAD (offloading.input.payload) •  Offloading job output •  Property: OffloadingJobProperties. OUTPUT_PAYLOAD (offloading.output.payload) •  Takes comma separated list of paths •  Used to build the replication package •  Job sender needs to set these properties for offloading to work! OFFLOADING FRAMEWORK Job Payload
  • 40.
    40 •  Configures JobConsumers •  Configures the topic white/black listing properties of each instance •  What jobs to execute on what instance •  Configures the distribution •  Configuration applies for both, clustered and non-clustered installations OFFLOADING BROWSER UI
  • 41.
  • 42.
  • 43.
  • 44.
    44 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  • 45.
    45 •  New JobConsumer • Class: WorkflowOffloadingJobConsumer •  Topic: com/adobe/granite/workflow/offloading •  Can launch new workflows •  Expects the workflow model on the job payload •  Expects the workflow payload on the job payload •  For use with clustered and non-clustered installations WORKFLOW DISTRIBUTION
  • 46.
    46 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS,value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified)
  • 47.
    47 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS,value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Define the job topic
  • 48.
    48 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS,value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Create service component •  Must register with topic •  Implement new JobConsumer interface
  • 49.
    49 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS,value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Access job properties (payload) •  Read workflow model and payload from job properties
  • 50.
    50 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS,value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Workflow specific •  Use workflow API to start workflow for the given model and payload
  • 51.
    51 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS,value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Use JobResult enumeration to report back the job status
  • 52.
    52 OVERVIEW Where to fitin the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  • 53.
    53 •  Default ingestionworkflow: “DAM Update Asset” •  Load is put on the instance where the workflow is started, usually the author •  New ingestion workflow: “DAM Update Asset Offloading” •  Needs to be manually enabled by changing the workflow launcher •  New workflow model with a single step: AssetOffloadingProcess •  Uses WorkflowExternalProcess API •  Creates a new job on topic: com/adobe/granite/workflow/offloading •  Allows distributing the default ingestion workflow •  Load is put on the instance where the job is distributed to •  Can be used to distribute in clustered and non-clustered installations DAM INGESTION
  • 54.
    54 @Component @Service public class AssetOffloadingProcessimplements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step)
  • 55.
    55 @Component @Service public class AssetOffloadingProcessimplements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Create service component •  Implement WorkflowExternalProcess interface •  Reference JobManager service
  • 56.
    56 @Component @Service public class AssetOffloadingProcessimplements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  DAM and Workflow specific •  Resolve to Asset •  Read model from meta data •  Read workflow payload from Asset path
  • 57.
    57 public Serializable execute(WorkItemworkItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  ValueMap for job properties •  Put model and payload on job properties •  Used by the JobConsumer
  • 58.
    58 public Serializable execute(WorkItemworkItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Build offloading payload properties •  Comma separated list of paths •  Put them on the job payload as well •  Only used for non-clustered distribution
  • 59.
    59 public Serializable execute(WorkItemworkItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Create job using JobManager service •  Use topic from job consumer •  Put job payload properties •  Return the jobId as the workflow process id (workflow specific)
  • 60.
    60 @Component @Service public class AssetOffloadingProcessimplements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ …. } public boolean hasFinished(Serializable externalProcessId, ..){ // returns null, if job is finished Job offloadingJob = jobManager.getJobById((String) externalProcessId); return offloadingJob == null; } } DAM INGESTION Create Job (from workflow step) •  Workflow API specific callback •  Process id = jobId, from execute() •  Query job by jobId •  Workflow step finished when job is finished
  • 61.
    61 •  Choose ajob topic •  Create JobConsumer component and register with topic chosen •  To create a new job use new JobManager.addJob() API with the topic chosen and the job payload •  Add offloading payload to job payload •  Bundle and deploy JobConsumer on topology instances •  Enable/Disable the new topic on the instances, using Offloading Browser DEVELOPMENT - RECIPE
  • 62.
    62 TAKE AWAY Discovery Offloading WorkflowDistribution DAM Ingestion Sling Granite AEM Job Distribution
  • 63.
    63 TAKE AWAY Discovery Offloading WorkflowDistribution DAM Ingestion Sling Granite AEM Job Distribution org.apache.sling.discovery.api •  Discovers topology •  Cross-Cluster detection •  Foundation for job distribution
  • 64.
    64 TAKE AWAY Discovery Offloading WorkflowDistribution DAM Ingestion Sling Granite AEM Job Distribution org.apache.sling.event •  Uses Sling Discovery •  New JobConsumer API and job topics •  New JobManager API for creating new distributed jobs •  Distributes jobs based on available job topics and job queue configuration. •  Distributes in the whole topology, including clustered and non- clustered instances •  Can execute cluster local jobs only
  • 65.
    65 TAKE AWAY Discovery Offloading WorkflowDistribution DAM Ingestion Sling Granite AEM Job Distribution com.adobe.granite.offloading.core •  Builds on top of Sling Distributed Jobs •  Does not perform distribution •  Detects jobs distributed to non-clustered instances •  Transports the jobs and payload to non-clustered instances •  Uses replication for transport •  Does not execute jobs
  • 66.
    66 TAKE AWAY Discovery Offloading WorkflowDistribution DAM Ingestion Sling Granite AEM Job Distribution com.adobe.granite.workflow.core •  Defines a job consumer to distribute the execution of whole workflows •  Defines topic com/adobe/granite/workflow/ offloading •  Implements WorkflowOffloadingJobConsumer component •  Supports clustered and non clustered installations
  • 67.
    67 TAKE AWAY Discovery Offloading WorkflowDistribution DAM Ingestion Sling Granite AEM Job Distribution cq-dam-core •  Makes use of com/adobe/granite/workflow/ offloading topic from Workflow Distribution •  New workflow step (external step) that creates a new job on topic com/adobe/granite/workflow/ offloading •  New “DAM Update Asset Offloading” workflow •  Supports clustered and non clustered configurations
  • 68.
    68 TAKE AWAY Potential Future Discovery OffloadingWorkflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
  • 69.
    69 TAKE AWAY Potential Future Discovery OffloadingWorkflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing New OSGi Specification • Distributed Eventing • Cloud Computing
  • 70.
    70 TAKE AWAY Potential Future Discovery OffloadingWorkflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing Job Distribution • Improved load balancing • Pull based distribution
  • 71.
    71 TAKE AWAY Potential Future Discovery OffloadingWorkflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing Offloading • As part of job distribution • Even simpler setup
  • 72.
    72 TAKE AWAY Questions? Discovery Offloading WorkflowDistribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
  • 73.