0
FROM EVENTING TO JOB PROCESSING
Carsten Ziegeler | Adobe Research Switzerland
1
2
•  RnD Team at Adobe Research Switzerland
•  Co-founder Adobe Granite
•  OSGi Core Platform and Enterprise Expert Groups...
3
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distri...
4
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distri...
5
OSGI EVENT ADMIN
Publish / Subscribe Model
OSGi Event Admin
Component
A
publish
deliver
Component
X
Component
Y
6
•  OSGi event is a data object with
•  Topic (hierarchical namespace)
•  Properties (key-value-pairs)
•  Page Event
•  T...
7
•  Publisher creates event object
•  Sends event through EventAdmin service
•  Either sync or async delivery
•  Subscrib...
8
•  Immediate delivery to available subscribers
•  No guarantee of delivery
•  No distributed delivery
OSGI EVENT ADMIN
P...
9
•  Immediate delivery to available subscribers
•  No guarantee of delivery
•  No distributed delivery
OSGI EVENT ADMIN
P...
10
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distr...
11
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distr...
12
INSTALLATION SCENARIOS
Clustered CRX
Single
Instance
CRX
Instance
1
Instance
2
Instance
3
13
Instance: Unique Id (Sling ID)
TOPOLOGIES I
Apache Sling Discovery
Clustered CRXCRX
ID : A ID : X ID : 42ID : 1
Single
...
14
•  Instance: Unique Id (Sling ID)
•  Cluster: Unique Id and leader
TOPOLOGIES I
Apache Sling Discovery
Cluster 99Cluste...
15
•  Topology: Set of clusters
TOPOLOGIES I
Apache Sling Discovery
Cluster 99Cluster 35
Clustered CRXCRX
ID : A ID : X ID...
16
TOPOLOGIES I
Apache Sling Discovery
•  Instance: Unique Id (Sling ID)
•  Cluster: Unique Id and leader
•  Topology: Set...
Cluster 99
17
•  Instance
•  Sling ID
•  Optional: Name and description
•  Belongs to a cluster
•  Might be the cluster le...
18
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distr...
19
•  Job : Guaranteed processing, exactly once
•  Exactly one job consumer
•  Started by client code, e.g. for replicatio...
20
•  Sling Job Manager handles and distributes jobs
•  Delivers job to a job consumer…
•  …and waits for response
•  Retr...
21
STARTING / PROCESSING A JOB I
Apache Sling Job Handling
public interface JobConsumer {	
	
String PROPERTY_TOPICS = "job...
22
@Component	
@Service(value={JobConsumer.class})	
@Property(name=JobConsumer.PROPERTY_TOPICS, value="org/apache/sling/jo...
23
•  New jobs are immediately persisted (resource tree / repository)
•  Jobs are “pushed” to the processing instance
•  P...
24
•  Queue is configurable
•  Queue is started on demand in own thread
•  And stopped if unused for some time
•  Types
• ...
25
•  Job Manager Configuration = Main Queue Configuration
•  Maximum parallel jobs (15)
•  Retries (10)
•  Retry Delay
• ...
26
MONITORING – WEB CONSOLE
Apache Sling Job Handling
27
MONITORING – WEB CONSOLE
Apache Sling Job Handling
28
•  Each instance determines enabled job topics
•  Derived from Job Consumers (new API required)
•  Can be whitelisted/b...
29
•  Scalability in AEM:
•  DAM Ingestion
•  Non-Clustered installation requirement
•  The term Offloading:
•  In AEM use...
30
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distr...
31
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distr...
32
SLING JOB DISTRIBUTION
Topology
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
...
33
SLING JOB DISTRIBUTION
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manag...
34
SLING JOB DISTRIBUTION
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manag...
35
SLING JOB DISTRIBUTION
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manag...
36
SLING JOB DISTRIBUTION
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manag...
37
•  Detects offloading jobs
•  Transport of job and job
payload between origin
and target instance
•  Uses replication f...
38
•  Replication agents are created automatically
•  Uses naming convention
•  Needs manual adjustments (replication user...
39
•  Additional properties required
•  Offloading job input
•  Property: OffloadingJobProperties. INPUT_PAYLOAD (offloadi...
40
•  Configures Job Consumers
•  Configures the topic white/black listing properties of each instance
•  What jobs to exe...
41
OFFLOADING BROWSER
UI
42
OFFLOADING BROWSER
UI
43
OFFLOADING BROWSER
UI
44
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distr...
45
•  New JobConsumer
•  Class: WorkflowOffloadingJobConsumer
•  Topic: com/adobe/granite/workflow/offloading
•  Can launc...
46
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public c...
47
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public c...
48
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public c...
49
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public c...
50
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public c...
51
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public c...
52
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distr...
53
•  Default ingestion workflow: “DAM Update Asset”
•  Load is put on the instance where the workflow is started, usually...
54
@Component
@Service
public class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference
private JobMana...
55
@Component
@Service
public class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference
private JobMana...
56
@Component
@Service
public class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference
private JobMana...
57
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset =...
58
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset =...
59
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset =...
60
@Component
@Service
public class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference
private JobMana...
61
•  Choose a job topic
•  Create JobConsumer component and register with topic chosen
•  To create a new job use new Job...
62
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
63
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
org.apache.sling....
64
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
org.apache.sling....
65
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
com.adobe.granite...
66
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
com.adobe.granite...
67
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
cq-dam-core
•  Ma...
68
TAKE AWAY
Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
...
69
TAKE AWAY
Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
...
70
TAKE AWAY
Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
...
71
TAKE AWAY
Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
...
72
TAKE AWAY
Questions?
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix ...
73
Upcoming SlideShare
Loading in...5
×

Adobe AEM - From Eventing to Job Processing

4,849

Published on

Evolve13 presentation from OSGi Event Admin, Sling Discovery, Sling Job Handling to Granite Offloading and AEM Dam Ingestion

Published in: Technology, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,849
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
124
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Transcript of "Adobe AEM - From Eventing to Job Processing"

  1. 1. FROM EVENTING TO JOB PROCESSING Carsten Ziegeler | Adobe Research Switzerland 1
  2. 2. 2 •  RnD Team at Adobe Research Switzerland •  Co-founder Adobe Granite •  OSGi Core Platform and Enterprise Expert Groups •  Member of the ASF •  Current PMC Chair of Apache Sling •  Apache Sling, Felix, ACE •  Conference Speaker •  Technical Reviewer •  Article/Book Author cziegele@adobe.com @cziegeler ABOUT
  3. 3. 3 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
  4. 4. 4 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
  5. 5. 5 OSGI EVENT ADMIN Publish / Subscribe Model OSGi Event Admin Component A publish deliver Component X Component Y
  6. 6. 6 •  OSGi event is a data object with •  Topic (hierarchical namespace) •  Properties (key-value-pairs) •  Page Event •  Topic: com/day/cq/wcm/core/page •  Properties: path, change type (add/remove/edit) etc. OSGI EVENT ADMIN Publish / Subscribe Model
  7. 7. 7 •  Publisher creates event object •  Sends event through EventAdmin service •  Either sync or async delivery •  Subscriber is an OSGi service (EventHandler) •  Service registration properties •  Interested topic(s) •  com/day/cq/wcm/core/* •  Additional filters (optional) •  (type="add“) OSGI EVENT ADMIN Publish / Subscribe Model
  8. 8. 8 •  Immediate delivery to available subscribers •  No guarantee of delivery •  No distributed delivery OSGI EVENT ADMIN Publish / Subscribe Model
  9. 9. 9 •  Immediate delivery to available subscribers •  No guarantee of delivery •  No distributed delivery OSGI EVENT ADMIN Publish / Subscribe Model DiscoverySling Job Distribution
  10. 10. 10 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  11. 11. 11 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution New in 5.6.1
  12. 12. 12 INSTALLATION SCENARIOS Clustered CRX Single Instance CRX Instance 1 Instance 2 Instance 3
  13. 13. 13 Instance: Unique Id (Sling ID) TOPOLOGIES I Apache Sling Discovery Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3
  14. 14. 14 •  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader TOPOLOGIES I Apache Sling Discovery Cluster 99Cluster 35 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Leader
  15. 15. 15 •  Topology: Set of clusters TOPOLOGIES I Apache Sling Discovery Cluster 99Cluster 35 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Leader TopologyTopology •  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader
  16. 16. 16 TOPOLOGIES I Apache Sling Discovery •  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader •  Topology: Set of clusters Cluster 99 Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3 Leader Topology Cluster 35 Leader
  17. 17. Cluster 99 17 •  Instance •  Sling ID •  Optional: Name and description •  Belongs to a cluster •  Might be the cluster leader •  Additional properties which are distributed •  Extensible through own services (PropertyProvider) •  E.g. data center, region or enabled job topics •  Cluster •  Elects (stable) leader •  Stable instance ordering •  TopologyEventListener: receives events on topology changes TOPOLOGIES II Apache Sling Discovery ID : 42 Instance 3 Topology Leader
  18. 18. 18 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  19. 19. 19 •  Job : Guaranteed processing, exactly once •  Exactly one job consumer •  Started by client code, e.g. for replication, workflow... •  Job topic •  Payload is a serializable map JOB HANDLING I Apache Sling Job Handling
  20. 20. 20 •  Sling Job Manager handles and distributes jobs •  Delivers job to a job consumer… •  …and waits for response •  Retry and failover •  Notification listeners (fail, retry, success) JOB HANDLING I Apache Sling Job Handling
  21. 21. 21 STARTING / PROCESSING A JOB I Apache Sling Job Handling public interface JobConsumer { String PROPERTY_TOPICS = "job.topics"; enum JobResult { OK, FAILED, CANCEL, ASYNC } JobResult process(Job job); } public interface JobManager { Job addJob(String topic, String optionalName, Map<String, Object> properties); … } Starting a job Processing a job Note: Starting/processing of jobs through Event Admin is deprecated but still supported New in 5.6.1
  22. 22. 22 @Component @Service(value={JobConsumer.class}) @Property(name=JobConsumer.PROPERTY_TOPICS, value="org/apache/sling/jobs/backup") public class BackupJobConsumer implements JobConsumer { @Override public JobResult process(final Job job) { // do backup return JobResult.OK; } } STARTING / PROCESSING A JOB II Apache Sling Job Handling New in 5.6.1
  23. 23. 23 •  New jobs are immediately persisted (resource tree / repository) •  Jobs are “pushed” to the processing instance •  Processing instances use different queues •  Associated with job topic(s) •  Main queue •  0..n custom queues •  For example: replication agent queue or workflow queue JOB HANDLING II Apache Sling Job Handling
  24. 24. 24 •  Queue is configurable •  Queue is started on demand in own thread •  And stopped if unused for some time •  Types •  Ordered queue (eg replication) •  Parallel queues: Plain and Topic Round Robin (eg workflow) •  Limit for parallel threads per queue •  Number of retries (-1 = endless) •  Retry delay •  Thread priority JOB QUEUE Apache Sling Job Handling
  25. 25. 25 •  Job Manager Configuration = Main Queue Configuration •  Maximum parallel jobs (15) •  Retries (10) •  Retry Delay •  Eventing Thread Pool Configuration •  Used by all queues •  Pool size (35) = Maximum parallel jobs for a single instance ADDITIONAL CONFIGURATIONS Apache Sling Job Handling
  26. 26. 26 MONITORING – WEB CONSOLE Apache Sling Job Handling
  27. 27. 27 MONITORING – WEB CONSOLE Apache Sling Job Handling
  28. 28. 28 •  Each instance determines enabled job topics •  Derived from Job Consumers (new API required) •  Can be whitelisted/blacklisted (in Job Consumer Manager) •  Announced through Topology •  Job Distribution depends on enabled job topics and queue type •  Potential set of instances derived from topology (enabled job topics) •  Ordered : processing on leader only, one job after the other •  Parallel: Round robin distribution on all potential instances •  Local cluster instances have preference •  Failover •  Instance crash: leader redistributes jobs to available instances •  Leader change taken into account •  On enabled job topics changes: potential redistribution JOB DISTRIBUTION Apache Sling Job Handling
  29. 29. 29 •  Scalability in AEM: •  DAM Ingestion •  Non-Clustered installation requirement •  The term Offloading: •  In AEM used for all things job distribution and topology in clustered and non- clustered installations, e.g. ‘Offloading Browser’ •  More technically it’s ‘only’ a little add-on in Granite to Sling Job Distribution for handling non-clustered installations WHY OFFLOADING
  30. 30. 30 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  31. 31. 31 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  32. 32. 32 SLING JOB DISTRIBUTION Topology Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job Manager Job Consumer Topic: A Job Consumer Topic: B Job Consumer Topic: C
  33. 33. 33 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerA Job Consumer Topic: A Job Consumer Topic: B A:2 Job Consumer Topic: CJob Topology
  34. 34. 34 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerB Job Consumer Topic: A Job Consumer Topic: B B:3 Job Consumer Topic: CJob Topology
  35. 35. 35 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerC Job Consumer Topic: A Job Consumer Topic: B C:4 Job Consumer Topic: CJob ? Topology
  36. 36. 36 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4 Job ManagerC Job Consumer Topic: A Job Consumer Topic: B C:4 Job Consumer Topic: CJob Offloading Framework Topology
  37. 37. 37 •  Detects offloading jobs •  Transport of job and job payload between origin and target instance •  Uses replication for the transport •  No distribution of jobs •  No execution of jobs OFFLOADING FRAMEWORK Overview Instance Sling ID: 1 Job Manager Offloading Job Manager C:4 Instance Sling ID: 4 Job Manager Job Consumer Topic: C Offloading Job Manager C:4 C Job Replication 1 2 3 5 4 6 7 Topology
  38. 38. 38 •  Replication agents are created automatically •  Uses naming convention •  Needs manual adjustments (replication user) OFFLOADING FRAMEWORK Replication Worker Sling ID: 4 Author Sling ID: 1 offloading_4 (outgoing agent) offloading_reverse_4 (reverse agent) offloading_outbox (outbox agent)
  39. 39. 39 •  Additional properties required •  Offloading job input •  Property: OffloadingJobProperties. INPUT_PAYLOAD (offloading.input.payload) •  Offloading job output •  Property: OffloadingJobProperties. OUTPUT_PAYLOAD (offloading.output.payload) •  Takes comma separated list of paths •  Used to build the replication package •  Job sender needs to set these properties for offloading to work! OFFLOADING FRAMEWORK Job Payload
  40. 40. 40 •  Configures Job Consumers •  Configures the topic white/black listing properties of each instance •  What jobs to execute on what instance •  Configures the distribution •  Configuration applies for both, clustered and non-clustered installations OFFLOADING BROWSER UI
  41. 41. 41 OFFLOADING BROWSER UI
  42. 42. 42 OFFLOADING BROWSER UI
  43. 43. 43 OFFLOADING BROWSER UI
  44. 44. 44 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  45. 45. 45 •  New JobConsumer •  Class: WorkflowOffloadingJobConsumer •  Topic: com/adobe/granite/workflow/offloading •  Can launch new workflows •  Expects the workflow model on the job payload •  Expects the workflow payload on the job payload •  For use with clustered and non-clustered installations WORKFLOW DISTRIBUTION
  46. 46. 46 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified)
  47. 47. 47 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Define the job topic
  48. 48. 48 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Create service component •  Must register with topic •  Implement new JobConsumer interface
  49. 49. 49 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Access job properties (payload) •  Read workflow model and payload from job properties
  50. 50. 50 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Workflow specific •  Use workflow API to start workflow for the given model and payload
  51. 51. 51 @Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer (Simplified) •  Use JobResult enumeration to report back the job status
  52. 52. 52 OVERVIEW Where to fit in the stack Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  53. 53. 53 •  Default ingestion workflow: “DAM Update Asset” •  Load is put on the instance where the workflow is started, usually the author •  New ingestion workflow: “DAM Update Asset Offloading” •  Needs to be manually enabled by changing the workflow launcher •  New workflow model with a single step: AssetOffloadingProcess •  Uses WorkflowExternalProcess API •  Creates a new job on topic: com/adobe/granite/workflow/offloading •  Allows distributing the default ingestion workflow •  Load is put on the instance where the job is distributed to •  Can be used to distribute in clustered and non-clustered installations DAM INGESTION
  54. 54. 54 @Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step)
  55. 55. 55 @Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Create service component •  Implement WorkflowExternalProcess interface •  Reference JobManager service
  56. 56. 56 @Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  DAM and Workflow specific •  Resolve to Asset •  Read model from meta data •  Read workflow payload from Asset path
  57. 57. 57 public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  ValueMap for job properties •  Put model and payload on job properties •  Used by the JobConsumer
  58. 58. 58 public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Build offloading payload properties •  Comma separated list of paths •  Put them on the job payload as well •  Only used for non-clustered distribution
  59. 59. 59 public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); } DAM INGESTION Create Job (from workflow step) •  Create job using JobManager service •  Use topic from job consumer •  Put job payload properties •  Return the jobId as the workflow process id (workflow specific)
  60. 60. 60 @Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ …. } public boolean hasFinished(Serializable externalProcessId, ..){ // returns null, if job is finished Job offloadingJob = jobManager.getJobById((String) externalProcessId); return offloadingJob == null; } } DAM INGESTION Create Job (from workflow step) •  Workflow API specific callback •  Process id = jobId, from execute() •  Query job by jobId •  Workflow step finished when job is finished
  61. 61. 61 •  Choose a job topic •  Create JobConsumer component and register with topic chosen •  To create a new job use new JobManager.addJob() API with the topic chosen and the job payload •  Add offloading payload to job payload •  Bundle and deploy JobConsumer on topology instances •  Enable/Disable the new topic on the instances, using Offloading Browser DEVELOPMENT - RECIPE
  62. 62. 62 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution
  63. 63. 63 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution org.apache.sling.discovery.api •  Discovers topology •  Cross-Cluster detection •  Foundation for job distribution
  64. 64. 64 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution org.apache.sling.event •  Uses Sling Discovery •  New JobConsumer API and job topics •  New JobManager API for creating new distributed jobs •  Distributes jobs based on available job topics and job queue configuration. •  Distributes in the whole topology, including clustered and non- clustered instances •  Can execute cluster local jobs only
  65. 65. 65 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution com.adobe.granite.offloading.core •  Builds on top of Sling Distributed Jobs •  Does not perform distribution •  Detects jobs distributed to non-clustered instances •  Transports the jobs and payload to non-clustered instances •  Uses replication for transport •  Does not execute jobs
  66. 66. 66 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution com.adobe.granite.workflow.core •  Defines a job consumer to distribute the execution of whole workflows •  Defines topic com/adobe/granite/workflow/ offloading •  Implements WorkflowOffloadingJobConsumer component •  Supports clustered and non clustered installations
  67. 67. 67 TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution cq-dam-core •  Makes use of com/adobe/granite/workflow/ offloading topic from Workflow Distribution •  New workflow step (external step) that creates a new job on topic com/adobe/granite/workflow/ offloading •  New “DAM Update Asset Offloading” workflow •  Supports clustered and non clustered configurations
  68. 68. 68 TAKE AWAY Potential Future Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
  69. 69. 69 TAKE AWAY Potential Future Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing New OSGi Specification • Distributed Eventing • Cloud Computing
  70. 70. 70 TAKE AWAY Potential Future Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing Job Distribution • Improved load balancing • Pull based distribution
  71. 71. 71 TAKE AWAY Potential Future Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing Offloading • As part of job distribution • Even simpler setup
  72. 72. 72 TAKE AWAY Questions? Discovery Offloading Workflow Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
  73. 73. 73
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×