Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Architectural changes in the repo in 6.1 and beyond

160 views

Published on

Alfresco DevCon 2019 presentation covering all changes to the ACS repository in 6.1 and an outlook to the future beyond 6.1

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Architectural changes in the repo in 6.1 and beyond

  1. 1. Architectural changes in the Content Repository in 6.1 and beyond Alex Mukha Stefan Kopf
  2. 2. Plan to cover: ● Where we come from ● Monitoring (Micrometer.io / Prometheus) ● ACS and ActiveMQ ● New policies and behaviours with events ● New Transform Service ● New Alfresco Identity Service ● Deployment in AWS ● Removed and deprecated features ● Where we are going to (Unified) Roadmap
  3. 3. Learn. Connect. Collaborate. Safe Harbour Statement • The information contained in these presentations is intended to outline general product direction and should not be relied upon in making purchasing decisions. • The content is for informational purposes only and may not be incorporated into any contract. • The information presented is not a commitment, promise, or legal obligation to deliver any material, code or functionality. • Any references to the development, release, and timing of any features or functionality described for these products remains at Alfresco's sole discretion • Product capabilities, timeframes and features are subject to change and should not be viewed as Alfresco commitments.
  4. 4. Learn. Connect. Collaborate. Where we come from 1.0 2005 Spring Explorer DB Lucene
  5. 5. Learn. Connect. Collaborate. Where we come from 1.0 2005 Spring Explorer DB Lucene 2.0 2005 Spring Explorer DB Lucene
  6. 6. Learn. Connect. Collaborate. Where we come from 1.0 Spring Explorer DB Lucene 2.0 Spring Explorer DB Lucene 3.0 2008 Spring Share DB Lucene
  7. 7. Learn. Connect. Collaborate. Where we come from 1.0 Spring Explorer DB Lucene 2.0 Spring Explorer DB Lucene 3.0 2008 Spring Share DB 4.0 2011 Spring DB SOLR Lucene Share
  8. 8. Learn. Connect. Collaborate. Composition Today Based on • Spring Context • Policy/Behaviour framework Guaranteed to be • in-process • in-context • in-transaction • synchronous
  9. 9. Learn. Connect. Collaborate. What we need • out-of-process • out-of-context • out-of-transaction • asynchronous
  10. 10. Learn. Connect. Collaborate. Evolution Build MeasureLearn
  11. 11. Monitoring in ACS
  12. 12. Learn. Connect. Collaborate. Micrometer New endpoint in ACS 6.1(enterprise-only): /alfresco/s/prometheus A Prometheus compatible scraping (read only webscript). The metrics collection is powered by the micrometer library (Vendor-neutral application metrics facade) https://micrometer.io/ https://prometheus.io/ https://grafana.com/
  13. 13. Learn. Connect. Collaborate. Micrometer Available metrics Basic JVM metrics: CPU, memory, GC, Threads and etc. Configuration: metrics.jvmMetricsReporter.enabled=true Grafana dashboard: https://grafana.com/dashboards/470
  14. 14. Learn. Connect. Collaborate. Micrometer Available metrics Rest API call metrics: ● 1 timer “restapi_execution_time” with 2 labels: ○ Type of request (GET, POST,PUT,DELETE) ○ The path to the webscript (e.g: /alfresco/api/- default-/public/alfresco/versions/1/people) if path is enabled Configuration: metrics.restMetricsReporter.enabled=true metrics.restMetricsReporter.path.enabled=false Grafana sample dashboard: https://github.com/Alfresco/acs- packaging/tree/master/docs/micrometer
  15. 15. Learn. Connect. Collaborate. Micrometer Available metrics DB layer metrics: ● 1 timer “queries_execution_time” with 2 labels ○ Type of query (Select, Update, Insert, Delete) ○ The mybatis query ID (e.g: alfresco.node.select_ChildAssocOfParentByName) ● 2 gauges: num_connections_active and num_connections_idle Configuration: metrics.dbMetricsReporter.enabled=true metrics.dbMetricsReporter.query.enabled=true metrics.dbMetricsReporter.query.statements.enabled=false Grafana sample dashboard: https://github.com/Alfresco/acs- packaging/tree/master/docs/micrometer
  16. 16. Learn. Connect. Collaborate. Micrometer Available metrics Tomcat metrics: ● servlet_request timer (number of responses with particular response times) Configuration: metrics.tomcatMetricsReporter.enabled=true Grafana sample dashboard: https://github.com/Alfresco/acs- packaging/tree/master/docs/micrometer
  17. 17. Learn. Connect. Collaborate. Micrometer Future plans ● Integrate monitoring into Helm charts and CloudWatch in AWS ● More metrics: ○ Audit ○ Renditions ○ Hazelcast caches ○ Logged in users ○ etc..
  18. 18. ACS and ActiveMQ
  19. 19. Learn. Connect. Collaborate. ACS and ActiveMQ • Leading Open Source messaging platform • Reliable, high performance messaging • Fully supports JMS 1.1 and J2EE 1.4 with support for transient, persistent, transactional and XA messaging • Supports a variety of Cross Language Clients and Protocols from Java, C, C++, C#, Ruby, Perl, Python, PHP – OpenWire – Stomp – AMQP – MQTT
  20. 20. Learn. Connect. Collaborate. ACS and ActiveMQ • Alfresco Content Services (ACS) used ActiveMQ for message queuing with various products, for example Alfresco Media Management or Alfresco Sync Service (Desktop Sync) • Starting from ACS 6.1, ActiveMQ is a mandatory requirement for the ACS Repository and its new Raw Events module/capability
  21. 21. Learn. Connect. Collaborate. ACS and ActiveMQ Connect ACS to ActiveMQ: Define the location of ActiveMQ in your alfresco- global.properties file: messaging.broker.url=failover:(tcp://server:61616)?timeout =3000 where server is the host name of the server where ActiveMQ is installed. Notes: • When you set up ActiveMQ, the Alfresco Content Services messaging subsystem is set to start up automatically. • Any changes to broker URL requires ACS restart to apply the changes.
  22. 22. Learn. Connect. Collaborate. ACS and ActiveMQ Ways to deploy ActiveMQ: ● Containerized deployment* https://github.com/Alfresco/acs-deployment ○ docker-compose ○ Helm / Kubernetes ● Using distribution zip http://activemq.apache.org/download.html ● ...or your favourite managed offering, e.g. AmazonMQ * Conteinerized deployments of ACS use Alfresco ActiveMQ image: https://github.com/Alfresco/alfresco-docker-activemq Can be also deployed via Helm chart: https://github.com/Alfresco/alfresco-activemq-deployment
  23. 23. Transactional Events
  24. 24. Learn. Connect. Collaborate. Two Generals’ Problem Possible inconsistencies: • DB succeeds, but MB fails • MB succeeds, but DB fails “Two Generals’ Problem” In Enterprise Software: • Distributed Transactions (XA)
  25. 25. Learn. Connect. Collaborate. Distributed Transactions Severe additional complexity: • Transaction manager required Additional constraints • All data sources need to support XA ⇒ In 6.1: NO Distributed Transactions
  26. 26. Learn. Connect. Collaborate. Residual Error False Positive: Messages have been sent out, but DB transaction is rolled back False Negative: DB transaction succeeded, but Message Broker fails TransactionAwareEventProducer • Implements 2nd approach • Collects messages during transaction • Sends all at once after commit
  27. 27. Policies and Behaviours
  28. 28. Learn. Connect. Collaborate. Recap: What are Policies And Behaviours Policy / Behaviour framework: Implementation of the observer pattern in Alfresco Policy: Advertisement of an Extension Point provided by a component Behaviour: Subscription of a Policy, i.e. code that gets executed when a Policy is triggered
  29. 29. Learn. Connect. Collaborate. EventBehaviour Existing Behaviours: • JavaBehaviour • ScriptBehaviour New in 6.1: • EventBehaviour
  30. 30. Learn. Connect. Collaborate. EventBehaviour Example @Autowired private TransactionAwareEventProducer taep; @Autowired private PolicyComponent policyComponent; @Override public void configure() throws Exception { EventBehaviour eventBehaviour = new EventBehaviour(taep, sourceQueue, this, "createEvent", Behaviour.NotificationFrequency.EVERY_EVENT); policyComponent.bindClassBehaviour( ContentServicePolicies.OnContentUpdatePolicy.QNAME, RenditionModel.ASPECT_RENDITIONED, eventBehaviour); ... } public OnContentUpdatePolicyEvent createEvent(NodeRef node, boolean newContent) { OnContentUpdatePolicyEvent event = new OnContentUpdatePolicyEvent(); event.setNodeRef(node.toString()); event.setNewContent(newContent); ... return event; }
  31. 31. New Transform Service
  32. 32. Learn. Connect. Collaborate. Transform Service Bottleneck… ● Document transformations are used heavily within ACS. All content imported into ACS generates renditions, for example thumbnails and previews ● Most of the transformations were synchronous, consuming resources of ACS JVM or underlying OS. ● Security risk of running them on the same machine as ACS ● Transformations have been a limiting factor when ingesting large amounts of content, this means that, in high load situations, transformations had to be disabled
  33. 33. Learn. Connect. Collaborate. Transform Service Overview New Transform Service features: ● Microservice architecture ○ Independent scalability ○ Cost effective ○ High throughput ● Co-exists with existing transformations in ACS ● Containerised deployments only ● ACS 6.1+ ● Enterprise-only ● Supports a smaller subset of available transformations comparing to 6.0
  34. 34. Learn. Connect. Collaborate. ACS Async Rendition Service V1 REST APIV0 REST API Rendition Service 2 Rendition Service Thumbnail Service Action Service Content Service Shared File Store <<Docker>> Content Update Transform Service Router <<Docker>> Transformers <<Docker>> Transformers <<Docker>> Transformers <<Docker>> Transformers <<Docker>> Transform Response Transform Request TransformerTransformerLocal Transformers Solr 6 101010 101010 101010
  35. 35. Learn. Connect. Collaborate. Transform Service Configuration local.transform.service.enabled=true transform.service.enabled=true Rendition 2 Definition Bean <bean id="renditionDefinition2DocLib" class="org.alfresco.repo.rendition2.RenditionDefinition2Impl"> <constructor-arg name="renditionName" value="doclib"/> <constructor-arg name="targetMimetype" value="image/png"/> <constructor-arg name="transformOptions"> <map> <entry key="resizeWidth" value="100"/> <entry key="resizeHeight" value="100"/> <entry key="allowEnlargement" value="false" /> <entry key="maintainAspectRatio" value="true"/> <entry key="thumbnail" value="true"/> <entry key="timeout" value="${system.thumbnail.definition.default.timeoutMs}" /> </map> </constructor-arg> <constructor-arg name="registry" ref="renditionDefinitionRegistry2"/> </bean> ImageMagick ‘flat’ Transform Options alphaRemove autoOrient startPage endPage cropGravity cropWidth cropHeight cropPercentage cropXOffset cropYOffset thumbnail resizeHeight resizeWidth resizePercentage allowEnlargement maintainAspectRatio
  36. 36. Learn. Connect. Collaborate. Transform Service Future plans • New Transform Service will increase the number of supported transformations. – Integrate with Search Services (to text conversion) – Extensibility – AI and more.. • The synchronous renditions are deprecated and will be removed from content repository completely • Metadata extraction is moved out from content repository
  37. 37. Alfresco Identity Service
  38. 38. Learn. Connect. Collaborate. Alfresco Identity Service Overview Alfresco Identity Service provides authentication and proof of identity, in the form of a standard token (JSON Web Token https://jwt.io ), understood by Digital Business Platform components: ○ ACS 6.1 REST APIs (v1) and CMIS ○ APS 1.10 REST APIs ○ AGS 3.1 REST API (v1) ○ ADF 2.6 Alfresco Identity Service is based on Keycloak and supports: ○ LDAP (OpenLDAP) ○ SAML 2.0 (PingFederate) ○ OpenID Connect
  39. 39. Learn. Connect. Collaborate. Alfresco Identity Service Deployment ● The deployment artifact is the alfresco-identity- service Helm Chart ○ http://kubernetes- charts.alfresco.com/stable/alfresco-identity- service-1.0.0.tgz ● Deployed, by default, as part of the alfresco- infrastructure Helm Chart ○ http://kubernetes- charts.alfresco.com/stable/alfresco- infrastructure-3.0.0.tgz ● Is NOT part of ACS 6.1 Helm charts Contains: ● Keycloak with Alfresco Theme ● PostgreSQL ● Ingress
  40. 40. Learn. Connect. Collaborate. Alfresco Identity Service support in ACS Represented by authentication subsystem: ● alfresco/subsystems/Authentication/identity-service Based on Keycloak libs: • https://github.com/keycloak/keycloak (Supports JWT standard) Users and groups are still managed in ACS: • Can auto-create users • Works with LDAP sync (recommended) JWT and basic to access APIs in ACS AIS and ACS are (almost) pre-configured for dev/test (test real is present in AIS)
  41. 41. Learn. Connect. Collaborate. Alfresco Identity Service Configuration Configure ACS alfresco-global.properties: authentication.chain=ais:identity-service,... identity-service.authentication.enabled=true identity-service.authentication.validation.failure.silent=true identity-service.authentication.defaultAdministratorUserNames=admin identity-service.authentication.allowGuestLogin=true identity-service.authentication.enable-username-password-authentication=true identity-service.auth-server-url=http://localhost:8180/auth identity-service.realm=alfresco identity-service.ssl-required=none identity-service.resource=alfresco identity-service.public-client=true Configure AIS: https://github.com/Alfresco/alfresco-identity-service/tree/master/docs
  42. 42. Learn. Connect. Collaborate. Alfresco Identity Service Simplified sequence diagram: Client AIS ACS Request JWT Full example: https://github.com/Alfresco/alfresco-identity- service/blob/master/docs/resource/sequence/high-level-ldap-auth- sequence.png JWT: kfYclR.. GET /some-api Authorization: Bearer kfYcIR... Get signing keys Verify JWT Process request Response to client
  43. 43. Learn. Connect. Collaborate. Alfresco Identity Service Future plans • User and group management will be moved from ACS into AIS • Authentication chain will be deprecated and removed. List of supported authentication types will be extended in AIS. • (Maybe) Move management of ACLs to AIS
  44. 44. Deployment in AWS
  45. 45. Learn. Connect. Collaborate. HELM Charts Since 6.0 • New additional deployment option • Existing deployment options still available • Codification of documentation and recommended best practices • Reference deployment: starting point for your own environment description • w/o params: Brings everything in a container • Parameterize to use your infrastructure
  46. 46. Learn. Connect. Collaborate. 6.1 AWS Deployment • Set of CloudFormation templates plus scripts and tooling • Sets up: – VPC + Bastion – EKS + Worker Nodes – S3 incl replication – Aurora DB – AmazonMQ – IAM permissions
  47. 47. Learn. Connect. Collaborate. AWS Architecture
  48. 48. Learn. Connect. Collaborate. ACS on AWS Source code locations ● CFN templates + AWS alterations https://github.com/Alfresco/acs-deployment-aws ● ACS Helm Charts https://github.com/Alfresco/acs-deployment ● Charts Repository https://github.com/Alfresco/charts
  49. 49. Removed and deprecated features
  50. 50. Learn. Connect. Collaborate. Removed and deprecated features ● Removed (configuration and code) ○ CIFS ○ NTLM ● Deprecated (will be removed in future versions) ○ Synchronous transformations ○ Cloud Sync
  51. 51. Learn. Connect. Collaborate. Beyond • Extract common components • Identity Service – Move profiles to AIS – Move user/group info to AIS – Perform authentication outside of ACS – All requests carry JWT token • Move to Async, candidates – Auditing – Metadata extraction – Search indexing – Rules / Actions • Deployment options for other cloud provider
  52. 52. Thanks! Reach us on email: alex.mukha@alfresco.com stefan.kopf@alfresco.com GitHub: https://github.com/Alfresco JIRA: https://issues.alfresco.com
  53. 53. Learn. Connect. Collaborate. New Raw Events Motivation (the “why?”) • decoupling • asynchronous processing • integration • fault-tolerance • spikability How? • An evolutionary path to asynchronous processing based upon policies / behaviours • The event model enables us to execute code when an event happens in the system. These events are referred to as “policies”. With “behaviours”, you can register code that is executed when these events occur.
  54. 54. Learn. Connect. Collaborate. New Raw Events Motivation (the “why?”) • decoupling • asynchronous processing • integration • fault-tolerance • spikability How? • An evolutionary path to asynchronous processing based upon policies / behaviours • The event model enables us to execute code when an event happens in the system. These events are referred to as “policies”. With “behaviours”, you can register code that is executed when these events occur.
  55. 55. Learn. Connect. Collaborate. New Raw Events
  56. 56. Learn. Connect. Collaborate. New Raw Events The basic data which is captured for each event: • id • type • authenticatedUser • executingUser • timestamp • schema
  57. 57. Learn. Connect. Collaborate. New Raw Events EventProducers AbstractEventProducer • Abstract helper to send events to an endpoint. The AbstractEventProducer acts as a wrapper that provides marshalling for a Camel ProducerTemplate. • A client has the option of creating an event producer without supplying an endpoint. In this case, an endpoint must be provided with every send operation. • A client also has the option to provide an ObjectMapper that will be used to marshal basic POJOs to JSON, before sending the event. TransactionAwareEventProducer • Events are scheduled to be sent in post-commit phase.
  58. 58. Learn. Connect. Collaborate. New Raw Events EventBehaviour • Event based Behaviour. • A client uses an EventBehaviour to bind a send event behaviour to a Class-level Policy. • The endpoint uri can be a queue or a topic. – e.g: jms:acs-repo-rendition- events?jmsMessageType=Text • The event behavior delegates the generation of the event to a method pointer. The pointer is represented by an instance object and method name.
  59. 59. Learn. Connect. Collaborate. New Raw Events Event Behaviour Notification Frequency • EVERY_EVENT: The event handler is then just executed wherever it is being invoked in the code. The name of this notification frequency implies that the event handler will be called multiple times, but that is not the case. • TRANSACTION_COMMIT: This is the default, if the notification frequency is not specified. The event handler is queued and invoked at the end of the transaction, after it has been committed. A proxy around the event handler manages the queuing. • FIRST_EVENT: The event handler is invoked just after the transaction is started. A proxy around the event handler manages this.

×