Architectural changes in the repo in 6.1 and beyond

Architectural changes in the Content Repository in 6.1
and beyond
Alex Mukha
Stefan Kopf

Plan to cover:
● Where we come from
● Monitoring (Micrometer.io /
Prometheus)
● ACS and ActiveMQ
● New policies and behaviours with
events
● New Transform Service
● New Alfresco Identity Service
● Deployment in AWS
● Removed and deprecated
features
● Where we are going to
(Unified) Roadmap

Learn. Connect. Collaborate.
Safe
Harbour
Statement
• The information contained in these presentations is
intended to outline general product direction and should
not be relied upon in making purchasing decisions.
• The content is for informational purposes only and may
not be incorporated into any contract.
• The information presented is not a commitment, promise,
or legal obligation to deliver any material, code or
functionality.
• Any references to the development, release, and timing of
any features or functionality described for these products
remains at Alfresco's sole discretion
• Product capabilities, timeframes and features are subject
to change and should not be viewed as Alfresco
commitments.

Where we come from
1.0
2005
Spring
Explorer
DB
Lucene

Where we come from
1.0
2005
Spring
Explorer
DB
Lucene
2.0
2005
Spring
Explorer
DB
Lucene

Where we come from
1.0
Spring
Explorer
DB
Lucene
2.0
Spring
Explorer
DB
Lucene
3.0
2008
Spring
Share
DB
Lucene

Where we come from
1.0
Spring
Explorer
DB
Lucene
2.0
Spring
Explorer
DB
Lucene
3.0
2008
Spring
Share
DB
4.0
2011
Spring
DB
SOLR
Lucene
Share

Composition
Today
Based on
• Spring Context
• Policy/Behaviour framework
Guaranteed to be
• in-process
• in-context
• in-transaction
• synchronous

What we
need
• out-of-process
• out-of-context
• out-of-transaction
• asynchronous

Evolution
Build
MeasureLearn

Micrometer
New endpoint in ACS 6.1(enterprise-only):
/alfresco/s/prometheus
A Prometheus compatible scraping (read only
webscript). The metrics collection is powered by the
micrometer library (Vendor-neutral application metrics
facade)
https://micrometer.io/
https://prometheus.io/
https://grafana.com/

Micrometer
Available metrics
Basic JVM metrics: CPU, memory, GC, Threads and
etc.
Configuration:
metrics.jvmMetricsReporter.enabled=true
Grafana dashboard:
https://grafana.com/dashboards/470

Micrometer
Available metrics
Rest API call metrics:
● 1 timer “restapi_execution_time” with 2 labels:
○ Type of request (GET, POST,PUT,DELETE)
○ The path to the webscript (e.g: /alfresco/api/-
default-/public/alfresco/versions/1/people) if
path is enabled
Configuration:
metrics.restMetricsReporter.enabled=true
metrics.restMetricsReporter.path.enabled=false
Grafana sample dashboard:
https://github.com/Alfresco/acs-
packaging/tree/master/docs/micrometer

Micrometer
Available metrics
DB layer metrics:
● 1 timer “queries_execution_time” with 2 labels
○ Type of query (Select, Update, Insert, Delete)
○ The mybatis query ID (e.g:
alfresco.node.select_ChildAssocOfParentByName)
● 2 gauges: num_connections_active and
num_connections_idle
Configuration:
metrics.dbMetricsReporter.enabled=true
metrics.dbMetricsReporter.query.enabled=true
metrics.dbMetricsReporter.query.statements.enabled=false
Grafana sample dashboard: https://github.com/Alfresco/acs-

Micrometer
Available metrics
Tomcat metrics:
● servlet_request timer (number of responses with particular
response times)
Configuration:
metrics.tomcatMetricsReporter.enabled=true
Grafana sample dashboard: https://github.com/Alfresco/acs-

Micrometer
Future plans
● Integrate monitoring into Helm charts and CloudWatch in
AWS
● More metrics:
○ Audit
○ Renditions
○ Hazelcast caches
○ Logged in users
○ etc..

ACS and ActiveMQ
• Leading Open Source messaging platform
• Reliable, high performance messaging
• Fully supports JMS 1.1 and J2EE 1.4 with support
for transient, persistent, transactional and XA
messaging
• Supports a variety of Cross Language Clients and
Protocols from Java, C, C++, C#, Ruby, Perl,
Python, PHP
– OpenWire
– Stomp
– AMQP
– MQTT

ACS and ActiveMQ
• Alfresco Content Services (ACS) used ActiveMQ
for message queuing with various products, for
example Alfresco Media Management or Alfresco
Sync Service (Desktop Sync)
• Starting from ACS 6.1, ActiveMQ is a mandatory
requirement for the ACS Repository and its new
Raw Events module/capability

ACS and ActiveMQ
Connect ACS to ActiveMQ:
Define the location of ActiveMQ in your alfresco-
global.properties file:
messaging.broker.url=failover:(tcp://server:61616)?timeout
=3000
where server is the host name of the server where
ActiveMQ is installed.
Notes:
• When you set up ActiveMQ, the Alfresco Content Services
messaging subsystem is set to start up automatically.
• Any changes to broker URL requires ACS restart to apply
the changes.

ACS and ActiveMQ
Ways to deploy ActiveMQ:
● Containerized deployment*
https://github.com/Alfresco/acs-deployment
○ docker-compose
○ Helm / Kubernetes
● Using distribution zip
http://activemq.apache.org/download.html
● ...or your favourite managed offering, e.g.
AmazonMQ
* Conteinerized deployments of ACS use Alfresco ActiveMQ
image:
https://github.com/Alfresco/alfresco-docker-activemq
Can be also deployed via Helm chart:
https://github.com/Alfresco/alfresco-activemq-deployment

Two
Generals’
Problem
Possible inconsistencies:
• DB succeeds, but MB fails
• MB succeeds, but DB fails
“Two Generals’ Problem”
In Enterprise Software:
• Distributed Transactions (XA)

Distributed
Transactions
Severe additional complexity:
• Transaction manager required
Additional constraints
• All data sources need to support XA
⇒ In 6.1:
NO Distributed Transactions

Residual
Error
False Positive:
Messages have been sent out, but DB transaction
is rolled back
False Negative:
DB transaction succeeded, but Message Broker
fails
TransactionAwareEventProducer
• Implements 2nd approach
• Collects messages during transaction
• Sends all at once after commit

Recap:
What are
Policies
And
Behaviours
Policy / Behaviour framework:
Implementation of the observer pattern in
Alfresco
Policy:
Advertisement of an Extension Point provided by
a component
Behaviour:
Subscription of a Policy, i.e. code that gets
executed when a Policy is triggered

EventBehaviour
Existing Behaviours:
• JavaBehaviour
• ScriptBehaviour
New in 6.1:
• EventBehaviour

EventBehaviour
Example
@Autowired
private TransactionAwareEventProducer taep;
@Autowired
private PolicyComponent policyComponent;
@Override
public void configure() throws Exception
{
EventBehaviour eventBehaviour = new EventBehaviour(taep, sourceQueue, this,
"createEvent", Behaviour.NotificationFrequency.EVERY_EVENT);
policyComponent.bindClassBehaviour(
ContentServicePolicies.OnContentUpdatePolicy.QNAME,
RenditionModel.ASPECT_RENDITIONED, eventBehaviour);
...
}
public OnContentUpdatePolicyEvent createEvent(NodeRef node, boolean newContent)
{
OnContentUpdatePolicyEvent event = new OnContentUpdatePolicyEvent();
event.setNodeRef(node.toString());
event.setNewContent(newContent);
...
return event;
}

Transform Service
Bottleneck…
● Document transformations are used heavily within
ACS. All content imported into ACS generates
renditions, for example thumbnails and previews
● Most of the transformations were synchronous,
consuming resources of ACS JVM or underlying
OS.
● Security risk of running them on the same machine
as ACS
● Transformations have been a limiting factor when
ingesting large amounts of content, this means
that, in high load situations, transformations had to
be disabled

Transform Service
Overview
New Transform Service features:
● Microservice architecture
○ Independent scalability
○ Cost effective
○ High throughput
● Co-exists with existing transformations in ACS
● Containerised deployments only
● ACS 6.1+
● Enterprise-only
● Supports a smaller subset of available
transformations comparing to 6.0

ACS Async Rendition Service
V1 REST APIV0 REST API
Rendition Service 2
Rendition Service
Thumbnail Service
Action Service
Content Service Shared File Store
<<Docker>>
Content Update
Transform Service
Router
<<Docker>>
Transformers
<<Docker>>
Transformers
<<Docker>>
Transformers
<<Docker>>
Transformers
<<Docker>>
Transform Response
Transform Request
TransformerTransformerLocal Transformers
Solr 6
101010
101010
101010

Transform Service Configuration
local.transform.service.enabled=true
transform.service.enabled=true
Rendition 2 Definition Bean
<bean id="renditionDefinition2DocLib"
class="org.alfresco.repo.rendition2.RenditionDefinition2Impl">
<constructor-arg name="renditionName" value="doclib"/>
<constructor-arg name="targetMimetype" value="image/png"/>
<constructor-arg name="transformOptions">
<map>
<entry key="resizeWidth" value="100"/>
<entry key="resizeHeight" value="100"/>
<entry key="allowEnlargement" value="false" />
<entry key="maintainAspectRatio" value="true"/>
<entry key="thumbnail" value="true"/>
<entry key="timeout"
value="${system.thumbnail.definition.default.timeoutMs}" />
</map>
</constructor-arg>
<constructor-arg name="registry" ref="renditionDefinitionRegistry2"/>
</bean>
ImageMagick ‘flat’ Transform Options
alphaRemove
autoOrient
startPage
endPage
cropGravity
cropWidth
cropHeight
cropPercentage
cropXOffset
cropYOffset
thumbnail
resizeHeight
resizeWidth
resizePercentage
allowEnlargement
maintainAspectRatio

Transform Service
Future plans
• New Transform Service will increase the number of
supported transformations.
– Integrate with Search Services (to text
conversion)
– Extensibility
– AI and more..
• The synchronous renditions are deprecated and
will be removed from content repository
completely
• Metadata extraction is moved out from content
repository

Alfresco Identity Service
Overview
Alfresco Identity Service provides authentication and
proof of identity, in the form of a standard token (JSON
Web Token https://jwt.io ), understood by Digital
Business Platform components:
○ ACS 6.1 REST APIs (v1) and CMIS
○ APS 1.10 REST APIs
○ AGS 3.1 REST API (v1)
○ ADF 2.6
Alfresco Identity Service is based on Keycloak and
supports:
○ LDAP (OpenLDAP)
○ SAML 2.0 (PingFederate)
○ OpenID Connect

Deployment
● The deployment artifact is the alfresco-identity-
service Helm Chart
○ http://kubernetes-
charts.alfresco.com/stable/alfresco-identity-
service-1.0.0.tgz
● Deployed, by default, as part of the alfresco-
infrastructure Helm Chart
○ http://kubernetes-
charts.alfresco.com/stable/alfresco-
infrastructure-3.0.0.tgz
● Is NOT part of ACS 6.1 Helm charts
Contains:
● Keycloak with Alfresco Theme
● PostgreSQL
● Ingress

support in ACS
Represented by authentication subsystem:
● alfresco/subsystems/Authentication/identity-service
Based on Keycloak libs:
• https://github.com/keycloak/keycloak
(Supports JWT standard)
Users and groups are still managed in ACS:
• Can auto-create users
• Works with LDAP sync (recommended)
JWT and basic to access APIs in ACS
AIS and ACS are (almost) pre-configured for dev/test (test real is
present in AIS)

Configuration
Configure ACS alfresco-global.properties:
authentication.chain=ais:identity-service,...
identity-service.authentication.enabled=true
identity-service.authentication.validation.failure.silent=true
identity-service.authentication.defaultAdministratorUserNames=admin
identity-service.authentication.allowGuestLogin=true
identity-service.authentication.enable-username-password-authentication=true
identity-service.auth-server-url=http://localhost:8180/auth
identity-service.realm=alfresco
identity-service.ssl-required=none
identity-service.resource=alfresco
identity-service.public-client=true
Configure AIS:
https://github.com/Alfresco/alfresco-identity-service/tree/master/docs

Simplified sequence diagram:
Client AIS ACS
Request JWT
Full example: https://github.com/Alfresco/alfresco-identity-
service/blob/master/docs/resource/sequence/high-level-ldap-auth-
sequence.png
JWT: kfYclR..
GET /some-api
Authorization: Bearer kfYcIR...
Get signing keys
Verify JWT
Process request
Response to client

Future plans
• User and group management will be moved from
ACS into AIS
• Authentication chain will be deprecated and
removed. List of supported authentication types
will be extended in AIS.
• (Maybe) Move management of ACLs to AIS

HELM Charts
Since 6.0
• New additional deployment option
• Existing deployment options still available
• Codification of documentation and
recommended best practices
• Reference deployment: starting point for your
own environment description
• w/o params: Brings everything in a container
• Parameterize to use your infrastructure

6.1
AWS
Deployment
• Set of CloudFormation templates plus scripts
and tooling
• Sets up:
– VPC + Bastion
– EKS + Worker Nodes
– S3 incl replication
– Aurora DB
– AmazonMQ
– IAM permissions

AWS
Architecture

ACS on AWS
Source code locations
● CFN templates + AWS alterations
https://github.com/Alfresco/acs-deployment-aws
● ACS Helm Charts
https://github.com/Alfresco/acs-deployment
● Charts Repository
https://github.com/Alfresco/charts

Removed and
deprecated features

Removed and
deprecated features
● Removed (configuration and code)
○ CIFS
○ NTLM
● Deprecated (will be removed in future
versions)
○ Synchronous transformations
○ Cloud Sync

Beyond
• Extract common components
• Identity Service
– Move profiles to AIS
– Move user/group info to AIS
– Perform authentication outside of ACS
– All requests carry JWT token
• Move to Async, candidates
– Auditing
– Metadata extraction
– Search indexing
– Rules / Actions
• Deployment options for other cloud provider

Thanks!
Reach us on email:
alex.mukha@alfresco.com
stefan.kopf@alfresco.com
GitHub:
https://github.com/Alfresco
JIRA:
https://issues.alfresco.com

New Raw Events
Motivation (the “why?”)
• decoupling
• asynchronous processing
• integration
• fault-tolerance
• spikability
How?
• An evolutionary path to asynchronous processing
based upon policies / behaviours
• The event model enables us to execute code when
an event happens in the system. These events are
referred to as “policies”. With “behaviours”, you
can register code that is executed when these
events occur.

New Raw Events

New Raw Events
The basic data which is captured for each event:
• id
• type
• authenticatedUser
• executingUser
• timestamp
• schema

New Raw Events
EventProducers
AbstractEventProducer
• Abstract helper to send events to an endpoint. The
AbstractEventProducer acts as a wrapper that
provides marshalling for a Camel
ProducerTemplate.
• A client has the option of creating an event
producer without supplying an endpoint. In this
case, an endpoint must be provided with every
send operation.
• A client also has the option to provide an
ObjectMapper that will be used to marshal basic
POJOs to JSON, before sending the event.
TransactionAwareEventProducer
• Events are scheduled to be sent in post-commit
phase.

New Raw Events
EventBehaviour
• Event based Behaviour.
• A client uses an EventBehaviour to bind a send
event behaviour to a Class-level Policy.
• The endpoint uri can be a queue or a topic.
– e.g: jms:acs-repo-rendition-
events?jmsMessageType=Text
• The event behavior delegates the generation of the
event to a method pointer. The pointer is
represented by an instance object and method
name.

New Raw Events
Event Behaviour
Notification Frequency
• EVERY_EVENT: The event handler is then just
executed wherever it is being invoked in the code.
The name of this notification frequency implies that
the event handler will be called multiple times, but
that is not the case.
• TRANSACTION_COMMIT: This is the default, if
the notification frequency is not specified. The
event handler is queued and invoked at the end of
the transaction, after it has been committed. A
proxy around the event handler manages the
queuing.
• FIRST_EVENT: The event handler is invoked just
after the transaction is started. A proxy around the
event handler manages this.

Architectural changes in the repo in 6.1 and beyond

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Architectural changes in the repo in 6.1 and beyond

Similar to Architectural changes in the repo in 6.1 and beyond (20)

Recently uploaded

Recently uploaded (20)

Architectural changes in the repo in 6.1 and beyond