John Newton, founder and CTO of Alfresco, describes how Amazon Aurora enables the Alfresco Content Management System to store, manage, and retrieve billions of documents and related information with fast and linear scalability. Using new techniques of information modeling, indexing, and processing with the recently launched Aurora database, Alfresco can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. This session addresses the challenges of scaling document repositories to this level; architectural approaches for coordinating data; search and storage technologies such as Aurora, Solr, Amazon EBS, and Amazon S3; the breadth of use cases that modern content systems need to support; and how to support user applications that require subsecond response times. The result is a solution that once would have required large data centers to support but can now be handled cost-effectively with AWS and Aurora.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
John Newton, Founder & CTO Alfresco
@johnnewton
October 2015
DAT309
Scaling Massive Content
Stores with Amazon Aurora
Alfresco on AWS

2. What to expect from the session
• Challenges of scaling to billions of documents
• Architectural approaches of managing data, search, and storage
with Amazon Aurora, Solr, Amazon EBS, and Amazon S3
• The breadth of use cases of content at scale
• How to support user applications that require sub-second response
times
• Moving from large data centers to cost-effective management with
AWS and Amazon Aurora

4. Alfresco in action
Government
Finance
Manufacturing
Transportation & Utilities
Healthcare Other

5. Content in dynamic
context
Consumerized search &
discovery
Secure & mobile
collaboration
Invisible Information
Governance
SIMPLE
SMART
Powerful metadata,
rules & relationships
Easy process
creation & analysis
CONTENT PROCESS
Modern & agile
architectureOPEN
Cloud integration,
sync & scale
Open source, open
APIs
Difference
@johnnewton alfresco.com/awsreinvent

6. Somewhere deep in the Nevada Desert

7. in a secret underground location

8. someone is trying to store…
One Billion
Documents!!!
http://www.warnerbros.com/austin-powers-international-man-mystery

9. Some have tried before … and failed!
We’ll
configure
1 Million
SharePoint
Servers!!!

10. Digital transformation is driving huge flows of content
Gartner Nexus
PWC 6th Annual Digital IQ Survey, 2014
Digital
Business
Cloud Social
Big DataMobile

11. Content use cases at scale
Enterprise
Document Library
Loans &
Policies
Claims & Case
Processing
Transaction &
Logistics Records
Research &
Analysis
Real-time
Video
Internet of
Things
Medical &
Personnel Records
Government
Records & Archives
Discovery &
Litigation

12. Content management applications
Document
Library
Image
Management
File Sync &
Share
Search &
Retrieval
Business
Process
Management
Records
Management
Case
Management
Media
Management
Information
Archiving

13. What is this
“Content”?!!!

14. Content vs. data vs. files vs. EFSS
Data Files EFSS Content and ECM

15. Content architecture as a big data problem
15
Files /
Renditions
Metadata
Directory CategoriesRelationships
Indexes
Search
Activities
Security People
APIs
Processes /
Tasks
Rules
Semantics
Types
Content
Object
Access Create – Manage – Distribute – Use
Context
Database
Distributed
FSDatabase
Solr /
Elasticsearch

16. Content at scale in the enterprise
Users at Scale
Concurrency Content Count
Read/Write
Throughput
Geographic
Distribution
Volume Size

17. The problem with traditional approaches
Provisioning and
Administration
Geographic Distribution Lack of Agility
Lack of Redundancy Lack of Elasticity

18. It’s enough to get one “fired”

19. Unique challenges of very large repositories
19
Scaling Up
• Clustered Servers
• Clustered Database
• Clustered Indexes
• Read Replicas
Scaling Out
• Sharding
• Federation
• Replication
• Shared Nothing

20. Content management architecture
20
Alfresco Share
Alfresco Repository
Alfresco SOLR
Activiti Workflow
Engine
Database
FS Content
Store
Indexes
S3
Amazon RDS
Amazon Elastic Block
Store or Ephemeral
PIOPS EBS
(or Amazon Glacier)
EC2

21. Content management architecture
21
Alfresco Share
Alfresco
Repository
Alfresco SOLR
Activiti
Workflow
Protocols
APIs (CMIS)
Media
Mgmt
Desktop and
Mail Client
Mobile App Cloud Sync
Database
FS Content
Store
Indexes
Records
Management
Reports &
Analytics
Reports and
Analytics Server
Media Transform
Services
Transforms
Authentication
Auditing
Rules/Policies
Web Scripts
Scheduled Jobs

22. Process management architecture
Database Elasticsearch
Files
Amazon EBS
Process
Mining
Activiti Engine
Tomcat / Jetty
Process
Virtual
Machine
Tasks Processes Jobs
Activiti REST App REST Admin REST
MS Office
Protocol
Activiti
Analysis
AngularJS
Activiti
App
AngularJS
Activiti
Admin
AngularJS
MS
Office
Activiti
Mobile
iOS / Android
Activiti
Designer
Eclipse

23. Scaling in tiers
Alfresco
Transformation Server
Alfresco
Transformation Server
Alfresco Solr
Alfresco Local Repo
(Index Tracking)
Alfresco Solr
Alfresco Local Repo
(Index Tracking)
Alfresco Repository Alfresco Repository
Alfresco Share Alfresco Share
Alfresco Activiti Suite
Alfresco Activiti Suite

24. Multi-tenant Cloud Service on AWS
RDS
Activities
Route53 (DNS)
S3
ELB
Layer7
solr trans
/share
/alfresco
haproxy
haproxy
varnish
/share
/alfresco
haproxy
/share
/alfresco
haproxy
haproxy
varnish
haproxy
varnish
web nodes
alfresco nodes
solr
solr trans
trans

25. You can scale
my server any
day, Baby!!!

26. Data meta-model
A
B
C
D
Folder
Folder
Doc
Doc
rendition
Class
Type Aspect
Property
Association
Constraint
Child
Association
Folder
Document
contains
name
name
content
Auditable who by
when
rendition
Type
Child Association
Type
Association
Property
Property
Property
Aspect
Model Metadata Organization
1 Billion 15 Billion

27. Database schema
• Tables
• Indexes
• Concurrency
• Throughput Techniques
• Striping
• Parallelism
• Sharding
Table Size (GB)
alf_child_assoc 448
alf_content_data 149
alf_content_url 202
alf_node 711
alf_node_aspects 217
alf_node_properties 1,524

28. Amazon Aurora difference
AZ 1 AZ 2
EBS
mirror
EBS
mirror
Amazon S3
EBS
Standby
Instance
Primary
Instance
AZ 1 AZ 3
Amazon S3
Primary
Instance
AZ 2
Replica
Instance
• Highly available — synchronous vs. asynchronous replication
• Significantly more efficient use of network I/O
• Self-healing, fault-tolerant, instant crash recovery
MySQL with standby Amazon Aurora
async
4/6 quorum
PiTR
Sequential
write
Sequential
write
Distributed
writes
Amazon Elastic
Block Store (EBS)

29. Modifications required for Aurora

30. Index and search architecture
Full-Text Query
Metadata Query
Facets & Buckets
Security Filters
Results Processing
Credit: Ryan Tobora
ThinkBig, Teradata
http://thinkbig.teradata.com/solrcl
oud-terminology/
Text Extraction
Metadata Injection
& Path Processing
Shingles
ACL Processing
Results ProcessTerm-hit Highlighting
x 20 instances

31. Storage Layer
File storage architecture
In Place: AWS Import/Export Snowball
Direct
Streaming
Aurora EBS
Metadata Content
Metadata
Content
Archive Layer
S3 Amazon Glacier
Metadata Content
File
System
Protocols
APIs

32. Security architecture
Web Filtering
Enterprise
Authentication
Access Control
Lists and Caveats
Query Filters and
Object Policies

33. Do you suppose we
can put it together
with some string and
Scotch Tape?!!

34. Provisioning VLDB repos
10
new architecture allows room to scale the environment to support 2013 and 2014 roadmap
plans while still supporting an environment that will be reliable and robust.
Additionally, this environment support disaster recovery capabilities as well, guaranteeing that in
case of a severe outage, that backups are stored and quick turn around can occur to restore the
environment.
Below is a screenshot of the Customer Deployment Portal in which Stanford will be able to scale
the Alfresco environment seamlessly within a web based UI.
Flexibility
This new architecture will also utilize new Cloud Ops tools that will allow increased flexibility in
the administration of the Alfresco environment. This gives Stanford the flexibility to grow or
shrink the different environments based on demand, pricing, or performance. While the need for
flexibility of the environment might be minimal in production, this will be especially advantageous
as Stanford develops on the Alfresco service, and needs to rapidly spin up and down
test/development environments.
Self-Service
The Customer Deployment Portal will be one of the benefits of moving to the proposed
environment. The Customer Deployment Portal is a web based administration tool that allows
Stanford to self-service their environment. Stanford will be able to setup, deploy, change, and
monitor the different AWS environments through a user friendly and intuitive web interface.
This includes control over the number of virtual machines, size of the virtual machines, load
balancers, databases, storage sizes and types, and more.
ContainersDevOps
Data as a Service
Indexing and
Search as a Service
Files as a Service
Rolling Deployment
• Nginx
• HA Proxy
• Varnish
• Alfresco Share
• Alfresco Repo
• Alfresco
Analytics
• Alfresco Media
• Activiti
• Solr
• ActiveMQ
• Transform
• Database
• Storage
• LDAP
• Email Server
• Logs
• Monitoring
Security

35. Large-scale benchmarking
BM01 User scenarios
BM02 User concurrency on single
node
BM03 Solr Performance
BM04 Concurrent Load and Access
– multi-user
BM05 User Invite and Tenant
Provisioning
BM06 Workflow service
performance
BM07 Workflow API performance
BM08 High concurrency in Multi-
Tenancy
https://wiki.alfresco.com/wiki/Benchmark_Testing_with_Alfresco
https://github.com/AlfrescoBenchmark/alfresco-benchmark
Benchmark Server
Tomcat 7
Rest API
MongoDB
Config Data
Services
MongoDB
Test Data
UI
Benchmark Driver (xN)
Benchmark Driver (xN)
Benchmark Driver
Tomcat 7 Extras
(Selenium)
Servers / APIs Servers / APIs
Load Balancer
Servers / APIs
Test
Services
Rest API

36. BM4 test execution environment: 1.2B docs
UI Test x 20 m3.2xlarge
Simulate 500 Users
• Selenium / Firefox
• 1 hour constant load
• 10 sec think time
UI Test UI Test
Alfresco Alfresco Alfresco x 10 c3.2xlarge
Alfresco with Share
and Repo
Solr x 20 c3.8xlargeSolr Solr
Aurora x 1 db.r3.4xlarge
ELB
Sharded Solr Cloud
sites folders files transactions dbSize GB
10,804 1,168,206 1,168,206,000 15,475,064 3,185
Simulate AWS
Import/Export
(in place)

37. Benchmark results
• Document load rate 1200 per sec
• 4.3 Million per Hour on 10 nodes!
• Load rate consistent beyond 1B
• CPU loads:
• Database: 70-80% in Bulk Load
• Alfresco: ~50%
• Solr: <60%
• CMIS API Calls (OASIS Standard)
• Aurora indexes efficient at 3.2TB
• NAME (=, LIKE) ~ 20ms
• IN_FOLDER (sorted, limited) ~ 160ms
• Sub-second login times and good,
linear responses for other actions
• Open Library: ~3s
• Page Results: <1s
• Navigate to Site: ~1s
• Individual search ~1s
• 500 concurrent search: ~3s response
• CPU loads:
• Database: <10%
• Alfresco: 25-30%
• Solr: 25-30%
No size-related bottlenecks with 1.2 billion documents
Bulk Operations User Operations

38. AWS
What a difference
3-6 Months
Questionable Scale
Little Redundancy
Lots of $$$
< 30 mins
10x Faster
Elastic, Fault-Tolerant
Open, Cost Effective
ECM ECM ECM
Search Search Search
FS FS FS
Hardware Hardware Hardware
Load Balancer
DR Plan
HSM HSM HSM
ECM ECM ECM
ELB
Alfresco Alfresco Alfresco
Solr Solr Solr
S3
EC2 EC2 EC2
AZ1 AZ2 AZ3
Aurora
EBS

39. What a difference
3-6 Months
Questionable Scale
Little Redundancy
Lots of $$$
< 30 mins
10x Faster
Elastic, Fault-Tolerant
Open, Cost Effective

40. Well, what am I
supposed to do
with all this
frickin’
hardware?!!

41. Future: globally distributed environment
41
Distributed FS Distributed FS
Sharding Sharding

42. Thank you!
Vegas, Baby!
Thank you!

43. What the world
needs now,
is evaluation sheets!
No, not just for some,
But for @johnnewton!
alfresco.com/awsreinvent