Open Repository 2019 developer track presentation. https://ty.yinlinchen.com/n4984n5c

1. A Multi-Tenancy Cloud-Native
Digital Library Platform
Yinlin Chen, Jim Tuttle, William A. Ingram
{ylchen, jim.tuttle, waingram}@vt.edu
Information Technologies and Services
Virginia Tech Libraries

2. Agenda
• Cloud-native concept
• Virginia Tech Digital Library Platform (VTDLP)
• Design strategy
• Architecture overview
• Implementation overview
• VTL experiences

3. Cloud-native Concept
• Entire infrastructure is deployed in the Cloud (AWS)
• Platform is composed of a suite of microservices and
managed services
• Focus on the business logic and workflow
• Utilize the advantages provided by the Cloud

4. Virginia Tech Digital Library Platform (VTDLP)
• New services to Digital Library Platform
– ID Minting service, Access Service, Metadata service, …
• Migrating legacy services to Digital Library Platform
– IAWA, VTechWork, …
Preservation Data Modeling Presentation

5. VTDLP Overview
Preservation staging
SW Virginia
IAWA Images
VtechWork ETDs
Storage
APTrustAmazon S3
Presentation
Others
IAWA
Others
BeyondVT
.
.
.
Batch Metadata
Service
Metadata ServiceID Minting Service
Resolution
Service
Serialization
Service
Other Services

6. Design Strategy
• Cloud native (AWS ecosystem)
• Microservice (AWS lambda)
• Serverless (AWS managed services)
• CI/CD Pipeline
• Caching as much as possible
– Static files
– Lambda functions
• Automation as much as possible
– Infrastructure as code
– No manual provisioning or managing servers

7. AWS Ecosystem
Amazon
S3
Amazon
Glacier
AWS
Lambda
Amazon
DynamoDB
Amazon
CloudFront
Amazon
Route 53
Amazon
CloudWatch
AWS
CloudTrail
AWS
CloudFormation
IAM
Amazon API
Gateway
Amazon
SQS
Amazon
SNS
AWS CLI
Network & Content Delivery Compute & Database
Management
AWS
Organizations
Messaging Security & Identity Storage
Services
Amazon
Pinpoint
Amazon
Cognito
Amazon
EC2
Amazon ES
AWS Certificate
Manager
AWS
Amplify

8. Software stacks
React AWS Amplify Node.js Python
Web App Microservice
(AWS Lambda)
AWS AppSync

9. Preservation Pipeline
Checksums
Fixity
Virus Scan
AWS S3
APTrust
PREMIS
Apache Airflow

10. Lambda Example – Metadata file
1. File upload to S3
2. S3 triggers a Lambda function
3. Lambda function parses file content and inserts/updates record in
the DynamoDB

11. Lambda Example – DynamoDB / ES
1. Data modifications in DynamoDB will trigger a Lambda function
2. Lambda function captures changes and updates Amazon ES

12. Presentation - Multi-Tenant Architecture
DB
Application Hub
Search
App1 App2 AppN

13. Web App
Amazon Route 53
Amazon API
Gateway
Amazon
CloudFront
AWS
Certificate Manager
Amazon Cognito
Amazon S3
AWS Lambda
Amazon
DynamoDB
Amazon
Elasticsearch
Service
AWS Cloud

14. The International Archive of Women in Architecture
• A level 0 compliant image server using Amazon S3 and Amazon
CloudFront
• Tiles images, manifest JSON files, and etc.
• Terabytes of scan images to be processed

15. Image processing workflow
AWS Lambda
Amazon S3
Amazon S3
AWS Batch
Batch Job –
image set 1
Batch Job –
image set 2
Batch Job –
image set 3
Batch Job –
image set N
Tiles &
Manifest
Raw
images
Amazon
CloudWatch
Rule
Amazon Elastic
File System
Amazon EC2

16. Batch job - IIIF_S3 Docker
Amazon S3
• Command
• Parameters
• Environment
variables
• vCPUs
• Memory
Tiles &
Manifest
IIIF
Amazon Elastic
File System
AWS Batch

17. CI/CD with AWS
Developers AWS
CodePipeline
AWS CodeBuild
AWS CloudFormation
Amazon S3
AWS Lambda
Amazon API Gateway
(1) (2)
(3)
(4)
(5)
(6)
(7)

18. Cloud benefit - Backup examples
• S3
– Amazon S3 is 99.999999999% durability and 99.99% availability.
– On average, may lose one of 10,000 objects every 10 million years
or so.
– Cross-region replication
• DynamoDB
– Point-in-time recovery (Last 35 days)
– On-Demand Backup (Stored in S3)
• ElasticSearch
– Daily snapshots (Last 14 days)
– On-Demand Backup (Stored in S3)

19. VTL Experiences
• Entire development team is AWS certified
• One AWS Certification Subject Matter Expert (SME)
• AWS trainings and conferences
• Thinking and implementing new ideas the Cloud way

20. Q & A
Thank You!