Cloud computing platforms provide many options to store data, and their capabilities are expanding at an incredible speed. In this session, Nick provides an overview of different cloud-native persistence options, with examples of how each type is used, and where it fits on the Serverless scale. He presents solutions and offerings (such as AWS Neptune and Amazon Serverless Aurora) from major vendors in that space which cater for a variety of requirements and use cases and talks about considerations that should be made when selecting these.

1. 3
/in/nick-do/
nick.do@datumhq.com

2. > print(“Hello World!”)
✓ 1+ Year of experience using AWS
✓ 2 x AWS certified
✓ Also love to chat with your about
electromagnetics, magnets &
motors
✓ Love learning & sharing
Serverless Meetup Auckland – 17/04/18

3. What I want to cover
✓ Cloud-native applications
✓ Persistent characteristics
✓ Choice of approach - on the Serverless scale
✓ Cloud-native persistent options – OpenEdx example
✓ Considerations
Serverless Meetup Auckland – 17/04/18

4. 1. Cloud-native applications
Utilize benefits of the cloud
ScaleDurability Availability
Serverless Meetup Auckland – 17/04/18

5. 1. Cloud-native applications
DecoupledSmall Stateless
Tends to follow Microservices architecture
Serverless Meetup Auckland – 17/04/18

6. 2. Persistent characteristics
✓ Meta data
✓ Documents (XML, YAML, JSON)
✓ Stream (Logs, Time series…)
✓ Object (Media, sound..)
✓ Transactional queries
Balancer
Web Server
Reverse Proxy
Application Server
Data type
= format for encoding data
Serverless Meetup Auckland – 17/04/18

7. 2. Persistent characteristics
Consistency model
= rules of memory operation for each UPDATE
https://www.slideshare.net/springerw/eventually-consistent
Serverless Meetup Auckland – 17/04/18

8. 2. Persistent characteristics
Persistency
= the keeping of data that you are working with
https://briantroy.com/2010/11/02/big-data-storage-isnt-enough/Serverless Meetup Auckland – 17/04/18

9. 2. Persistent characteristics
Latency
= time taken to perform a READ or WRITE
ms seconds minutes hours
Caches
NoSQL
SQL
HDFS
Object
Depends on size
Serverless Meetup Auckland – 17/04/18

10. 2. Persistent characteristics
✓ Data type
✓ Consistency model
✓ Persistency
✓ Latency
 Concurrency
 Streaming capability
 Availability/SLA
 Durability
…
Cassandra
Serverless Meetup Auckland – 17/04/18

11. 3. Choice of approach
Managed services
Serverless services
only accessible via APISelf-host (DIY)
Scaling delay • You control • Within minutes or hours • Instant or within seconds
Pricing model • Servers up time • Cold storage & bandwidth • Pay per request
Security
& Compliance
• Implemented with best
practices
• Rely on Compliances offered
by the provider
• You control • Invulnerable to classic attacks
(cross site scripting,
injections…)
• Rely on Compliances offered
by the provider
Control • Full control (reaction
speed, optimization,
tenancy, compliance)
• Highly restrictive
• Leave you with control of
scaling, encryption & backup
• No control, not even scaling
Best fit • High resource utilization
• Predictable demand
• Strict SLA & Compliances
• Operation expertise
• Focus on growth
• Can afford some operation work
• Focus on growth
• Loose latency requirements
• Unpredictable demand
• Peace of mind!
Serverless Meetup Auckland – 17/04/18

12. 4. Persistence options – case study:
• edX: non profit organization founded by MIT & Havard
• Open edX: LMS & MOOC platform released in June 2013
• Theoretically possible: 1,000,000 concurrent users
• Powers edx.org & cousera.com
Serverless Meetup Auckland – 17/04/18

13. 4. Persistence options – case study:
edx.org & cousera.com
Serverless Meetup Auckland – 17/04/18

14. VM local & Block storage
Amazon
EC2
Amazon
EBS
&
GCP
Compute Engine
GCP
Persistent Disk
&
Workload
Management
• Very low latency disk
• High IPOS
• High throughput
• Self-managed
Example • Web, application server
• Literally anything
Azure
Virtual Machine
Azure
Disk Storage
&
4. Persistence options – case study:
Serverless Meetup Auckland – 17/04/18

15. Object Storage
Workload
Management
• Static object storage
• Static web hosting
• Static content distribution
• Serverless
Example • Static website
• Video, audio, document
GCP
Cloud Storage
Amazon S3
Azure
Storage - Blobs
4. Persistence options – case study:
Serverless Meetup Auckland – 17/04/18

16. File Storage
Workload
Management
• Shared file system
• Low latency access
• Access semantic required
(strong consistency, locking)
• Managed Services
Example • CMS Template storage
• Configuration management
Amazon EFS
Azure
Storage - Files
GCP
Persistent Disk
(multiple VM, read-only)
4. Persistence options – case study:
Serverless Meetup Auckland – 17/04/18

17. Message Queue
Workload
Management
• Pull-based delivery
• Decoupling components
• Serverless
Example • Processing request queue
GCP
Task Queue
Amazon SQS
Azure
Storage - Queue
4. Persistence options – case study:
Serverless Meetup Auckland – 17/04/18

18. Relational database
Workload
Management
• OLTP/OLAP
• Structured data
• SQL Query capability
• Managed Services
Example • Student details
• Sessions ID
• Data warehouse & analytics
Amazon
RDS & Aurora
Amazon
Redshift
Azure
SQL Database
Azure
SQL Data
warehouse
GCP
Cloud SQL
GCP
Cloud Spanner
GCP
BigQuery
4. Persistence options – case study:
Serverless Meetup Auckland – 17/04/18

19. Non-Relational database
Workload
Management
• OLTP
• Flexible structured data
• Highly scalable
• Managed Services
Example • Course contents
• Forums
GCP
Cloud Datastore
GCP
Cloud Bigtable
Amazon
DynamoDB
Azure
CosmoDB
4. Persistence options – case study:
Serverless Meetup Auckland – 17/04/18

20. In-memory caches
Workload
Management
• In memory key-value store
• Database Query result cache
• Compute result cache
• Managed Services
Example • Session information
• Save READ heavy workload
CDN
Workload
Management
• Static content distribution
• Access controlled static
content distribution
• Managed Services
Example • Reduce static website
load speed
GCP
Cloud CDN
Amazon
CloudFront
Azure
CDNAzure
Cache
Amazon
Elasticache
Amazon
DynamoDB
Accelerator
4. Persistence options – case study:
Serverless Meetup Auckland – 17/04/18

21. 4. Persistence options – NEW: AWS Neptune
Graph database
Workload
Management
• OLTP/OLAP
• Highly connected data
• Managed Services
Example • Knowledge graph
• Social networking
• Recommendation engine
https://www.awsgeek.com/posts/amazon-neptune-notes/
https://aws.amazon.com/neptune/Serverless Meetup Auckland – 17/04/18

22. 4. Persistence options – NEW: Serverless Aurora
• Serverless SQL database
• Upto 64TB
• SQL Workload:
• Intermittent
• Unpredictable
• Rapid scaling
• Pay for cold storage &
requests
https://aws.amazon.com/blogs/aws/in-the-works-amazon-aurora-serverless/
Serverless Meetup Auckland – 17/04/18

23. 5. Considerations - technical
• Data Format
• Consistency model
• Persistency
• Latency
• Concurrency
• Streaming capability
• Availability/SLA
• Durability
…
JSON object
• Read
• Write
Use the simplest type of storage
… ALLOWED
Simple
Read & Write
Object
Storage
File
Storage
Consistent latency OLTP
Databases
Data
warehouse
Serverless Meetup Auckland – 17/04/18

24. 5. Considerations - business
✓ Self-host (DIY)
✓ Managed services
✓ Serverless
Compliances requires LOTS of work
… compliant cloud applications requires compliant components.
AWS Lambda got PCI DSS compliance
(Payment Card Industry) in June 2017
AWS RDS got HIPAA compliance
(Healthcare data) in Sep 2017
https://aws.amazon.com/blogs/security/aws-adds-12-more-services-to-its-
pci-dss-compliance-program/
Serverless Meetup Auckland – 17/04/18
https://aws.amazon.com/about-aws/whats-new/2017/09/amazon-rds-for-sql-
server-and-amazon-rds-for-mariadb-achieve-hipaa-eligibility/

25. 5. Considerations - business
✓ DevOps
✓ Managed services
✓ Serverless
Starts with NO operation
… initially DropBox started with just UI on top of Amazon
S3.
https://www.slideshare.net/ChiaraCilardo/dropbox-architecture-and-business-prospective
Serverless Meetup Auckland – 17/04/18

26. 5. Considerations - business
✓ DevOps
✓ Managed services
✓ Serverless
DevOps is NOT going away soon
… instead of fully embracing 1 approach: analyze, try , and
gain.
https://medium.com/openwhisk/uncovering-the-magic-how-serverless-platforms-really-work-3cb127b05f71
https://github.com/openfaas/faas
IBM OpenWhisk – Serverless
Computing is built on top of
DevOps
OpenFaaS – Serverless
Functions with first class metric
support
Serverless Meetup Auckland – 17/04/18

27. Thank you for your time!
Any questions?

28. Bonus: Key-value vs. Object vs. Block vs. File
https://blog.rackspace.com/introduction-to-object-storage
Key-value vs. Object storage:
• Both access using API calls
• Object storage allow you to look into and query the content
Block/Blobs storage:
• SAN, iSCSI, RAID …
• The lowest level + attach directly to OS
File storage:
• CIFS or NFS
• Still a file system like Block/Blobs, but indirect access via translated
command
Serverless Meetup Auckland – 17/04/18

29. Bonus: Should I use Amazon DynamoDB?
https://read.acloud.guru/why-amazon-dynamodb-isnt-for-everyone-and-how-to-decide-when-it-s-for-you-aefc52ea9476
Serverless Meetup Auckland – 17/04/18

30. Bonus: GCP data store – a selection guide
https://medium.com/google-cloud/a-gcp-flowchart-a-day-2d57cc109401
Serverless Meetup Auckland – 17/04/18

31. I wish to acknowledge:
• Mr. Andreas Mueller – General Manager at API Talent
• Mr. Kevin Tran – Senior Software Architect, Tech Evangelist
• Mr. Nghia Nguyen – Senior Solution Architect at Datacom
Honorable mention:
• Mr. Harel Aharony - eCommerce Platform Lead at XM Developments
References:
• https://medium.com/google-cloud/a-gcp-flowchart-a-day-2d57cc109401
• https://medium.com/openwhisk/uncovering-the-magic-how-serverless-platforms-really-work-3cb127b05f71
• https://medium.com/@james.anderson_88068/aws-first-look-advancements-in-aurora-fcb7ff4c8b9e
• https://read.acloud.guru/the-serverless-spectrum-147b02cb2292
• https://read.acloud.guru/simon-wardley-is-a-big-fan-of-containers-despite-what-you-might-think-18c9f5352147
• https://thenewstack.io/continuum-cloud-native-topologies/
• https://aws.amazon.com/blogs/aws/in-the-works-amazon-aurora-serverless/
• https://read.acloud.guru/why-amazon-dynamodb-isnt-for-everyone-and-how-to-decide-when-it-s-for-you-aefc52ea9476
Good reads I wish to share:
• “The Phoenix Project” by Gene Kim, George Spafford, and Kevin Behr
• “Serverless Architectures on AWS” by Peter Sbarski
• “Architecting for Scale” by Lee Atchison
/in/nick-do/
nick.do@datumhq.com
Serverless Meetup Auckland – 17/04/18