Cloud computing platforms provide many options to store data, and their capabilities are expanding at an incredible speed. In this session, Nick provides an overview of different cloud-native persistence options, with examples of how each type is used, and where it fits on the Serverless scale. He presents solutions and offerings (such as AWS Neptune and Amazon Serverless Aurora) from major vendors in that space which cater for a variety of requirements and use cases and talks about considerations that should be made when selecting these.
2. > print(“Hello World!”)
✓ 1+ Year of experience using AWS
✓ 2 x AWS certified
✓ Also love to chat with your about
electromagnetics, magnets &
motors
✓ Love learning & sharing
Serverless Meetup Auckland – 17/04/18
3. What I want to cover
✓ Cloud-native applications
✓ Persistent characteristics
✓ Choice of approach - on the Serverless scale
✓ Cloud-native persistent options – OpenEdx example
✓ Considerations
Serverless Meetup Auckland – 17/04/18
6. 2. Persistent characteristics
✓ Meta data
✓ Documents (XML, YAML, JSON)
✓ Stream (Logs, Time series…)
✓ Object (Media, sound..)
✓ Transactional queries
Balancer
Web Server
Reverse Proxy
Application Server
Data type
= format for encoding data
Serverless Meetup Auckland – 17/04/18
7. 2. Persistent characteristics
Consistency model
= rules of memory operation for each UPDATE
https://www.slideshare.net/springerw/eventually-consistent
Serverless Meetup Auckland – 17/04/18
8. 2. Persistent characteristics
Persistency
= the keeping of data that you are working with
https://briantroy.com/2010/11/02/big-data-storage-isnt-enough/Serverless Meetup Auckland – 17/04/18
9. 2. Persistent characteristics
Latency
= time taken to perform a READ or WRITE
ms seconds minutes hours
Caches
NoSQL
SQL
HDFS
Object
Depends on size
Serverless Meetup Auckland – 17/04/18
10. 2. Persistent characteristics
✓ Data type
✓ Consistency model
✓ Persistency
✓ Latency
Concurrency
Streaming capability
Availability/SLA
Durability
…
Cassandra
Serverless Meetup Auckland – 17/04/18
11. 3. Choice of approach
Managed services
Serverless services
only accessible via APISelf-host (DIY)
Scaling delay • You control • Within minutes or hours • Instant or within seconds
Pricing model • Servers up time • Cold storage & bandwidth • Pay per request
Security
& Compliance
• Implemented with best
practices
• Rely on Compliances offered
by the provider
• You control • Invulnerable to classic attacks
(cross site scripting,
injections…)
• Rely on Compliances offered
by the provider
Control • Full control (reaction
speed, optimization,
tenancy, compliance)
• Highly restrictive
• Leave you with control of
scaling, encryption & backup
• No control, not even scaling
Best fit • High resource utilization
• Predictable demand
• Strict SLA & Compliances
• Operation expertise
• Focus on growth
• Can afford some operation work
• Focus on growth
• Loose latency requirements
• Unpredictable demand
• Peace of mind!
Serverless Meetup Auckland – 17/04/18
12. 4. Persistence options – case study:
• edX: non profit organization founded by MIT & Havard
• Open edX: LMS & MOOC platform released in June 2013
• Theoretically possible: 1,000,000 concurrent users
• Powers edx.org & cousera.com
Serverless Meetup Auckland – 17/04/18
23. 5. Considerations - technical
• Data Format
• Consistency model
• Persistency
• Latency
• Concurrency
• Streaming capability
• Availability/SLA
• Durability
…
JSON object
• Read
• Write
Use the simplest type of storage
… ALLOWED
Simple
Read & Write
Object
Storage
File
Storage
Consistent latency OLTP
Databases
Data
warehouse
Serverless Meetup Auckland – 17/04/18
24. 5. Considerations - business
✓ Self-host (DIY)
✓ Managed services
✓ Serverless
Compliances requires LOTS of work
… compliant cloud applications requires compliant components.
AWS Lambda got PCI DSS compliance
(Payment Card Industry) in June 2017
AWS RDS got HIPAA compliance
(Healthcare data) in Sep 2017
https://aws.amazon.com/blogs/security/aws-adds-12-more-services-to-its-
pci-dss-compliance-program/
Serverless Meetup Auckland – 17/04/18
https://aws.amazon.com/about-aws/whats-new/2017/09/amazon-rds-for-sql-
server-and-amazon-rds-for-mariadb-achieve-hipaa-eligibility/
25. 5. Considerations - business
✓ DevOps
✓ Managed services
✓ Serverless
Starts with NO operation
… initially DropBox started with just UI on top of Amazon
S3.
https://www.slideshare.net/ChiaraCilardo/dropbox-architecture-and-business-prospective
Serverless Meetup Auckland – 17/04/18
26. 5. Considerations - business
✓ DevOps
✓ Managed services
✓ Serverless
DevOps is NOT going away soon
… instead of fully embracing 1 approach: analyze, try , and
gain.
https://medium.com/openwhisk/uncovering-the-magic-how-serverless-platforms-really-work-3cb127b05f71
https://github.com/openfaas/faas
IBM OpenWhisk – Serverless
Computing is built on top of
DevOps
OpenFaaS – Serverless
Functions with first class metric
support
Serverless Meetup Auckland – 17/04/18
28. Bonus: Key-value vs. Object vs. Block vs. File
https://blog.rackspace.com/introduction-to-object-storage
Key-value vs. Object storage:
• Both access using API calls
• Object storage allow you to look into and query the content
Block/Blobs storage:
• SAN, iSCSI, RAID …
• The lowest level + attach directly to OS
File storage:
• CIFS or NFS
• Still a file system like Block/Blobs, but indirect access via translated
command
Serverless Meetup Auckland – 17/04/18
29. Bonus: Should I use Amazon DynamoDB?
https://read.acloud.guru/why-amazon-dynamodb-isnt-for-everyone-and-how-to-decide-when-it-s-for-you-aefc52ea9476
Serverless Meetup Auckland – 17/04/18
30. Bonus: GCP data store – a selection guide
https://medium.com/google-cloud/a-gcp-flowchart-a-day-2d57cc109401
Serverless Meetup Auckland – 17/04/18
31. I wish to acknowledge:
• Mr. Andreas Mueller – General Manager at API Talent
• Mr. Kevin Tran – Senior Software Architect, Tech Evangelist
• Mr. Nghia Nguyen – Senior Solution Architect at Datacom
Honorable mention:
• Mr. Harel Aharony - eCommerce Platform Lead at XM Developments
References:
• https://medium.com/google-cloud/a-gcp-flowchart-a-day-2d57cc109401
• https://medium.com/openwhisk/uncovering-the-magic-how-serverless-platforms-really-work-3cb127b05f71
• https://medium.com/@james.anderson_88068/aws-first-look-advancements-in-aurora-fcb7ff4c8b9e
• https://read.acloud.guru/the-serverless-spectrum-147b02cb2292
• https://read.acloud.guru/simon-wardley-is-a-big-fan-of-containers-despite-what-you-might-think-18c9f5352147
• https://thenewstack.io/continuum-cloud-native-topologies/
• https://aws.amazon.com/blogs/aws/in-the-works-amazon-aurora-serverless/
• https://read.acloud.guru/why-amazon-dynamodb-isnt-for-everyone-and-how-to-decide-when-it-s-for-you-aefc52ea9476
Good reads I wish to share:
• “The Phoenix Project” by Gene Kim, George Spafford, and Kevin Behr
• “Serverless Architectures on AWS” by Peter Sbarski
• “Architecting for Scale” by Lee Atchison
/in/nick-do/
nick.do@datumhq.com
Serverless Meetup Auckland – 17/04/18