SlideShare a Scribd company logo
1 of 27
Download to read offline
Lessons Learnt building a
Distributed Linked List on S3
@theManikJindal | Adobe
I am HE
@theManikJindal
Computer Scientist, Adobe
P.S. You get a chocolate if you
ask a question.
What is a Distributed Linked List?
• Linked List: An ordered set of nodes, each node containing a link to the next one.
• Distributed Linked List:
• Data (nodes) is stored on multiple machines for durability and to allow
reading at scale.
• Multiple compute instances write concurrently for resiliency and to write
at scale.
What’s the problem we’re trying to solve?
• Adobe I/O Events enables our customers to create event subscriptions for events
generated by products and services across Adobe.
• Our customers can consume events via Webhook PUSH or PULL them from the
Journaling API.
• A Journal, is an ordered list of events - much like a ledger or a log where new
entries (events) are added to the end of the ledger and the ledger keeps growing.
What’s the problem we’re trying to solve?
• Each event subscription has a corresponding Journal – and every event that we
receive needs to be written to the correct Journal.
• The order of events once written into the Journal cannot be changed. Newer events
can only be added to the end. A client application reads the events in this order.
• We have millions of events per hour (and counting) and over a few thousand
subscriptions – we also need to operate in near real time.
Approach-1: We stored everything in a Database
• We stored all the events in a document database (Azure Cosmos DB).
• We ordered the events by an auto-increment id.
• We partitioned the events’ data in the database by the subscription id.
I slept at night because we did not order the events by timestamps.
Approach-1: FAILED
• Reading the events was very expensive. It required database lookups, ordering and
data transfer of a few hundred KBs per request.
• Our clients could not read events at scale - we were not able to handle more than
a dozen clients reading concurrently.
• Incident: A single client brought us down in a day by publishing large events
frequently enough. (We hit Cosmos DB’s 10 GB partition limit.)
Approach-1: Lessons Learnt
• Partition Policy: Do not partition data by something that a client has control over.
(Example: user_id, client_id, etc.)
• Databases were not the right fit for storing messages ranging from a few KBs to
hundreds of KBs.
• RTFM: Cosmos DB's pricing is based on reserved capacity in terms of a strange unit
- "RU/s". There wasn't a way to tell how much we'll need.
Approach-2: We used a mix of Database & Blob Storage
• We moved out just the event payloads to a blob storage (AWS S3), to quickly rectify
the partition limit problem.
• We also batched the event writes to reduce our storage costs.
Approach-2: Failed
• Reading events was still very expensive. It required database lookups, ordering and
then up to a hundred S3 GETs per request.
• Due to batching, writing events was actually cheaper than reading them.
• Our clients still could not read events at scale. Even after storing the event payload
separately the Database continued to be the chokepoint.
Approach-2: Lessons Learnt
• Databases were not the right fit - we never updated any records and only deleted
them based on a TTL policy.
• Because we used a managed database service, we were able to scale event reads
for the time being by throwing money at the problem.
• Batching needed to be optimized for reading and not writing. The system writes
once, clients read at least once.
Searching for a Solution: What do we know?
• A database should not be involved in the critical path of reading events. Because a
single partition will not scale and there is no partitioning policy that can distribute
loads effectively.
• Blob storage (AWS S3) is ideal for storing event payloads, but it is costly. So, we will
have to batch the events to save on costs, but we should batch them in a way that
we optimize reading costs.
Searching for a Solution: Distributed Linked List
• Let’s build a linked list of S3 objects. Each S3 object not only contains a batch of
events but also contains information to find the next S3 object on the list.
• Because each S3 object knows where the next S3 object is, we will eliminate
database from the critical path of reading events and then reading events can
potentially scale as much as S3 itself.
• Because we will batch to optimize reading costs, the cost of reading events could be
as low as S3 GET requests ($4 per 10 million requests).
Distributed Linked List – The Hard Problem
• From a system architecture standpoint, we needed multiple compute instances to
process the incoming stream of events - for both resiliency and to write events at
scale.
• However, we had a hard problem on our hands, we needed to order the writes
coming from multiple compute instances – otherwise we’d risk corrupting the data.
Well, I thought, how hard could it be? We could always acquire a lock...
Distributed Linked List – Distributed Locking
• To order the writes from multiple compute instances, we used a distributed lock on
Redis. A compute instance would acquire a lock to insert into the list.
• Turned out that distributed locks could not always guarantee mutual exclusion. If
the lock failed the results would have been catastrophic, and hence, we could not
take a chance – howsoever small.
• To rectify this we explored ZooKeeper and sought help to find a lock that could
work for us – in vain.
A good read on distributed locks by Martin Kleppmann - https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
Distributed Linked List – Zookeeper
No major cloud provider provides a managed ZooKeeper service.
If you want us to run ZooKeeper ourselves, we will need another
full time DevOps engineer.
Sathyajith Bhat
(Senior DevOps Engineer)
Distributed Linked List – Say No to Distributed Locks
The difference between ‘a distributed lock’ and ‘a distributed lock that
works in practice and is on the critical path for high throughput writes
and magically is not the bottleneck’ is ~50 person years. [sic]
Michael Marth
(Director, Engineering)
Distributed Linked List – A Lock Free Solution
• In the first step we batch and store the event payloads in blob storage (S3), and
then order the different batches using an auto-increment id in database (MySQL).
This step can be performed independently by each compute instance.
• In the second step we transfer the database order to blob storage. The algorithm to
do so is idempotent and, hence, works lock free with multiple compute instances.
We even mathematically proved the algorithm for correctness.
Inspiring Architecture Design – AWS SQS
• SQS ensures that all messages are processed by the means of client
acknowledgement. This allowed us to not worry about losing any data and we
based our internal event stream on an SQS-like queue.
• SQS also helps distributes load amongst the worker, which came in really handy for
us to scale in and out depending on the traffic.
Inspiring Algorithm Design – AWS RDS MySQL
• We love managed services and RDS was a perfect choice for us – guaranteeing both
durability and performance.
• We use an auto-increment id to order writes. Even though MySQL’s auto increment
id is very performant, we still reduced the number of times we used it. We did so by
only inserting a row in the table for a batch of events and not for every event.
• We also made sure to delete older rows that have already been processed. This way
any internal locking that MySQL performs has lesser rows to contend over.
Inspiring Algorithm Design – AWS S3
• An S3 object cannot be partially updated or appended to.
• Hence, we could not construct a linked list in the traditional sense, i.e., update the
pointer to the next node.
• Instead, we had to know the location of the next S3 object in advance.
Inspiring API Design – AWS S3
• Not only does AWS S3 charge by the number of requests, even the performance
limits on a partition are defined in terms of the number of requests/second.
• Originally, in an API call a client could specify the number of events it wanted to
consume. However, this simple feature took away our control over the number of
S3 GET requests we in-turn had to make.
• We changed the API definition so that a client can now only specify a limit on the
number of events and the API is allowed to give them a lesser number of events.
Inspiring Architecture Design – AWS S3
• AWS S3 partitions the data using object key prefixes and the performance limits on
S3 are also defined on a per partition level.
• Because we wanted to maximize the performance we spread out our object keys
and used completely random GUIDs as the object keys.
• Due to performance considerations, we also added the capability to add more S3
buckets at runtime so the system can start distributing load over them, if needed.
Results – Performance
Batching
60%
S3 Writes
8%
Database
2%
Time
Delay
30%
Average Time Taken to write a batch
of Events
~ 2 seconds
S3 Reads
67%Auth
33%
Average Time Taken to read a batch
of Events
~ 0.06
seconds
Results – Costs
11.11
4.50
10.80
26.41
0.72
7.20
0.29
8.21
0.00
5.00
10.00
15.00
20.00
25.00
30.00
DB + Redis S3 Writes S3 Reads Total
Cost of ingesting 1M events/min for 100 clients. ($/hour)
Approach-2 Now
Conclusion: What did we learn?
• Always understand the problem you’re trying to solve. Find out how the system is
intended to be used and design accordingly. Also, find out the aspects in which the
system is allowed to be less than perfect – for us it was being near real time.
• Do no think of a system architecture and then try to fit various services in it later,
rather design the architecture in the stride of the underlying components you use.
• Listen to experienced folks to cut losses early. (Distributed Locks, ZooKeeper)
Find me everywhere as
@theManikJindal
Please spare two minutes for feedback
https://tinyurl.com/tmj-dll

More Related Content

What's hot

Cloud Security Engineering - Tools and Techniques
Cloud Security Engineering - Tools and TechniquesCloud Security Engineering - Tools and Techniques
Cloud Security Engineering - Tools and TechniquesGokul Alex
 
Managing Cloud Security Risks in Your Organization
Managing Cloud Security Risks in Your OrganizationManaging Cloud Security Risks in Your Organization
Managing Cloud Security Risks in Your OrganizationCharles Lim
 
Security on Cloud Computing
Security on Cloud Computing Security on Cloud Computing
Security on Cloud Computing Reza Pahlava
 
Cloud Computing Security Organization Assessments Service Categories Responsi...
Cloud Computing Security Organization Assessments Service Categories Responsi...Cloud Computing Security Organization Assessments Service Categories Responsi...
Cloud Computing Security Organization Assessments Service Categories Responsi...SlideTeam
 
Security Issues in Cloud Computing
Security Issues in Cloud ComputingSecurity Issues in Cloud Computing
Security Issues in Cloud ComputingJyotika Pandey
 
Chap 5 software as a service (saass)
Chap 5 software as a service (saass)Chap 5 software as a service (saass)
Chap 5 software as a service (saass)Raj Sarode
 
Security in the cloud Workshop HSTC 2014
Security in the cloud Workshop HSTC 2014Security in the cloud Workshop HSTC 2014
Security in the cloud Workshop HSTC 2014Akash Mahajan
 
Cloud computing security
Cloud computing securityCloud computing security
Cloud computing securitymaheralgamdi
 
Cloud Computing Security
Cloud Computing SecurityCloud Computing Security
Cloud Computing SecurityPiyush Mittal
 
Data Security Essentials for Cloud Computing - JavaOne 2013
Data Security Essentials for Cloud Computing - JavaOne 2013Data Security Essentials for Cloud Computing - JavaOne 2013
Data Security Essentials for Cloud Computing - JavaOne 2013javagroup2006
 
Cloud Computing security issues
Cloud Computing security issuesCloud Computing security issues
Cloud Computing security issuesPradeepti Kamble
 
security Issues of cloud computing
security Issues of cloud computingsecurity Issues of cloud computing
security Issues of cloud computingprachupanchal
 
Securing Applications in the Cloud
Securing Applications in the CloudSecuring Applications in the Cloud
Securing Applications in the CloudSecurity Innovation
 
Best-Practices-Web-Usability
Best-Practices-Web-UsabilityBest-Practices-Web-Usability
Best-Practices-Web-UsabilityLarry Wilson
 
On technical security issues in cloud computing
On technical security issues in cloud computingOn technical security issues in cloud computing
On technical security issues in cloud computingsashi799
 
Cloud Computing - Security Benefits and Risks
Cloud Computing - Security Benefits and RisksCloud Computing - Security Benefits and Risks
Cloud Computing - Security Benefits and RisksWilliam McBorrough
 
Cloud Computing Security
Cloud Computing SecurityCloud Computing Security
Cloud Computing SecurityNithin Raj
 

What's hot (19)

Cloud Security Engineering - Tools and Techniques
Cloud Security Engineering - Tools and TechniquesCloud Security Engineering - Tools and Techniques
Cloud Security Engineering - Tools and Techniques
 
Cloud security
Cloud security Cloud security
Cloud security
 
Managing Cloud Security Risks in Your Organization
Managing Cloud Security Risks in Your OrganizationManaging Cloud Security Risks in Your Organization
Managing Cloud Security Risks in Your Organization
 
Security on Cloud Computing
Security on Cloud Computing Security on Cloud Computing
Security on Cloud Computing
 
Cloud Computing Security Organization Assessments Service Categories Responsi...
Cloud Computing Security Organization Assessments Service Categories Responsi...Cloud Computing Security Organization Assessments Service Categories Responsi...
Cloud Computing Security Organization Assessments Service Categories Responsi...
 
Security Issues in Cloud Computing
Security Issues in Cloud ComputingSecurity Issues in Cloud Computing
Security Issues in Cloud Computing
 
Chap 5 software as a service (saass)
Chap 5 software as a service (saass)Chap 5 software as a service (saass)
Chap 5 software as a service (saass)
 
Security in the cloud Workshop HSTC 2014
Security in the cloud Workshop HSTC 2014Security in the cloud Workshop HSTC 2014
Security in the cloud Workshop HSTC 2014
 
Cloud computing security
Cloud computing securityCloud computing security
Cloud computing security
 
Cloud Computing Security
Cloud Computing SecurityCloud Computing Security
Cloud Computing Security
 
Data Security Essentials for Cloud Computing - JavaOne 2013
Data Security Essentials for Cloud Computing - JavaOne 2013Data Security Essentials for Cloud Computing - JavaOne 2013
Data Security Essentials for Cloud Computing - JavaOne 2013
 
Cloud Computing security issues
Cloud Computing security issuesCloud Computing security issues
Cloud Computing security issues
 
security Issues of cloud computing
security Issues of cloud computingsecurity Issues of cloud computing
security Issues of cloud computing
 
Securing Applications in the Cloud
Securing Applications in the CloudSecuring Applications in the Cloud
Securing Applications in the Cloud
 
Best-Practices-Web-Usability
Best-Practices-Web-UsabilityBest-Practices-Web-Usability
Best-Practices-Web-Usability
 
Cloud security (domain11 14)
Cloud security (domain11 14)Cloud security (domain11 14)
Cloud security (domain11 14)
 
On technical security issues in cloud computing
On technical security issues in cloud computingOn technical security issues in cloud computing
On technical security issues in cloud computing
 
Cloud Computing - Security Benefits and Risks
Cloud Computing - Security Benefits and RisksCloud Computing - Security Benefits and Risks
Cloud Computing - Security Benefits and Risks
 
Cloud Computing Security
Cloud Computing SecurityCloud Computing Security
Cloud Computing Security
 

Similar to Lessons learnt building a Distributed Linked List on S3

(BDT313) Amazon DynamoDB For Big Data
(BDT313) Amazon DynamoDB For Big Data(BDT313) Amazon DynamoDB For Big Data
(BDT313) Amazon DynamoDB For Big DataAmazon Web Services
 
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Emprovise
 
Why Scale Matters and How the Cloud is Really Different (at scale)
Why Scale Matters and How the Cloud is Really Different (at scale)Why Scale Matters and How the Cloud is Really Different (at scale)
Why Scale Matters and How the Cloud is Really Different (at scale)Amazon Web Services
 
Cloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark AnalyticsCloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark Analyticsamesar0
 
Automate Your Big Data Workflows (SVC201) | AWS re:Invent 2013
Automate Your Big Data Workflows (SVC201) | AWS re:Invent 2013Automate Your Big Data Workflows (SVC201) | AWS re:Invent 2013
Automate Your Big Data Workflows (SVC201) | AWS re:Invent 2013Amazon Web Services
 
Designing your SaaS Database for Scale with Postgres
Designing your SaaS Database for Scale with PostgresDesigning your SaaS Database for Scale with Postgres
Designing your SaaS Database for Scale with PostgresOzgun Erdogan
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2kartraj
 
(SPOT305) Event-Driven Computing on Change Logs in AWS | AWS re:Invent 2014
(SPOT305) Event-Driven Computing on Change Logs in AWS | AWS re:Invent 2014(SPOT305) Event-Driven Computing on Change Logs in AWS | AWS re:Invent 2014
(SPOT305) Event-Driven Computing on Change Logs in AWS | AWS re:Invent 2014Amazon Web Services
 
AWS Canberra WWPS Summit 2013 - AWS for Web Applications
AWS Canberra WWPS Summit 2013 - AWS for Web ApplicationsAWS Canberra WWPS Summit 2013 - AWS for Web Applications
AWS Canberra WWPS Summit 2013 - AWS for Web ApplicationsAmazon Web Services
 
Introduction to AWS Glue: Data Analytics Week at the SF Loft
Introduction to AWS Glue: Data Analytics Week at the SF LoftIntroduction to AWS Glue: Data Analytics Week at the SF Loft
Introduction to AWS Glue: Data Analytics Week at the SF LoftAmazon Web Services
 
AWS Big Data in everyday use at Yle
AWS Big Data in everyday use at YleAWS Big Data in everyday use at Yle
AWS Big Data in everyday use at YleRolf Koski
 
Data warehousing in the era of Big Data: Deep Dive into Amazon Redshift
Data warehousing in the era of Big Data: Deep Dive into Amazon RedshiftData warehousing in the era of Big Data: Deep Dive into Amazon Redshift
Data warehousing in the era of Big Data: Deep Dive into Amazon RedshiftAmazon Web Services
 
Getting Started with Amazon Redshift
Getting Started with Amazon RedshiftGetting Started with Amazon Redshift
Getting Started with Amazon RedshiftAmazon Web Services
 
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWSAWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWSAmazon Web Services
 
Deep Dive: Amazon Elastic MapReduce
Deep Dive: Amazon Elastic MapReduceDeep Dive: Amazon Elastic MapReduce
Deep Dive: Amazon Elastic MapReduceAmazon Web Services
 
Architecting applications in the AWS cloud
Architecting applications in the AWS cloudArchitecting applications in the AWS cloud
Architecting applications in the AWS cloudCloud Genius
 
Serverless at Lifestage
Serverless at LifestageServerless at Lifestage
Serverless at LifestageBATbern
 

Similar to Lessons learnt building a Distributed Linked List on S3 (20)

(BDT313) Amazon DynamoDB For Big Data
(BDT313) Amazon DynamoDB For Big Data(BDT313) Amazon DynamoDB For Big Data
(BDT313) Amazon DynamoDB For Big Data
 
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
 
BDA311 Introduction to AWS Glue
BDA311 Introduction to AWS GlueBDA311 Introduction to AWS Glue
BDA311 Introduction to AWS Glue
 
Real-Time Event Processing
Real-Time Event ProcessingReal-Time Event Processing
Real-Time Event Processing
 
Why Scale Matters and How the Cloud is Really Different (at scale)
Why Scale Matters and How the Cloud is Really Different (at scale)Why Scale Matters and How the Cloud is Really Different (at scale)
Why Scale Matters and How the Cloud is Really Different (at scale)
 
Cloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark AnalyticsCloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark Analytics
 
Automate Your Big Data Workflows (SVC201) | AWS re:Invent 2013
Automate Your Big Data Workflows (SVC201) | AWS re:Invent 2013Automate Your Big Data Workflows (SVC201) | AWS re:Invent 2013
Automate Your Big Data Workflows (SVC201) | AWS re:Invent 2013
 
Create cloud service on AWS
Create cloud service on AWSCreate cloud service on AWS
Create cloud service on AWS
 
Designing your SaaS Database for Scale with Postgres
Designing your SaaS Database for Scale with PostgresDesigning your SaaS Database for Scale with Postgres
Designing your SaaS Database for Scale with Postgres
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2
 
(SPOT305) Event-Driven Computing on Change Logs in AWS | AWS re:Invent 2014
(SPOT305) Event-Driven Computing on Change Logs in AWS | AWS re:Invent 2014(SPOT305) Event-Driven Computing on Change Logs in AWS | AWS re:Invent 2014
(SPOT305) Event-Driven Computing on Change Logs in AWS | AWS re:Invent 2014
 
AWS Canberra WWPS Summit 2013 - AWS for Web Applications
AWS Canberra WWPS Summit 2013 - AWS for Web ApplicationsAWS Canberra WWPS Summit 2013 - AWS for Web Applications
AWS Canberra WWPS Summit 2013 - AWS for Web Applications
 
Introduction to AWS Glue: Data Analytics Week at the SF Loft
Introduction to AWS Glue: Data Analytics Week at the SF LoftIntroduction to AWS Glue: Data Analytics Week at the SF Loft
Introduction to AWS Glue: Data Analytics Week at the SF Loft
 
AWS Big Data in everyday use at Yle
AWS Big Data in everyday use at YleAWS Big Data in everyday use at Yle
AWS Big Data in everyday use at Yle
 
Data warehousing in the era of Big Data: Deep Dive into Amazon Redshift
Data warehousing in the era of Big Data: Deep Dive into Amazon RedshiftData warehousing in the era of Big Data: Deep Dive into Amazon Redshift
Data warehousing in the era of Big Data: Deep Dive into Amazon Redshift
 
Getting Started with Amazon Redshift
Getting Started with Amazon RedshiftGetting Started with Amazon Redshift
Getting Started with Amazon Redshift
 
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWSAWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
 
Deep Dive: Amazon Elastic MapReduce
Deep Dive: Amazon Elastic MapReduceDeep Dive: Amazon Elastic MapReduce
Deep Dive: Amazon Elastic MapReduce
 
Architecting applications in the AWS cloud
Architecting applications in the AWS cloudArchitecting applications in the AWS cloud
Architecting applications in the AWS cloud
 
Serverless at Lifestage
Serverless at LifestageServerless at Lifestage
Serverless at Lifestage
 

More from AWS User Group Bengaluru

Building Efficient, Scalable and Resilient Front-end logging service with AWS
Building Efficient, Scalable and Resilient Front-end logging service with AWSBuilding Efficient, Scalable and Resilient Front-end logging service with AWS
Building Efficient, Scalable and Resilient Front-end logging service with AWSAWS User Group Bengaluru
 
Exploring opportunities with communities for a successful career
Exploring opportunities with communities for a successful careerExploring opportunities with communities for a successful career
Exploring opportunities with communities for a successful careerAWS User Group Bengaluru
 
Slack's transition away from a single AWS account
Slack's transition away from a single AWS accountSlack's transition away from a single AWS account
Slack's transition away from a single AWS accountAWS User Group Bengaluru
 
Building Efficient, Scalable and Resilient Front-end logging service with AWS
Building Efficient, Scalable and Resilient Front-end logging service with AWSBuilding Efficient, Scalable and Resilient Front-end logging service with AWS
Building Efficient, Scalable and Resilient Front-end logging service with AWSAWS User Group Bengaluru
 
Medlife's journey with AWS from 0(zero) orders to 6 digit mark
Medlife's journey with AWS from 0(zero) orders to 6 digit markMedlife's journey with AWS from 0(zero) orders to 6 digit mark
Medlife's journey with AWS from 0(zero) orders to 6 digit markAWS User Group Bengaluru
 
Exploring opportunities with communities for a successful career
Exploring opportunities with communities for a successful careerExploring opportunities with communities for a successful career
Exploring opportunities with communities for a successful careerAWS User Group Bengaluru
 
Lessons learnt building a Distributed Linked List on S3
Lessons learnt building a Distributed Linked List on S3Lessons learnt building a Distributed Linked List on S3
Lessons learnt building a Distributed Linked List on S3AWS User Group Bengaluru
 
Keynote - Chaos Engineering: Why breaking things should be practiced
Keynote - Chaos Engineering: Why breaking things should be practicedKeynote - Chaos Engineering: Why breaking things should be practiced
Keynote - Chaos Engineering: Why breaking things should be practicedAWS User Group Bengaluru
 
Decentralized enterprise architecture using Blockchain & AWS
Decentralized enterprise architecture using Blockchain & AWSDecentralized enterprise architecture using Blockchain & AWS
Decentralized enterprise architecture using Blockchain & AWSAWS User Group Bengaluru
 

More from AWS User Group Bengaluru (20)

Demystifying identity on AWS
Demystifying identity on AWSDemystifying identity on AWS
Demystifying identity on AWS
 
AWS Secrets for Best Practices
AWS Secrets for Best PracticesAWS Secrets for Best Practices
AWS Secrets for Best Practices
 
Cloud Security
Cloud SecurityCloud Security
Cloud Security
 
Medlife journey with AWS
Medlife journey with AWSMedlife journey with AWS
Medlife journey with AWS
 
Building Efficient, Scalable and Resilient Front-end logging service with AWS
Building Efficient, Scalable and Resilient Front-end logging service with AWSBuilding Efficient, Scalable and Resilient Front-end logging service with AWS
Building Efficient, Scalable and Resilient Front-end logging service with AWS
 
Exploring opportunities with communities for a successful career
Exploring opportunities with communities for a successful careerExploring opportunities with communities for a successful career
Exploring opportunities with communities for a successful career
 
Slack's transition away from a single AWS account
Slack's transition away from a single AWS accountSlack's transition away from a single AWS account
Slack's transition away from a single AWS account
 
Log analytics with ELK stack
Log analytics with ELK stackLog analytics with ELK stack
Log analytics with ELK stack
 
Serverless Culture
Serverless CultureServerless Culture
Serverless Culture
 
Refactoring to serverless
Refactoring to serverlessRefactoring to serverless
Refactoring to serverless
 
Amazon EC2 Spot Instances Workshop
Amazon EC2 Spot Instances WorkshopAmazon EC2 Spot Instances Workshop
Amazon EC2 Spot Instances Workshop
 
Building Efficient, Scalable and Resilient Front-end logging service with AWS
Building Efficient, Scalable and Resilient Front-end logging service with AWSBuilding Efficient, Scalable and Resilient Front-end logging service with AWS
Building Efficient, Scalable and Resilient Front-end logging service with AWS
 
Medlife's journey with AWS from 0(zero) orders to 6 digit mark
Medlife's journey with AWS from 0(zero) orders to 6 digit markMedlife's journey with AWS from 0(zero) orders to 6 digit mark
Medlife's journey with AWS from 0(zero) orders to 6 digit mark
 
AWS Secrets for Best Practices
AWS Secrets for Best PracticesAWS Secrets for Best Practices
AWS Secrets for Best Practices
 
Exploring opportunities with communities for a successful career
Exploring opportunities with communities for a successful careerExploring opportunities with communities for a successful career
Exploring opportunities with communities for a successful career
 
Lessons learnt building a Distributed Linked List on S3
Lessons learnt building a Distributed Linked List on S3Lessons learnt building a Distributed Linked List on S3
Lessons learnt building a Distributed Linked List on S3
 
Amazon EC2 Spot Instances
Amazon EC2 Spot InstancesAmazon EC2 Spot Instances
Amazon EC2 Spot Instances
 
Cost Optimization in AWS
Cost Optimization in AWSCost Optimization in AWS
Cost Optimization in AWS
 
Keynote - Chaos Engineering: Why breaking things should be practiced
Keynote - Chaos Engineering: Why breaking things should be practicedKeynote - Chaos Engineering: Why breaking things should be practiced
Keynote - Chaos Engineering: Why breaking things should be practiced
 
Decentralized enterprise architecture using Blockchain & AWS
Decentralized enterprise architecture using Blockchain & AWSDecentralized enterprise architecture using Blockchain & AWS
Decentralized enterprise architecture using Blockchain & AWS
 

Recently uploaded

Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekCzechDreamin
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka DoktorováCzechDreamin
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty SecureFemke de Vroome
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKUXDXConf
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsUXDXConf
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyJohn Staveley
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxJennifer Lim
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyUXDXConf
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...marcuskenyatta275
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCzechDreamin
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGDSC PJATK
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 

Recently uploaded (20)

Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering Teams
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 

Lessons learnt building a Distributed Linked List on S3

  • 1. Lessons Learnt building a Distributed Linked List on S3 @theManikJindal | Adobe
  • 2. I am HE @theManikJindal Computer Scientist, Adobe P.S. You get a chocolate if you ask a question.
  • 3. What is a Distributed Linked List? • Linked List: An ordered set of nodes, each node containing a link to the next one. • Distributed Linked List: • Data (nodes) is stored on multiple machines for durability and to allow reading at scale. • Multiple compute instances write concurrently for resiliency and to write at scale.
  • 4. What’s the problem we’re trying to solve? • Adobe I/O Events enables our customers to create event subscriptions for events generated by products and services across Adobe. • Our customers can consume events via Webhook PUSH or PULL them from the Journaling API. • A Journal, is an ordered list of events - much like a ledger or a log where new entries (events) are added to the end of the ledger and the ledger keeps growing.
  • 5. What’s the problem we’re trying to solve? • Each event subscription has a corresponding Journal – and every event that we receive needs to be written to the correct Journal. • The order of events once written into the Journal cannot be changed. Newer events can only be added to the end. A client application reads the events in this order. • We have millions of events per hour (and counting) and over a few thousand subscriptions – we also need to operate in near real time.
  • 6. Approach-1: We stored everything in a Database • We stored all the events in a document database (Azure Cosmos DB). • We ordered the events by an auto-increment id. • We partitioned the events’ data in the database by the subscription id. I slept at night because we did not order the events by timestamps.
  • 7. Approach-1: FAILED • Reading the events was very expensive. It required database lookups, ordering and data transfer of a few hundred KBs per request. • Our clients could not read events at scale - we were not able to handle more than a dozen clients reading concurrently. • Incident: A single client brought us down in a day by publishing large events frequently enough. (We hit Cosmos DB’s 10 GB partition limit.)
  • 8. Approach-1: Lessons Learnt • Partition Policy: Do not partition data by something that a client has control over. (Example: user_id, client_id, etc.) • Databases were not the right fit for storing messages ranging from a few KBs to hundreds of KBs. • RTFM: Cosmos DB's pricing is based on reserved capacity in terms of a strange unit - "RU/s". There wasn't a way to tell how much we'll need.
  • 9. Approach-2: We used a mix of Database & Blob Storage • We moved out just the event payloads to a blob storage (AWS S3), to quickly rectify the partition limit problem. • We also batched the event writes to reduce our storage costs.
  • 10. Approach-2: Failed • Reading events was still very expensive. It required database lookups, ordering and then up to a hundred S3 GETs per request. • Due to batching, writing events was actually cheaper than reading them. • Our clients still could not read events at scale. Even after storing the event payload separately the Database continued to be the chokepoint.
  • 11. Approach-2: Lessons Learnt • Databases were not the right fit - we never updated any records and only deleted them based on a TTL policy. • Because we used a managed database service, we were able to scale event reads for the time being by throwing money at the problem. • Batching needed to be optimized for reading and not writing. The system writes once, clients read at least once.
  • 12. Searching for a Solution: What do we know? • A database should not be involved in the critical path of reading events. Because a single partition will not scale and there is no partitioning policy that can distribute loads effectively. • Blob storage (AWS S3) is ideal for storing event payloads, but it is costly. So, we will have to batch the events to save on costs, but we should batch them in a way that we optimize reading costs.
  • 13. Searching for a Solution: Distributed Linked List • Let’s build a linked list of S3 objects. Each S3 object not only contains a batch of events but also contains information to find the next S3 object on the list. • Because each S3 object knows where the next S3 object is, we will eliminate database from the critical path of reading events and then reading events can potentially scale as much as S3 itself. • Because we will batch to optimize reading costs, the cost of reading events could be as low as S3 GET requests ($4 per 10 million requests).
  • 14. Distributed Linked List – The Hard Problem • From a system architecture standpoint, we needed multiple compute instances to process the incoming stream of events - for both resiliency and to write events at scale. • However, we had a hard problem on our hands, we needed to order the writes coming from multiple compute instances – otherwise we’d risk corrupting the data. Well, I thought, how hard could it be? We could always acquire a lock...
  • 15. Distributed Linked List – Distributed Locking • To order the writes from multiple compute instances, we used a distributed lock on Redis. A compute instance would acquire a lock to insert into the list. • Turned out that distributed locks could not always guarantee mutual exclusion. If the lock failed the results would have been catastrophic, and hence, we could not take a chance – howsoever small. • To rectify this we explored ZooKeeper and sought help to find a lock that could work for us – in vain. A good read on distributed locks by Martin Kleppmann - https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
  • 16. Distributed Linked List – Zookeeper No major cloud provider provides a managed ZooKeeper service. If you want us to run ZooKeeper ourselves, we will need another full time DevOps engineer. Sathyajith Bhat (Senior DevOps Engineer)
  • 17. Distributed Linked List – Say No to Distributed Locks The difference between ‘a distributed lock’ and ‘a distributed lock that works in practice and is on the critical path for high throughput writes and magically is not the bottleneck’ is ~50 person years. [sic] Michael Marth (Director, Engineering)
  • 18. Distributed Linked List – A Lock Free Solution • In the first step we batch and store the event payloads in blob storage (S3), and then order the different batches using an auto-increment id in database (MySQL). This step can be performed independently by each compute instance. • In the second step we transfer the database order to blob storage. The algorithm to do so is idempotent and, hence, works lock free with multiple compute instances. We even mathematically proved the algorithm for correctness.
  • 19. Inspiring Architecture Design – AWS SQS • SQS ensures that all messages are processed by the means of client acknowledgement. This allowed us to not worry about losing any data and we based our internal event stream on an SQS-like queue. • SQS also helps distributes load amongst the worker, which came in really handy for us to scale in and out depending on the traffic.
  • 20. Inspiring Algorithm Design – AWS RDS MySQL • We love managed services and RDS was a perfect choice for us – guaranteeing both durability and performance. • We use an auto-increment id to order writes. Even though MySQL’s auto increment id is very performant, we still reduced the number of times we used it. We did so by only inserting a row in the table for a batch of events and not for every event. • We also made sure to delete older rows that have already been processed. This way any internal locking that MySQL performs has lesser rows to contend over.
  • 21. Inspiring Algorithm Design – AWS S3 • An S3 object cannot be partially updated or appended to. • Hence, we could not construct a linked list in the traditional sense, i.e., update the pointer to the next node. • Instead, we had to know the location of the next S3 object in advance.
  • 22. Inspiring API Design – AWS S3 • Not only does AWS S3 charge by the number of requests, even the performance limits on a partition are defined in terms of the number of requests/second. • Originally, in an API call a client could specify the number of events it wanted to consume. However, this simple feature took away our control over the number of S3 GET requests we in-turn had to make. • We changed the API definition so that a client can now only specify a limit on the number of events and the API is allowed to give them a lesser number of events.
  • 23. Inspiring Architecture Design – AWS S3 • AWS S3 partitions the data using object key prefixes and the performance limits on S3 are also defined on a per partition level. • Because we wanted to maximize the performance we spread out our object keys and used completely random GUIDs as the object keys. • Due to performance considerations, we also added the capability to add more S3 buckets at runtime so the system can start distributing load over them, if needed.
  • 24. Results – Performance Batching 60% S3 Writes 8% Database 2% Time Delay 30% Average Time Taken to write a batch of Events ~ 2 seconds S3 Reads 67%Auth 33% Average Time Taken to read a batch of Events ~ 0.06 seconds
  • 25. Results – Costs 11.11 4.50 10.80 26.41 0.72 7.20 0.29 8.21 0.00 5.00 10.00 15.00 20.00 25.00 30.00 DB + Redis S3 Writes S3 Reads Total Cost of ingesting 1M events/min for 100 clients. ($/hour) Approach-2 Now
  • 26. Conclusion: What did we learn? • Always understand the problem you’re trying to solve. Find out how the system is intended to be used and design accordingly. Also, find out the aspects in which the system is allowed to be less than perfect – for us it was being near real time. • Do no think of a system architecture and then try to fit various services in it later, rather design the architecture in the stride of the underlying components you use. • Listen to experienced folks to cut losses early. (Distributed Locks, ZooKeeper)
  • 27. Find me everywhere as @theManikJindal Please spare two minutes for feedback https://tinyurl.com/tmj-dll