Dynamo is a highly available key-value storage system built by Amazon to power its e-commerce platform. It uses consistent hashing to partition data across nodes in a ring topology and achieves high availability of writes through techniques like vector clocks, hinted handoff, and quorums. Dynamo provides simple interfaces to store and retrieve data identified by unique keys at massive scales with low latency despite failures through an eventually consistent model.
High Performance Stream Processing and OptimizationsFarley Lai
Building scalable systems that process streams of data requires developers to take advantage of the parallelism capabilities offered by today's computer architectures. Existing imperative programming languages provide programmers low-level primitives such as threads, locks, and semaphores. However, programs developed using these primitives tend to be plagued by race conditions and deadlocks, which make it hard to understand and debug non-deterministic behaviors. Acknowledging these limitations, some programming language extensions and libraries (e.g., OpenMPI, OpenMP have been proposed to simplify programming parallel programs. Nevertheless, all these options still burden the programmer with annotating parallelism and specifying data sharing attributes to ensure data consistency.
In recent years, dataflows have attracted significant attention as a model for building highly parallel stream processing applications. According to this model, an application is defined as a graph of processing elements that are connected by communication channels. The processing elements may execute in parallel as long as they have sufficient data to process. A key feature of the dataflow model is that it explicitly capture parallelism and data dependencies between processing elements.
Even though the dataflows provide a simple computational model, using this model to build scalable systems is challenging as naive implementations introduce unexpected runtime scheduling overhead, consume significant memory resources, and are not energy efficient. Consequently, our goal is to develop compiler optimizations and efficient runtime environments for scalable dataflow systems. In the following, we will go through the model of computation, memory optimizations, energy efficiency and stream processing at the scale of cloud computing.
Static Memory Management for Efficient Mobile Sensing ApplicationsFarley Lai
Memory management is a crucial aspect of mobile sensing applications that must process high-rate data streams in an energy-efficient manner. Our work is done in the context of synchronous data-flow models in which applications are implemented as a graph of components that exchange data at fixed and known rates over FIFO channels. In this paper, we show that it is feasible to leverage the restricted semantics of synchronous data-flow models to optimize memory management. Our memory optimization approach includes two components: (1) We use abstract interpretation to analyze the complete memory behavior of a mobile sensing application and identify data sharing opportunities across components according to the live ranges of exchanged samples. Experiments indicate that the static analysis is precise for a majority of considered stream applications whose control logic does not depend on input data. (2) We propose novel heuristics for memory allocation that leverage the graph structure of applications to optimize data exchanges between application components to achieve not only significantly lower memory footprints but also increased stream processing throughput.
Programming is hard. Programming correct C and C++ is particularly hard. Indeed, both in C and certainly in C++, it is uncommon to see a screenful containing only well defined and conforming code.Why do professional programmers write code like this? Because most programmers do not have a deep understanding of the language they are using.While they sometimes know that certain things are undefined or unspecified, they often do not know why it is so. In these slides we will study small code snippets in C and C++, and use them to discuss the fundamental building blocks, limitations and underlying design philosophies of these wonderful but dangerous programming languages.
This content has a CC license. Feel free to use it for whatever you want. You may download the original PDF file from: http://www.pvv.org/~oma/DeepC_slides_oct2012.pdf
High Performance Stream Processing and OptimizationsFarley Lai
Building scalable systems that process streams of data requires developers to take advantage of the parallelism capabilities offered by today's computer architectures. Existing imperative programming languages provide programmers low-level primitives such as threads, locks, and semaphores. However, programs developed using these primitives tend to be plagued by race conditions and deadlocks, which make it hard to understand and debug non-deterministic behaviors. Acknowledging these limitations, some programming language extensions and libraries (e.g., OpenMPI, OpenMP have been proposed to simplify programming parallel programs. Nevertheless, all these options still burden the programmer with annotating parallelism and specifying data sharing attributes to ensure data consistency.
In recent years, dataflows have attracted significant attention as a model for building highly parallel stream processing applications. According to this model, an application is defined as a graph of processing elements that are connected by communication channels. The processing elements may execute in parallel as long as they have sufficient data to process. A key feature of the dataflow model is that it explicitly capture parallelism and data dependencies between processing elements.
Even though the dataflows provide a simple computational model, using this model to build scalable systems is challenging as naive implementations introduce unexpected runtime scheduling overhead, consume significant memory resources, and are not energy efficient. Consequently, our goal is to develop compiler optimizations and efficient runtime environments for scalable dataflow systems. In the following, we will go through the model of computation, memory optimizations, energy efficiency and stream processing at the scale of cloud computing.
Static Memory Management for Efficient Mobile Sensing ApplicationsFarley Lai
Memory management is a crucial aspect of mobile sensing applications that must process high-rate data streams in an energy-efficient manner. Our work is done in the context of synchronous data-flow models in which applications are implemented as a graph of components that exchange data at fixed and known rates over FIFO channels. In this paper, we show that it is feasible to leverage the restricted semantics of synchronous data-flow models to optimize memory management. Our memory optimization approach includes two components: (1) We use abstract interpretation to analyze the complete memory behavior of a mobile sensing application and identify data sharing opportunities across components according to the live ranges of exchanged samples. Experiments indicate that the static analysis is precise for a majority of considered stream applications whose control logic does not depend on input data. (2) We propose novel heuristics for memory allocation that leverage the graph structure of applications to optimize data exchanges between application components to achieve not only significantly lower memory footprints but also increased stream processing throughput.
Programming is hard. Programming correct C and C++ is particularly hard. Indeed, both in C and certainly in C++, it is uncommon to see a screenful containing only well defined and conforming code.Why do professional programmers write code like this? Because most programmers do not have a deep understanding of the language they are using.While they sometimes know that certain things are undefined or unspecified, they often do not know why it is so. In these slides we will study small code snippets in C and C++, and use them to discuss the fundamental building blocks, limitations and underlying design philosophies of these wonderful but dangerous programming languages.
This content has a CC license. Feel free to use it for whatever you want. You may download the original PDF file from: http://www.pvv.org/~oma/DeepC_slides_oct2012.pdf
Knockout is a JavaScript library that helps you to create responsive display(UI)
It is based on Model–view–viewmodel (MVVM) pattern
It provides a simple two-way data binding mechanism between your data model and UI
It was developed and is maintained as an open source project by Steve Sanderson, a Microsoft employee on July 5, 2010
Casandra is a open-source, distributed, highly scalable and fault-tolerant database. It is a best choice for managing structured, semi-structured or unstructured data at a large amount.
CouchDB has several features that help it stand out from the other databases in this rapidly growing field. Incremental map/reduce, peer to peer replication, mobile device synchronization, a realtime update feed, and the ability to host an application in the database itself (also known as a Couchapp) are just a few. See how companies such as the BBC, Radical Dynamic, Signal, and Incandescent Software are using CouchDB to solve their real world challenges.
The Google Chubby lock service for loosely-coupled distributed systemsRomain Jacotin
The Google Chubby lock service presented in 2006 is the inspiration for Apache ZooKeeper: let's take a deep dive into Chubby to better understand ZooKeeper and distributed consensus.
Dynamo Amazon’s Highly Available Key-value Store Giuseppe D.docxjacksnathalie
Dynamo: Amazon’s Highly Available Key-value Store
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati,
Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall
and Werner Vogels
Amazon.com
ABSTRACT
Reliability at massive scale is one of the biggest challenges we
face at Amazon.com, one of the largest e-commerce operations in
the world; even the slightest outage has significant financial
consequences and impacts customer trust. The Amazon.com
platform, which provides services for many web sites worldwide,
is implemented on top of an infrastructure of tens of thousands of
servers and network components located in many datacenters
around the world. At this scale, small and large components fail
continuously and the way persistent state is managed in the face
of these failures drives the reliability and scalability of the
software systems.
This paper presents the design and implementation of Dynamo, a
highly available key-value storage system that some of Amazon’s
core services use to provide an “always-on” experience. To
achieve this level of availability, Dynamo sacrifices consistency
under certain failure scenarios. It makes extensive use of object
versioning and application-assisted conflict resolution in a manner
that provides a novel interface for developers to use.
Categories and Subject Descriptors
D.4.2 [Operating Systems]: Storage Management; D.4.5
[Operating Systems]: Reliability; D.4.2 [Operating Systems]:
Performance;
General Terms
Algorithms, Management, Measurement, Performance, Design,
Reliability.
1. INTRODUCTION
Amazon runs a world-wide e-commerce platform that serves tens
of millions customers at peak times using tens of thousands of
servers located in many data centers around the world. There are
strict operational requirements on Amazon’s platform in terms of
performance, reliability and efficiency, and to support continuous
growth the platform needs to be highly scalable. Reliability is one
of the most important requirements because even the slightest
outage has significant financial consequences and impacts
customer trust. In addition, to support continuous growth, the
platform needs to be highly scalable.
One of the lessons our organization has learned from operating
Amazon’s platform is that the reliability and scalability of a
system is dependent on how its application state is managed.
Amazon uses a highly decentralized, loosely coupled, service
oriented architecture consisting of hundreds of services. In this
environment there is a particular need for storage technologies
that are always available. For example, customers should be able
to view and add items to their shopping cart even if disks are
failing, network routes are flapping, or data centers are being
destroyed by tornados. Therefore, the service responsible for
managing shopping carts requires that it can always write to and
read from its data store, and ...
Moving Forward Faster: How Monash University Automated Data on AWS with Commv...Amazon Web Services
Monash University, supporting 70,000+ students across 150 fields with over 1000 total IP professionals, sought to empower its central IT department and enhance organizational efficiency by modernizing its infrastructure. To do so, Monash set an aggressive timeline to move more than 150 terabytes of data from its on-premises datacenter – including 3,000+ virtual machines – to the AWS Cloud. Constrained by its existing data management solution, Monash University implemented Commvault on AWS for a simplified, comprehensive data management solution.
Graph Data: a New Data Management FrontierDemai Ni
Graph Data: a New Data Management Frontier -- Huawei’s view and Call for Collaboration by Demai Ni:
Huawei provides Enterprise Databases, and are actively exploring the latest technology to provide end-to-end Data Management Solution on Cloud. We are looking at to bridge classic RDMS to Graph Database on a distributed platform.
Critical Preflight Checks for Your EPM ApplicationsDatavail
The environment which houses your business critical EPM applications is complex.
Maybe as complex as the cockpit of an aircraft. Just as a pilot might not be able to build or fix everything on their plane, you might be using applications but not know how to build or fix everything that’s being used. This shouldn’t stop you from doing a pre-flight check to ensure that all your Hyperion systems are running properly and set for you and your end users.
Let’s talk about some different strategies to achieve this and give you the confidence in your systems so that you can know when things are running well—or more importantly, when they need attention before takeoff.
These are slides of the session that Jim Liddle gave at GigaSpaces Cloud Crowd event in the UK on 11th November 2009.
These slides concentrate on GigaSpaces VMWARE integration and the value proposition for using GigaSpaces for Private Clouds.
Knockout is a JavaScript library that helps you to create responsive display(UI)
It is based on Model–view–viewmodel (MVVM) pattern
It provides a simple two-way data binding mechanism between your data model and UI
It was developed and is maintained as an open source project by Steve Sanderson, a Microsoft employee on July 5, 2010
Casandra is a open-source, distributed, highly scalable and fault-tolerant database. It is a best choice for managing structured, semi-structured or unstructured data at a large amount.
CouchDB has several features that help it stand out from the other databases in this rapidly growing field. Incremental map/reduce, peer to peer replication, mobile device synchronization, a realtime update feed, and the ability to host an application in the database itself (also known as a Couchapp) are just a few. See how companies such as the BBC, Radical Dynamic, Signal, and Incandescent Software are using CouchDB to solve their real world challenges.
The Google Chubby lock service for loosely-coupled distributed systemsRomain Jacotin
The Google Chubby lock service presented in 2006 is the inspiration for Apache ZooKeeper: let's take a deep dive into Chubby to better understand ZooKeeper and distributed consensus.
Dynamo Amazon’s Highly Available Key-value Store Giuseppe D.docxjacksnathalie
Dynamo: Amazon’s Highly Available Key-value Store
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati,
Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall
and Werner Vogels
Amazon.com
ABSTRACT
Reliability at massive scale is one of the biggest challenges we
face at Amazon.com, one of the largest e-commerce operations in
the world; even the slightest outage has significant financial
consequences and impacts customer trust. The Amazon.com
platform, which provides services for many web sites worldwide,
is implemented on top of an infrastructure of tens of thousands of
servers and network components located in many datacenters
around the world. At this scale, small and large components fail
continuously and the way persistent state is managed in the face
of these failures drives the reliability and scalability of the
software systems.
This paper presents the design and implementation of Dynamo, a
highly available key-value storage system that some of Amazon’s
core services use to provide an “always-on” experience. To
achieve this level of availability, Dynamo sacrifices consistency
under certain failure scenarios. It makes extensive use of object
versioning and application-assisted conflict resolution in a manner
that provides a novel interface for developers to use.
Categories and Subject Descriptors
D.4.2 [Operating Systems]: Storage Management; D.4.5
[Operating Systems]: Reliability; D.4.2 [Operating Systems]:
Performance;
General Terms
Algorithms, Management, Measurement, Performance, Design,
Reliability.
1. INTRODUCTION
Amazon runs a world-wide e-commerce platform that serves tens
of millions customers at peak times using tens of thousands of
servers located in many data centers around the world. There are
strict operational requirements on Amazon’s platform in terms of
performance, reliability and efficiency, and to support continuous
growth the platform needs to be highly scalable. Reliability is one
of the most important requirements because even the slightest
outage has significant financial consequences and impacts
customer trust. In addition, to support continuous growth, the
platform needs to be highly scalable.
One of the lessons our organization has learned from operating
Amazon’s platform is that the reliability and scalability of a
system is dependent on how its application state is managed.
Amazon uses a highly decentralized, loosely coupled, service
oriented architecture consisting of hundreds of services. In this
environment there is a particular need for storage technologies
that are always available. For example, customers should be able
to view and add items to their shopping cart even if disks are
failing, network routes are flapping, or data centers are being
destroyed by tornados. Therefore, the service responsible for
managing shopping carts requires that it can always write to and
read from its data store, and ...
Moving Forward Faster: How Monash University Automated Data on AWS with Commv...Amazon Web Services
Monash University, supporting 70,000+ students across 150 fields with over 1000 total IP professionals, sought to empower its central IT department and enhance organizational efficiency by modernizing its infrastructure. To do so, Monash set an aggressive timeline to move more than 150 terabytes of data from its on-premises datacenter – including 3,000+ virtual machines – to the AWS Cloud. Constrained by its existing data management solution, Monash University implemented Commvault on AWS for a simplified, comprehensive data management solution.
Graph Data: a New Data Management FrontierDemai Ni
Graph Data: a New Data Management Frontier -- Huawei’s view and Call for Collaboration by Demai Ni:
Huawei provides Enterprise Databases, and are actively exploring the latest technology to provide end-to-end Data Management Solution on Cloud. We are looking at to bridge classic RDMS to Graph Database on a distributed platform.
Critical Preflight Checks for Your EPM ApplicationsDatavail
The environment which houses your business critical EPM applications is complex.
Maybe as complex as the cockpit of an aircraft. Just as a pilot might not be able to build or fix everything on their plane, you might be using applications but not know how to build or fix everything that’s being used. This shouldn’t stop you from doing a pre-flight check to ensure that all your Hyperion systems are running properly and set for you and your end users.
Let’s talk about some different strategies to achieve this and give you the confidence in your systems so that you can know when things are running well—or more importantly, when they need attention before takeoff.
These are slides of the session that Jim Liddle gave at GigaSpaces Cloud Crowd event in the UK on 11th November 2009.
These slides concentrate on GigaSpaces VMWARE integration and the value proposition for using GigaSpaces for Private Clouds.
Couchbase and Talena cohost a webinar covering a number of critical data management topics including:
- Key points to consider when securing Couchbase data assets against accidental data loss
- How to ensure compliance and security of PII and other sensitive data across replicated data sets
- Specific architectural considerations to ensure successful deployment and data management strategies in the Cloud
Computational Patterns of the Cloud - QCon NYC 2014Ines Sombra
The Cloud has undoubtedly changed the way we think about computing, IT operations, innovation, and entrepreneurship. But what are the computational patterns that have emerged from the pervasiveness of public clouds? What can we leverage to improve our organizations? And what are the challenges that we face going forward?
In this talk, I will introduce you to cloud computing’s paradigms and discuss their applications with practical examples from Engine Yard’s customers, peers, and partners. We will also cover antipatterns and myths. If you are curious about Cloud computing or want to improve your cloud strategy this talk is for you.
NOTE: Open an issue if you want me to explain something in more detail at the accompanying github repo: https://github.com/Randommood/QConNYC2014/
BluePi has done numerous migrations for large enterprises and SMBs alike. Based on this experience we have documented the considerations an organization needs to make before embarking on their journey to the cloud. Feel free to download - http://bluepiit.com/white-paper/
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Amazon Dynamo
1. Dynamo: Amazon’s Highly Available Key-Value Store
Farley Lai
University of Iowa
poyuan-lai@uiowa.edu
February 21, 2014
Farley Lai (UIOWA)
Amazon Dynamo (Big Data)
February 21, 2014
1 / 14
2. Motivation
MapReduce processes big data in a parallel and distributed fashion.
Daynamo forms the foundation of big data, namely, the storage.
Shopping Cart
Clients tend to insert and update items frequenty but review the cart to
check out only at the end. Is it fun for the sytem to always ask you to
retry later in minutes whenever there is an item inserted/updated in the
shopping cart?
Farley Lai (UIOWA)
Amazon Dynamo (Big Data)
February 21, 2014
2 / 14
3. SOA of Amazon’s Platform
Farley Lai (UIOWA)
Amazon Dynamo (Big Data)
February 21, 2014
3 / 14
4. Roles
Service Provider: Amazon
Service: Dynamo, the storage service
Customer: application/service vendors
Client: applications/services
User: human and/or bots
Service Level Agreements (SLA)
SLA are contracts signed by service providers and customers, specifying
the quality of service guaranteed for a client access distribution.
Example: service guaranteeing that it will provide a response within
300ms for 99.9% of its requests for a peak client load of 500 requests per
second.
Farley Lai (UIOWA)
Amazon Dynamo (Big Data)
February 21, 2014
4 / 14
5. What is Dynamo?
A distributed key-value storage service built on a ring topology with
high availability for writes
eventual consistency
Farley Lai (UIOWA)
Amazon Dynamo (Big Data)
February 21, 2014
5 / 14
6. Requirements and Assumptions
Requirements
Simple read/write to data items identified by unique keys
ACID: automicity, consistency, isolation and durability
SLA: latency constraints on the 99.9th percentile of the
distribution
Assumptions
Trusted environment and machines without security concerns
Farley Lai (UIOWA)
Amazon Dynamo (Big Data)
February 21, 2014
6 / 14
7. Problems, Techniques and Advantages
Problems
Partitioning
High write availability
Temporary failures
Permanent failures
Membership
Farley Lai (UIOWA)
Techniques
Advantages
Consistent Hashing
Vector clocks with
conlict resolution
Sloppy
Quorum,
hinted handoff
Incremental Scalability
Version size is decoupled
from update rates
High availability and durability guarantee despite
some unavailable replicas
Fast replica synchronization
decentralized registry for
storing membership and
liveness info
Merkle trees
Gossip protocol
Amazon Dynamo (Big Data)
February 21, 2014
7 / 14