This document summarizes a research paper on Bigtable, a distributed storage system from Google. Bigtable stores data in sparse, distributed, and persistent multidimensional sorted maps, indexed by row key, column key, and timestamp. It is designed to be scalable, fault-tolerant and handle massive amounts of data across thousands of commodity servers. Bigtable breaks data into tablets that are assigned to tablet servers, and uses mechanisms like compaction and caching to improve performance and manage storage. The system has proven capable of high throughput at Google, handling petabytes of data.
As part of NoSQL series, I presented Google Bigtable paper. In presentation I tried to give some plain introduction to Hadoop, MapReduce, HBase
www.scalability.rs
I promise that understand NoSQL is as easy as playing with LEGO bricks ! The Google Bigtable presented in 2006 is the inspiration for Apache HBase: let's take a deep dive into Bigtable to better understand Hbase.
For a long time, relational database management systems have been the only solution for persistent data store. However, with the phenomenal growth of data, this conventional way of storing has become problematic.
To manage the exponentially growing data traffic, largest information technology companies such as Google, Amazon and Yahoo have developed alternative solutions that store data in what have come to be known as NoSQL databases.
Some of the NoSQL features are flexible schema, horizontal scaling and no ACID support. NoSQL databases store and replicate data in distributed systems, often across datacenters, to achieve scalability and reliability.
The CAP theorem states that any networked shared-data system (e.g. NoSQL) can have at most two of three desirable properties:
• consistency(C) - equivalent to having a single up-to-date copy of the data
• availability(A) of that data (for reads and writes)
• tolerance to network partitions(P)
Because of this inherent tradeoff, it is necessary to sacrifice one of these properties. The general belief is that designers cannot sacrifice P and therefore have a difficult choice between C and A.
In this seminar two NoSQL databases are presented: Amazon's Dynamo, which sacrifices consistency thereby achieving very high availability and Google's BigTable, which guarantees strong consistency while provides only best-effort availability.
As part of NoSQL series, I presented Google Bigtable paper. In presentation I tried to give some plain introduction to Hadoop, MapReduce, HBase
www.scalability.rs
I promise that understand NoSQL is as easy as playing with LEGO bricks ! The Google Bigtable presented in 2006 is the inspiration for Apache HBase: let's take a deep dive into Bigtable to better understand Hbase.
For a long time, relational database management systems have been the only solution for persistent data store. However, with the phenomenal growth of data, this conventional way of storing has become problematic.
To manage the exponentially growing data traffic, largest information technology companies such as Google, Amazon and Yahoo have developed alternative solutions that store data in what have come to be known as NoSQL databases.
Some of the NoSQL features are flexible schema, horizontal scaling and no ACID support. NoSQL databases store and replicate data in distributed systems, often across datacenters, to achieve scalability and reliability.
The CAP theorem states that any networked shared-data system (e.g. NoSQL) can have at most two of three desirable properties:
• consistency(C) - equivalent to having a single up-to-date copy of the data
• availability(A) of that data (for reads and writes)
• tolerance to network partitions(P)
Because of this inherent tradeoff, it is necessary to sacrifice one of these properties. The general belief is that designers cannot sacrifice P and therefore have a difficult choice between C and A.
In this seminar two NoSQL databases are presented: Amazon's Dynamo, which sacrifices consistency thereby achieving very high availability and Google's BigTable, which guarantees strong consistency while provides only best-effort availability.
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...StreamNative
Apache Hudi is an open data lake platform, designed around the streaming data model. At its core, Hudi provides a transactions, upserts, deletes on data lake storage, while also enabling CDC capabilities. Hudi also provides a coherent set of table services, which can clean, compact, cluster and optimize storage layout for better query performance. Finally, Hudi's data services provide out-of-box support for streaming data from event systems into lake storage in near real-time.
In this talk, we will walk through an end-end use case for change data capture from a relational database, starting with capture changes using the Pulsar CDC connector and then demonstrate how you can use the Hudi deltastreamer tool to then apply these changes into a table on the data lake. We will discuss various tips to operationalizing and monitoring such pipelines. We will conclude with some guidance on future integrations between the two projects including a native Hudi/Pulsar connector and Hudi tiered storage.
The Google File System (GFS) presented in 2003 is the inspiration for the Hadoop Distributed File System (HDFS). Let's take a deep dive into GFS to better understand Hadoop.
Building robust CDC pipeline with Apache Hudi and DebeziumTathastu.ai
We have covered the need for CDC and the benefits of building a CDC pipeline. We will compare various CDC streaming and reconciliation frameworks. We will also cover the architecture and the challenges we faced while running this system in the production. Finally, we will conclude the talk by covering Apache Hudi, Schema Registry and Debezium in detail and our contributions to the open-source community.
What Every Developer Should Know About Database Scalabilityjbellis
Replication. Partitioning. Relational databases. Bigtable. Dynamo. There is no one-size-fits-all approach to scaling your database, and the CAP theorem proved that there never will be. This talk will explain the advantages and limits of the approaches to scaling traditional relational databases, as well as the tradeoffs made by the designers of newer distributed systems like Cassandra. These slides are from Jonathan Ellis's OSCON 09 talk: http://en.oreilly.com/oscon2009/public/schedule/detail/7955
An Overview of Spanner: Google's Globally Distributed DatabaseBenjamin Bengfort
Spanner is a globally distributed database that provides external consistency between data centers and stores data in a schema based semi-relational data structure. Not only that, Spanner provides a versioned view of the data that allows for instantaneous snapshot isolation across any segment of the data. This versioned isolation allows Spanner to provide globally consistent reads of the database at a particular time allowing for lock-free read-only transactions (and therefore no communications overhead for consensus during these types of reads). Spanner also provides externally consistent reads and writes with a timestamp-based linear execution of transactions and two phase commits. Spanner is the first distributed database that provides global sharding and replication with strong consistency semantics.
Scylla Summit 2022: Scylla 5.0 New Features, Part 1ScyllaDB
Discover the new features and capabilities of Scylla Open Source 5.0 directly from the engineers who developed it. This second block of lightning talks will cover the following topics:
- New IO Scheduler and Disk Parallelism
- Per-Service-Level Timeouts
- Better Workload Estimation for Backpressure and Out-of-Memory Conditions
- Large Partition Handling Improvements
- Optimizing Reverse Queries
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
This presentation on "Getting Started with HazelCast" was made by Sandeep Kumar Pandey from Lastminute.com in Core Java / BoJUG meetup group on 24th March.
"In this session, we are going to talk about high level architecture of Hazelcast framework and we will look into the Java Collections and concepts which has been used to build the framework. We will also have a live demo on Distributed Cache using Hazelcast."
Designed by Sanjay Ghemawat , Howard Gobioff and Shun-Tak Leung of Google in 2002-03.
Provides fault tolerance, serving large number of clients with high aggregate performance.
The field of Google is beyond the searching.
Google store the data in more than 15 thousands commodity hardware.
Handles the exceptions of Google and other Google specific challenges in their distributed file system.
CS295 Week5: Megastore - Providing Scalable, Highly Available Storage for Int...Varad Meru
Slides created as a part of CS 295's week 5 on Transactions and Systems.
CS 295 (Cloud Computing and BigData) at UCI - https://sites.google.com/site/cs295cloudcomputing/
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...StreamNative
Apache Hudi is an open data lake platform, designed around the streaming data model. At its core, Hudi provides a transactions, upserts, deletes on data lake storage, while also enabling CDC capabilities. Hudi also provides a coherent set of table services, which can clean, compact, cluster and optimize storage layout for better query performance. Finally, Hudi's data services provide out-of-box support for streaming data from event systems into lake storage in near real-time.
In this talk, we will walk through an end-end use case for change data capture from a relational database, starting with capture changes using the Pulsar CDC connector and then demonstrate how you can use the Hudi deltastreamer tool to then apply these changes into a table on the data lake. We will discuss various tips to operationalizing and monitoring such pipelines. We will conclude with some guidance on future integrations between the two projects including a native Hudi/Pulsar connector and Hudi tiered storage.
The Google File System (GFS) presented in 2003 is the inspiration for the Hadoop Distributed File System (HDFS). Let's take a deep dive into GFS to better understand Hadoop.
Building robust CDC pipeline with Apache Hudi and DebeziumTathastu.ai
We have covered the need for CDC and the benefits of building a CDC pipeline. We will compare various CDC streaming and reconciliation frameworks. We will also cover the architecture and the challenges we faced while running this system in the production. Finally, we will conclude the talk by covering Apache Hudi, Schema Registry and Debezium in detail and our contributions to the open-source community.
What Every Developer Should Know About Database Scalabilityjbellis
Replication. Partitioning. Relational databases. Bigtable. Dynamo. There is no one-size-fits-all approach to scaling your database, and the CAP theorem proved that there never will be. This talk will explain the advantages and limits of the approaches to scaling traditional relational databases, as well as the tradeoffs made by the designers of newer distributed systems like Cassandra. These slides are from Jonathan Ellis's OSCON 09 talk: http://en.oreilly.com/oscon2009/public/schedule/detail/7955
An Overview of Spanner: Google's Globally Distributed DatabaseBenjamin Bengfort
Spanner is a globally distributed database that provides external consistency between data centers and stores data in a schema based semi-relational data structure. Not only that, Spanner provides a versioned view of the data that allows for instantaneous snapshot isolation across any segment of the data. This versioned isolation allows Spanner to provide globally consistent reads of the database at a particular time allowing for lock-free read-only transactions (and therefore no communications overhead for consensus during these types of reads). Spanner also provides externally consistent reads and writes with a timestamp-based linear execution of transactions and two phase commits. Spanner is the first distributed database that provides global sharding and replication with strong consistency semantics.
Scylla Summit 2022: Scylla 5.0 New Features, Part 1ScyllaDB
Discover the new features and capabilities of Scylla Open Source 5.0 directly from the engineers who developed it. This second block of lightning talks will cover the following topics:
- New IO Scheduler and Disk Parallelism
- Per-Service-Level Timeouts
- Better Workload Estimation for Backpressure and Out-of-Memory Conditions
- Large Partition Handling Improvements
- Optimizing Reverse Queries
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
This presentation on "Getting Started with HazelCast" was made by Sandeep Kumar Pandey from Lastminute.com in Core Java / BoJUG meetup group on 24th March.
"In this session, we are going to talk about high level architecture of Hazelcast framework and we will look into the Java Collections and concepts which has been used to build the framework. We will also have a live demo on Distributed Cache using Hazelcast."
Designed by Sanjay Ghemawat , Howard Gobioff and Shun-Tak Leung of Google in 2002-03.
Provides fault tolerance, serving large number of clients with high aggregate performance.
The field of Google is beyond the searching.
Google store the data in more than 15 thousands commodity hardware.
Handles the exceptions of Google and other Google specific challenges in their distributed file system.
CS295 Week5: Megastore - Providing Scalable, Highly Available Storage for Int...Varad Meru
Slides created as a part of CS 295's week 5 on Transactions and Systems.
CS 295 (Cloud Computing and BigData) at UCI - https://sites.google.com/site/cs295cloudcomputing/
In recent years, we have seen an overwhelming number of TV commercials that promise that the Cloud can help with many problems, including some family issues. What stands behind the terms “Cloud” and “Cloud Computing,” and what we can actually expect from this phenomenon? A group of students of the Computer Systems Technology department and Dr. T. Malyuta, whom has been working with the Cloud technologies since its early days, will provide an overview of the business and technological aspects of the Cloud.
As part of NoSQL series, I presented Google Bigtable paper. In presentation I tried to give some plain introduction to Hadoop, MapReduce, HBase
www.scalability.rs
Privacy Preserving Public Auditing for Data Storage Security in Cloud.pptGirish Chandra
Introducing TPA(Third Party Auditor) to the cloud.It sends the information about the data stored in the cloud.It informs the user when any unauthorized user tries to steal his data from the cloud.
In the ever-evolving landscape of data management, databases play a crucial role in storing and organizing vast amounts of information. With the increasing complexity and scale of modern applications, having a flexible and efficient database structure is essential for smooth operations. One such innovative approach that has gained traction in recent years is snowflake cloning, a technique that enables organizations to optimize their database performance by creating virtual copies or snapshots of specific tables within a larger database.
VectorDB Schema Design 101 - Considerations for Building a Scalable and Perfo...Zilliz
People often say that vector search is easy, but that's not entirely true. Vector search is more than just vector indexing and a Python wrapper. If you want to build a high-performance, scalable, and production-ready vector search service, you need to consider many factors.
Very basic Introduction to Big Data. Touches on what it is, characteristics, some examples of Big Data frameworks. Hadoop 2.0 example - Yarn, HDFS and Map-Reduce with Zookeeper.
SURVEY ON IMPLEMANTATION OF COLUMN ORIENTED NOSQL DATA STORES ( BIGTABLE & CA...IJCERT JOURNAL
NOSQL is a database provides a mechanism for storage and retrieval of data that is modeled for huge amount of data which is used in big data and Cloud Computing . NOSQL systems are also called "Not only SQL" to emphasize that they may support SQL-like query languages. A basic classification of NOSQL is based on data model; they are like column, Document, Key-Value etc. The objective of this paper is to study and compare the implantation of various column oriented data stores like Bigtable, Cassandra.
Data Warehousing Trends, Best Practices, and Future OutlookJames Serra
Over the last decade, the 3Vs of data - Volume, Velocity & Variety has grown massively. The Big Data revolution has completely changed the way companies collect, analyze & store data. Advancements in cloud-based data warehousing technologies have empowered companies to fully leverage big data without heavy investments both in terms of time and resources. But, that doesn’t mean building and managing a cloud data warehouse isn’t accompanied by any challenges. From deciding on a service provider to the design architecture, deploying a data warehouse tailored to your business needs is a strenuous undertaking. Looking to deploy a data warehouse to scale your company’s data infrastructure or still on the fence? In this presentation you will gain insights into the current Data Warehousing trends, best practices, and future outlook. Learn how to build your data warehouse with the help of real-life use-cases and discussion on commonly faced challenges. In this session you will learn:
- Choosing the best solution - Data Lake vs. Data Warehouse vs. Data Mart
- Choosing the best Data Warehouse design methodologies: Data Vault vs. Kimball vs. Inmon
- Step by step approach to building an effective data warehouse architecture
- Common reasons for the failure of data warehouse implementations and how to avoid them
Cloud Architecture Patterns for Mere Mortals - Bill Wilder - Vermont Code Cam...Bill Wilder
How do you design applications for the cloud so that they will be scalable and reliable? In this talk, we will explain several architectural patterns which are popular for cloud computing: we will look at the need for the patterns generally, then look concretely at how you might realize them using capabilities of the Windows Azure Platform. CQRS, NoSQL, Sharding, and a few smaller patterns will be considered.
Presented by Bill Wilder at Vermont Code Camp III on Saturday September 10, 2011. http://blog.codingoutloud.com/2011/09/12/vermont-code-camp-iii/
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Bigtable: A Distributed Storage System for Structured Data
1. Outline
Introduction
Design
Implementation
Results
Conclusions
Bigtable: A Distributed Storage System for
Structured Data
Alvanos Michalis
April 6, 2009
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
2. Outline
Introduction
Design
Implementation
Results
Conclusions
1 Introduction
Motivation
2 Design
Data model
3 Implementation
Building blocks
Tablets
Compactions
Refinements
4 Results
Hardware Environment
Performance Evaluation
5 Conclusions
Real applications
Lessons
End
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
3. Outline
Introduction
Design
Motivation
Implementation
Results
Conclusions
Google!
Lots of Different kinds of data!
Crawling system URLs, contents, links, anchors, page-rank etc
Per-user data: preferences, recent queries/ search history
Geographic data, images etc ...
Many incoming requests
No commercial system is big enough
Scale is too large for commercial databases
May not run on their commodity hardware
No dependence on other vendors
Optimizations
Better Price/Performance
Building internally means the system can be applied across
many projects for low incremental cost
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
4. Outline
Introduction
Design
Motivation
Implementation
Results
Conclusions
Google goals
Fault-tolerant, persistent
Scalable
1000s of servers
Millions of reads/writes, efficient scans
Self-managing
Simple!
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
5. Outline
Introduction
Design
Data model
Implementation
Results
Conclusions
Bigtable
Definition
A Bigtable is a sparse, distributed, persistent multidimensional
sorted map.
The map is indexed by a row key, column key, and a timestamp;
each value in the map is an uninterpreted array of bytes.
(row:string, column:string, time:int64) -> string
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
6. Outline
Introduction
Design
Data model
Implementation
Results
Conclusions
Rows
The row keys in a table are arbitrary strings
Every read or write of data under a single row key is atomic
maintains data in lexicographic order by row key
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
7. Outline
Introduction
Design
Data model
Implementation
Results
Conclusions
Column Families
Grouped into sets called column families
All data stored in a column family is usually of the same type
A column family must be created before data can be stored
under any column key in that family
A column key is named using the following syntax:
family:qualifier
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
8. Outline
Introduction
Design
Data model
Implementation
Results
Conclusions
Timestamps
Each cell in a Bigtable can contain multiple versions of the
same data; these versions are indexed by timestamp (64-bit
integers).
Applications that need to avoid collisions must generate
unique timestamps themselves.
To make the management of versioned data less onerous, they
support two per-column-family settings that tell Bigtable to
garbage-collect cell versions automatically.
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
9. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
Infrastructure
Google WorkQueue (scheduler)
GFS: large-scale distributed file system
Master: responsible for metadata
Chunk servers: responsible for r/w large chunks of data
Chunks replicated on 3 machines; master responsible
Chubby: lock/file/name service
Coarse-grained locks; can store small amount of data in a lock
5 replicas; need a majority vote to be active
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
10. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
SSTable
Lives in GFS
Immutable, sorted file of key-value pairs
Chunks of data plus an index
Index is of block ranges, not values
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
11. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
Tablet Design
Large tables broken into tablets at row boundaries
Tablets hold contiguous rows
Approx 100 200 MB of data per tablet
Approx 100 tablets per machine
Fast recovery
Load-balancing
Built out of multiple SSTables
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
12. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
Tablet Location
Like a B+-tree, but fixed at 3 levels
How can we avoid creating a bottleneck at the root?
Aggressively cache tablet locations
Lookup starts from leaf (bet on it being correct); reverse on
miss
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
13. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
Tablet Assignment
Each tablet is assigned to one tablet server at a time. The
master keeps track of the set of live tablet servers, and the
current assignment of tablets to tablet servers.
Bigtable uses Chubby to keep track of tablet servers. When a
tablet server starts, it creates, and acquires an exclusive lock
on, a uniquely-named file in a specific Chubby directory.
Tablet server stops serving its tablets if loses its exclusive lock
The master is responsible for detecting when a tablet server is
no longer serving its tablets, and for reassigning those tablets
as soon as possible.
When a master is started by the cluster management system,
it needs to discover the current tablet assignments before it
can change them.
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
14. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
Serving a Tablet
Updates are logged
Each SSTable corresponds to a batch of updates or a
snapshot of the tablet taken at some earlier time
Memtable (sorted by key) caches recent updates
Reads consult both memtable and SSTables
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
15. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
Compactions
As write operations execute, the size of the memtable increases.
Minor compaction convert the memtable into an SSTable
Reduce memory usage
Reduce log traffic on restart
Merging compaction
Periodically executed in the background
Reduce number of SSTables
Good place to apply policy keep only N versions
Major compaction
Merging compaction that results in only one SSTable
No deletion records, only live data
Reclaim resources.
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
16. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
Refinements (1/2)
Group column families together into an SSTable. Segregating
column families that are not typically accessed together into
separate locality groups enables more efficient reads.
Can compress locality groups, using Bentley and McIlroy’s
scheme and a fast compression algorithm that looks for
repetitions.
Bloom Filters on locality groups allows to ask whether an
SSTable might contain any data for a specified row/column
pair. Drastically reduces the number of disk seeks required -
for non-existent rows or columns do not need to touch disk.
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
17. Outline
Introduction Building blocks
Design Tablets
Implementation Compactions
Results Refinements
Conclusions
Refinements (2/2)
Caching for read performance ( two levels of caching)
Scan Cache: higher-level cache that caches the key-value pairs
returned by the SSTable interface to the tablet server code.
Block Cache: lower-level cache that caches SSTables blocks
that were read from GFS.
Commit-log implementation
Speeding up tablet recovery (log entries)
Exploiting immutability
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
18. Outline
Introduction
Design Hardware Environment
Implementation Performance Evaluation
Results
Conclusions
Hardware Environment
Tablet servers were configured to use 1 GB of memory and to
write to a GFS cell consisting of 1786 machines with two 400
GB IDE hard drives each.
Each machine had two dual-core Opteron 2 GHz chips
Enough physical memory to hold the working set of all
running processes
Single gigabit Ethernet link
Two-level tree-shaped switched network with 100-200 Gbps
aggregate bandwidth at the root.
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
19. Outline
Introduction
Design Hardware Environment
Implementation Performance Evaluation
Results
Conclusions
Results Per Tablet Server
Number of 1000-byte values read/written per second.
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
20. Outline
Introduction
Design Hardware Environment
Implementation Performance Evaluation
Results
Conclusions
Results Aggregate Rate
Number of 1000-byte values read/written per second.
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
21. Outline
Introduction
Design Hardware Environment
Implementation Performance Evaluation
Results
Conclusions
Single tablet-server performance
The tablet server executes 1200 reads per second ( 75
MB/s), enough to saturate the tablet server CPUs because of
overheads in networking stack
Random and sequential writes perform better than random
reads (commit log and uses group commit)
No significant difference between random writes and
sequential writes (same commit log)
Sequential reads perform better than random reads (block
cache)
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
22. Outline
Introduction
Design Hardware Environment
Implementation Performance Evaluation
Results
Conclusions
Scaling
Aggregate throughput increases dramatically performance of
random reads from memory increases
However, performance does not increase linearly
Drop in per-server throughput
Imbalance in load: Re-balancing is throttled to reduce the
number of tablet movement and the load generated by
benchmarks shifts around as the benchmark progresses
The random read benchmark: transfer one 64KB block over
the network for every 1000-byte read and saturates shared 1
Gigabit links
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
23. Outline
Introduction
Real applications
Design
Lessons
Implementation
End
Results
Conclusions
Timestamps
Google Analytics
Google Earth
Personalized Search
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
24. Outline
Introduction
Real applications
Design
Lessons
Implementation
End
Results
Conclusions
Lessons learned
Large distributed systems are vulnerable to many types of
failures, not just the standard network partitions and fail-stop
failures
Memory and network corruption
Large clock skew
Extended and asymmetric network partitions
Bugs in other systems (Chubby !)
...
Delay adding new features until it is clear how the new
features will be used
A practical lesson: the importance of proper system-level
monitoring
Keep It Simple!
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data
25. Outline
Introduction
Real applications
Design
Lessons
Implementation
End
Results
Conclusions
END!
QUESTIONS ?
Alvanos Michalis Bigtable: A Distributed Storage System for Structured Data