SlideShare a Scribd company logo
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone – Object Store for Apache
Hadoop
Anu Engineer
aengineer@apache.org
Arpit Agarwal
arp@apache.org
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone – Why an Object Store
⬢ With workloads like IoT we are looking at scaling to trillions of objects.
⬢ Apache HDFS is designed for large objects – not for many small objects
– Small files create memory pressure on namenode.
⬢ Each small file creates a block in the datanode.
–Datanodes send all block information to namenode in BlockReports.
⬢ Both of these create scalability issues on Namenode.
⬢ Metadata in memory is the strength of the original GFS and HDFS design, but also its
weakness in scaling number of files and blocks.
⬢ An object store has simpler semantics than a file system and is easier to scale
Apache Hadoop, Hadoop, Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and other countries.
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone – Why an Object Store (continued)
⬢ Ozone attempts to scale to trillions of objects
– This presentation is about how we will get there.
⬢ Ozone is built on a distributed metadata store.
⬢ Avoids any single server becoming a bottleneck
⬢ More parallelism possible in both data and metadata operations
⬢ Build on well tested components and understood protocols
–RAFT for consensus
•RAFT is a protocol for reaching consensus between a set of machines in an unreliable
environment where machines and network may fail.
–Off-the-shelf Key-Value store like LevelDB
•LevelDB is an open-source standalone key-value store built by Google.
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alternative solutions to NameNode scalability
⬢ HDFS federation aims to address namespace and Block Space scalability issues.
–Federation deployment and planning adds complexity
–Requires changes to other components in the Hadoop stack
⬢ HDFS-8286 - Partial Namespace In Memory.
–Proposal to keep active working set of namespace in memory.
⬢ HDFS-5477 - Block Management as a Service.
–Proposed solution for block space scalability issue.
⬢ Ozone borrows many ideas from these and is a super set of these approaches.
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Presentation Outline
Ozone Introduction
Ozone Architectural Overview
Containers
Ozone - Bringing it all together
Bonus Slides - if we have time.
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Introduction
⬢ An Ozone URL
–http://hostname/myvolume/mybucket/mykey
⬢ An S3 URL
–http://hostname/mybucket/mykey
⬢ An Azure URL
–http://hostname/myaccount/mybucket/key
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Definitions
⬢ Storage Volume
–A notion similar to an account
–Allows admin controls on usage of the object store e.g. storage quota
–Created and managed by admins only
⬢ Bucket
–Consists of keys and objects
–Similar to a bucket in S3 or a container in Azure
–ACLs
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Definitions (continued)
⬢ Storage Key
–Unique in a bucket
⬢ Object
–Values in a bucket
–Each corresponds to a unique key within a bucket
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - REST API
⬢ POST - Creates Volumes and Buckets
–Only Admin creates volumes
–Bucket can be created by owner of the volume
⬢ PUT - Updates Volumes , Buckets and creates keys
–Only admin can change some volume settings
–Buckets have ACLs
–Creates Keys
1
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - REST API (continued)
⬢ GET - Lists volumes and buckets and allows reading of keys
–Lists Volumes
–List Buckets
–Get Keys
⬢ DELETE - Deletes volumes, buckets and keys.
–Delete Volumes
–Delete Buckets
–Removes the Key
1
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Components
⬢ Containers – Actual storage locations on Datanodes.
–We acknowledge the term container is overloaded. No relation to YARN containers or LXC.
–Assume container means a collection of blocks on a datanode for now.
–Containers deep dive to follow.
⬢ Ozone Handler - REST frontend for the Ozone rest protocol - deployed on datanodes.
⬢ Storage Container Manager (SCM) - Manages the container life cycle.
⬢ Ozone Key Space Manager (KSM) - Maps Ozone entities to Containers.
⬢ Container Client - Talks to KSM to discover the location of a container and sends IO
requests to the appropriate container.
1
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Overview
1
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager
1
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager
⬢ Key to container mapping service.
⬢ Keeps the key ranges to containers mapping in memory.
–Θ(number of containers) - 1 Exabyte cluster = 200M containers x 5GB each.
–Memory usage scales with number of containers and not number of keys.
⬢ KSM does NOT know about all the keys in the system.
⬢ KSM state is replicated via RAFT, NameNode-like snapshots.
1
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager
⬢ KSM knows about Ozone Volumes and Buckets.
⬢ KSM keeps a map of Volumes to container and buckets to containers.
⬢ KSM performs longest prefix match on a given string.
⬢ Example: The user wants to lookup a key - “/volume1/bucket1/key1”
–KSM authenticates the user, maps this key to a container and looks up the container location.
–Container client gets a token from the KSM and talks to the container on the data node.
–Container client makes a getKey call to the datanode container with the full key path.
–DataNode validates the access token and serves the value.
⬢ Contents of a bucket may span multiple containers.
1
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
KSM - Bucket spanning multiple containers
1
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved1
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers
1
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Container Framework
⬢ A shareable generic block service that can be used by distributed storage services.
⬢ Make it easier to develop new storage services - BYO storage format and naming
scheme.
⬢ Design Goals
–No single points of failure. All services are replicated.
–Avoid bottlenecks
•Minimize central state
•No verbose Block Reports
⬢ Lessons learned from large scale HDFS clusters.
⬢ Ideas from earlier proposals in HDFS community.
1
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Container Framework Components
2
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers
2
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers
⬢ A container is the unit of replication
–Size bounded by how quickly it can be re-replicated after a loss.
⬢ Each container is an independent key-value store.
–No requirements on the structure or format of keys/values.
–Keys are unique only within a container.
⬢ E.g. key-value pair could be one of
–An Ozone Key-Value pair
–An HDFS block ID and block contents
•Or part of a block, when a block spans containers.
2
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers (continued)
⬢ Each container has metadata
– Metadata consistency maintained via the RAFT protocol
– Metadata consists of keys and references to chunks.
⬢ Container metadata stored in LevelDB.
– Exact choice of KV store unimportant. LevelDB is already used by other Hadoop components.
⬢ A chunk is a piece of user data.
– Chunks are replicated via a data pipeline.
– Chunks can be of arbitrary sizes e.g. a few KB to GBs.
– Each chunk reference is a (file, offset, length) triplet.
⬢ Containers may garbage collect unreferenced chunks.
⬢ Each container independently decides how to map chunks to files
– Allow reauthoring files for performance, compaction and overwrites.
2
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers (continued)
2
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers support simple client operations
⬢ Write chunks - streaming writes
⬢ Put(key, List<ChunkReference>)
–The value is a list of chunk references.
–Putting a key makes previously written chunk data visible to readers.
–Put overwrites previous versions of the key.
⬢ Get(key)
–Returns a list of chunk references
⬢ Read chunks - streaming reads
⬢ Delete(key)
⬢ List Keys
2
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storage Container Manager
2
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storage Container Manager
⬢ A fault-tolerant replicated service
⬢ Replicates its own state using RAFT protocol
⬢ Provides Container Location Service to clients
–Given a container ID, return a list of nodes with replicas
–Mapping a container ID to Data Nodes (and vice versa)
⬢ Provides Cluster Membership Management
–Maintain list of live Data Nodes in the cluster
–Handle heartbeats from DataNodes
2
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storage Container Manager (continued)
⬢ Provides Replication services
–Detect lost container replicas and initiate re-replication
–Containers send a container report.
•Unlike HDFS block reports which include details about each block , a container report is a
summary of information.
•This is used by KSM for placement of containers
⬢ If a node suffers from disk failure or if a node is lost, the reconstruction is a local
activity which is coordinated via RAFT running on the data nodes.
2
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storage Container Manager
⬢ Maintains pre-created containers
⬢ Collects container operation statistics
⬢ Decides which Data Nodes form the replication set for a given container.
–The number of replication sets in a cluster is bounded
–Borrowing the work done by Facebook and RAMCloud (Copysets, Cidon et al. 2013).
⬢ Important - Knows nothing about keys
–Does NOT provide Naming Service (mapping keys to containers)
–e.g. KSM provides naming for Ozone.
2
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Conceptual Representation of Ozone and Container State
3
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved3
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Bringing it all together
3
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Bringing it all together - Ozone createVolume operations
Key Space Manager
Replicated containers
1: createVolume
Container Manager
2: Lookup(volName, Operation) 3: getContainer
4: putMetdata(VolumeName, Properties)
Ozone HandlerClient
Heartbeats & Reports
DataNodes
3
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Tracing an Ozone PutKey
Key Space Manager
Replicated containers
1: Ozone - putKey
2: Lookup(keyName, Operation)
4: putData(File, offset, Length, data)
Client
5: putMetadata(key, List<chunks>)
Ozone Handler
DataNodes
3
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Tracing an Ozone createVolume
OzoneVolume vol =
(new VolumeBuilder(pipeLine))
.setCreated(new Date())
.setOwnerName("bilbo")
.setClient(client)
.setName(“shire”)
.build();
POST /shire
keyData = {ContainerKeyData}
keyName = "shire"
containerName = "OzoneContainer"
metadata =
0."Created" -> "1449533074362"
1."CreatedBy" -> "gandalf"
2."Key" -> "VOLUME"
3."Owner" -> "bilbo"
Ozone REST Handler code Container wire and storage format
3
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Metadata operations
⬢ Any metadata write to a container is Replicated via RAFT.
⬢ Machines forming the replication set for a container comprise a pipeline.
⬢ A createVolume call reduces to putKey operation on the container.
⬢ putKey is consistent, atomic and durable.
⬢ All metadata data operations are done via putKey, getKey and deleteKey.
⬢ Data is written to one or more chunks and a key is updated to point to those chunks.
⬢ Updating the key makes the data visible in the namespace.
3
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Current State of Ozone
⬢ Stand alone container framework.
⬢ Single node ozone using container framework.
⬢ Full REST API -- Command Line Tools and Client Libs are fully functional.
⬢ Active development in branch HDFS-7240.
⬢ Work in progress:
–SCM
–KSM
–Replication Pipeline
–RAFT
3
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Acknowledgements
⬢ Ozone is being designed and developed by Jitendra Pandey, Chris Nauroth, Tsz Wo
(Nicholas) Sze, Jing Zhao, Suresh Srinivas, Sanjay Radia, Anu Engineer and Arpit
Agarwal.
⬢ The Apache community has been very helpful and we were supported by comments
and contributions from Kanaka Kumar Avvaru, Edward Bortnikov, Thomas Demoor,
Nick Dimiduk, Chris Douglas, Jian Fang, Lars Francke, Gautam Hegde, Lars Hofhansl,
Jakob Homan, Virajith Jalaparti, Charles Lamb, Steve Loughran, Haohui Mai, Colin
Patrick McCabe, Aaron Myers, Owen O’Malley, Liam Slusser, Jeff Sogolov, Enis
Soztutar, Andrew Wang, Fengdong Yu, Zhe Zhang & khanderao.
3
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved3
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You
3
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved3
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Bonus Slides
3
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Dynamic Container Partitioning
⬢ KSM deals with dynamic partitioning of containers.
⬢ Let us say that a user starts by uploading all his photographs to a bucket in ozone.
⬢ Since all the photographs are called IMG_* (thanks Apple), we will soon overflow the
5GB capacity of the container.
⬢ At this point we need to split the container.
4
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone KSM - Dynamic Container Partitioning
⬢ The container client attempts to write the Nth ozone key, IMG_N, gets a partition
required error.
⬢ Container client will take that error and return that info to KSM.
⬢ That error contains the info -- about the proposed split -- That is IMG_0- IMG_200 will
stay in this container and IMG_201-IMG_400 will move to next container.
⬢ Note: KSM initiates container partitioning but mechanics of the split are handled by
the Container Layer
4
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Dynamic Container Partitioning
⬢ One of the assumptions we have made about a container split is that the splits are on
the same datanode as the original container.
⬢ This allows us to reduce a split operation to a copy of Keys from one LevelDB to
another LevelDB.
⬢ if we need to move actual file data from one datanode to another -- we do support
container moves. However they are slow.
⬢ A split on the other hand will complete in seconds in most cases.
⬢ The split point is chosen by the container so that we are able to pick the 50th percentile
position that gives us reasonable chance at an equal partition of a container.
⬢ KSM does not know about the keys or the actual data sizes until much later.
⬢ So always relies on the container to tell it where the split should be.
4
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Dynamic Container Partitioning
⬢ A container split is done in KSM via updating the Tree. The range partition key we
maintain gets updated to reflect the fact that Keys - {IMG_0 - IMG_200} are on
container C1, and keys {IMG_201-IMG_Z} are on C2.
⬢ Container will update the SCM when the split is done.
⬢ This information is learned and maintained by KSM.
4
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Soft Quotas
⬢ In the HDFS world, a Quota is hard limit. It is actually conservative in quota
management.
⬢ In the ozone world, Quotas are soft quotas. That is users can and will be able to violate
it, but KSM/SCM will eventually learn about it and lock the volume out.
⬢ The key reason why this is different is because KSM/SCM is not involved in the
allocation of chunks.
⬢ The containers have a partial -- that is an isolated view of the data in a volume. Since
volumes can span many containers, it is possible for users to allocate chunks that
violate the volume quota, but eventually KSM will learn and disable further writes to a
volume.
4
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Missing namespace problem
⬢ One great thing about HDFS is Namenode.
–Despite scalability issues, in most cases Namenode does a wonderful job.
⬢ Ozone - Subtle problem if we lose the all replicas of a container.
⬢ We will not only lose data -- just as if HDFS lost its all 3 replicas, but we will also lose
information about which keys have been lost.
⬢ To solve this issue, we propose to have a on-disk eventually consistent log maintained
by a separate service.
–Records information about the keys that exist in the cluster.
⬢ This Scribe service logs the state of the cluster.
4
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Range reads
⬢ Ozone supports range reads and might support range writes like part upload in S3.
–Ozone achieves this by using the chunk mechanism.
–Chunks offer a stream like interface.
–You can seek to any location and read as many bytes as you want.
–This is used by ozone to support range reads
⬢ Periodic Scanner can reclaim unreferenced chunks.
4
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Scalability - What HDFS does well
⬢ HDFS NN stores all namespace metadata in memory (as per GFS)
–Scales to large clusters (5K) since all metadata in memory
•60K-100K tasks can share the Namenode
•Low latency
–Large data if files are large
⬢ Proof points of large data and large clusters
–Single Organizations have over 600PB in HDFS
–Single clusters with over 200PB using federation
–Large clusters of over 4K multi-core nodes hitting a single NN

More Related Content

What's hot

Hive+Tez: A performance deep dive
Hive+Tez: A performance deep diveHive+Tez: A performance deep dive
Hive+Tez: A performance deep dive
t3rmin4t0r
 
Apache Kudu: Technical Deep Dive


Apache Kudu: Technical Deep Dive

Apache Kudu: Technical Deep Dive


Apache Kudu: Technical Deep Dive


Cloudera, Inc.
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
enissoz
 
Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing
DataWorks Summit
 
Apache HBase Performance Tuning
Apache HBase Performance TuningApache HBase Performance Tuning
Apache HBase Performance Tuning
Lars Hofhansl
 
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
Simplify CDC Pipeline with Spark Streaming SQL and Delta LakeSimplify CDC Pipeline with Spark Streaming SQL and Delta Lake
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
Databricks
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
 
A Reference Architecture for ETL 2.0
A Reference Architecture for ETL 2.0 A Reference Architecture for ETL 2.0
A Reference Architecture for ETL 2.0
DataWorks Summit
 
Apache Iceberg: An Architectural Look Under the Covers
Apache Iceberg: An Architectural Look Under the CoversApache Iceberg: An Architectural Look Under the Covers
Apache Iceberg: An Architectural Look Under the Covers
ScyllaDB
 
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Data Engineer's Lunch #83: Strategies for Migration to Apache IcebergData Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Anant Corporation
 
Rds data lake @ Robinhood
Rds data lake @ Robinhood Rds data lake @ Robinhood
Rds data lake @ Robinhood
BalajiVaradarajan13
 
Chicago Data Summit: Apache HBase: An Introduction
Chicago Data Summit: Apache HBase: An IntroductionChicago Data Summit: Apache HBase: An Introduction
Chicago Data Summit: Apache HBase: An Introduction
Cloudera, Inc.
 
Ozone and HDFS's Evolution
Ozone and HDFS's EvolutionOzone and HDFS's Evolution
Ozone and HDFS's Evolution
DataWorks Summit
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudAmazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
Dvir Volk
 
Hudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilitiesHudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilities
Nishith Agarwal
 
Scaling HBase for Big Data
Scaling HBase for Big DataScaling HBase for Big Data
Scaling HBase for Big Data
Salesforce Engineering
 
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
StampedeCon
 
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data ProcessingApache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing
DataWorks Summit
 

What's hot (20)

Hive+Tez: A performance deep dive
Hive+Tez: A performance deep diveHive+Tez: A performance deep dive
Hive+Tez: A performance deep dive
 
Apache Kudu: Technical Deep Dive


Apache Kudu: Technical Deep Dive

Apache Kudu: Technical Deep Dive


Apache Kudu: Technical Deep Dive


 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
 
Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing
 
Apache HBase Performance Tuning
Apache HBase Performance TuningApache HBase Performance Tuning
Apache HBase Performance Tuning
 
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
Simplify CDC Pipeline with Spark Streaming SQL and Delta LakeSimplify CDC Pipeline with Spark Streaming SQL and Delta Lake
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
 
A Reference Architecture for ETL 2.0
A Reference Architecture for ETL 2.0 A Reference Architecture for ETL 2.0
A Reference Architecture for ETL 2.0
 
Apache Iceberg: An Architectural Look Under the Covers
Apache Iceberg: An Architectural Look Under the CoversApache Iceberg: An Architectural Look Under the Covers
Apache Iceberg: An Architectural Look Under the Covers
 
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Data Engineer's Lunch #83: Strategies for Migration to Apache IcebergData Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
 
Rds data lake @ Robinhood
Rds data lake @ Robinhood Rds data lake @ Robinhood
Rds data lake @ Robinhood
 
Chicago Data Summit: Apache HBase: An Introduction
Chicago Data Summit: Apache HBase: An IntroductionChicago Data Summit: Apache HBase: An Introduction
Chicago Data Summit: Apache HBase: An Introduction
 
Ozone and HDFS's Evolution
Ozone and HDFS's EvolutionOzone and HDFS's Evolution
Ozone and HDFS's Evolution
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudAmazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Hudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilitiesHudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilities
 
Scaling HBase for Big Data
Scaling HBase for Big DataScaling HBase for Big Data
Scaling HBase for Big Data
 
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
 
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data ProcessingApache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing
 

Similar to Ozone- Object store for Apache Hadoop

Ozone and HDFS’s evolution
Ozone and HDFS’s evolutionOzone and HDFS’s evolution
Ozone and HDFS’s evolution
DataWorks Summit
 
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage SubsystemEvolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage Subsystem
DataWorks Summit/Hadoop Summit
 
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage  object store integration in production (final)Hadoop & cloud storage  object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)
Chris Nauroth
 
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionHadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
DataWorks Summit/Hadoop Summit
 
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionHadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
DataWorks Summit/Hadoop Summit
 
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage SubsystemEvolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage Subsystem
DataWorks Summit/Hadoop Summit
 
CBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFSCBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFS
DataWorks Summit
 
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
DataWorks Summit
 
Hadoop 3 in a Nutshell
Hadoop 3 in a NutshellHadoop 3 in a Nutshell
Hadoop 3 in a Nutshell
DataWorks Summit/Hadoop Summit
 
Running Services on YARN
Running Services on YARNRunning Services on YARN
Running Services on YARN
DataWorks Summit/Hadoop Summit
 
Big data spain keynote nov 2016
Big data spain keynote nov 2016Big data spain keynote nov 2016
Big data spain keynote nov 2016
alanfgates
 
Evolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage SubsystemEvolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage Subsystem
DataWorks Summit/Hadoop Summit
 
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBaseHBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
Cloudera, Inc.
 
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
Big Data Spain
 
Apache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and FutureApache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and Future
DataWorks Summit/Hadoop Summit
 
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for BeginnersThe Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
Edelweiss Kammermann
 
Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016
alanfgates
 
Apache Hive on ACID
Apache Hive on ACIDApache Hive on ACID
Apache Hive on ACID
Hortonworks
 
Apache Hive on ACID
Apache Hive on ACIDApache Hive on ACID
Apache Hive on ACID
DataWorks Summit/Hadoop Summit
 
Apache Hadoop 3.0 Community Update
Apache Hadoop 3.0 Community UpdateApache Hadoop 3.0 Community Update
Apache Hadoop 3.0 Community Update
DataWorks Summit
 

Similar to Ozone- Object store for Apache Hadoop (20)

Ozone and HDFS’s evolution
Ozone and HDFS’s evolutionOzone and HDFS’s evolution
Ozone and HDFS’s evolution
 
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage SubsystemEvolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage Subsystem
 
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage  object store integration in production (final)Hadoop & cloud storage  object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)
 
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionHadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
 
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionHadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
 
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage SubsystemEvolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage Subsystem
 
CBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFSCBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFS
 
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
 
Hadoop 3 in a Nutshell
Hadoop 3 in a NutshellHadoop 3 in a Nutshell
Hadoop 3 in a Nutshell
 
Running Services on YARN
Running Services on YARNRunning Services on YARN
Running Services on YARN
 
Big data spain keynote nov 2016
Big data spain keynote nov 2016Big data spain keynote nov 2016
Big data spain keynote nov 2016
 
Evolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage SubsystemEvolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage Subsystem
 
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBaseHBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
 
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
 
Apache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and FutureApache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and Future
 
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for BeginnersThe Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
 
Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016
 
Apache Hive on ACID
Apache Hive on ACIDApache Hive on ACID
Apache Hive on ACID
 
Apache Hive on ACID
Apache Hive on ACIDApache Hive on ACID
Apache Hive on ACID
 
Apache Hadoop 3.0 Community Update
Apache Hadoop 3.0 Community UpdateApache Hadoop 3.0 Community Update
Apache Hadoop 3.0 Community Update
 

More from Hortonworks

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks
 
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyIoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
Hortonworks
 
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakGetting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Hortonworks
 
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsJohns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
Hortonworks
 
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysCatch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Hortonworks
 
HDF 3.2 - What's New
HDF 3.2 - What's NewHDF 3.2 - What's New
HDF 3.2 - What's New
Hortonworks
 
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Hortonworks
 
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsInterpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Hortonworks
 
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeIBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
Hortonworks
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
Hortonworks
 
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleAccelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
Hortonworks
 
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATATIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
Hortonworks
 
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Hortonworks
 
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseDelivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Hortonworks
 
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseMaking Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
Hortonworks
 
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationWebinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Hortonworks
 
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementDriving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
Hortonworks
 
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
Hortonworks
 
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks
 
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCUnlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Hortonworks
 

More from Hortonworks (20)

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
 
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyIoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
 
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakGetting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
 
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsJohns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
 
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysCatch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
 
HDF 3.2 - What's New
HDF 3.2 - What's NewHDF 3.2 - What's New
HDF 3.2 - What's New
 
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
 
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsInterpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
 
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeIBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
 
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleAccelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
 
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATATIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
 
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
 
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseDelivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
 
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseMaking Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
 
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationWebinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
 
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementDriving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
 
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
 
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
 
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCUnlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
 

Recently uploaded

EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
moinahousna
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
kumarjarun2010
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
Axel Rennoch
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
Anant Gupta
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 

Recently uploaded (20)

EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 

Ozone- Object store for Apache Hadoop

  • 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone – Object Store for Apache Hadoop Anu Engineer aengineer@apache.org Arpit Agarwal arp@apache.org
  • 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone – Why an Object Store ⬢ With workloads like IoT we are looking at scaling to trillions of objects. ⬢ Apache HDFS is designed for large objects – not for many small objects – Small files create memory pressure on namenode. ⬢ Each small file creates a block in the datanode. –Datanodes send all block information to namenode in BlockReports. ⬢ Both of these create scalability issues on Namenode. ⬢ Metadata in memory is the strength of the original GFS and HDFS design, but also its weakness in scaling number of files and blocks. ⬢ An object store has simpler semantics than a file system and is easier to scale Apache Hadoop, Hadoop, Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and other countries.
  • 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone – Why an Object Store (continued) ⬢ Ozone attempts to scale to trillions of objects – This presentation is about how we will get there. ⬢ Ozone is built on a distributed metadata store. ⬢ Avoids any single server becoming a bottleneck ⬢ More parallelism possible in both data and metadata operations ⬢ Build on well tested components and understood protocols –RAFT for consensus •RAFT is a protocol for reaching consensus between a set of machines in an unreliable environment where machines and network may fail. –Off-the-shelf Key-Value store like LevelDB •LevelDB is an open-source standalone key-value store built by Google.
  • 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Alternative solutions to NameNode scalability ⬢ HDFS federation aims to address namespace and Block Space scalability issues. –Federation deployment and planning adds complexity –Requires changes to other components in the Hadoop stack ⬢ HDFS-8286 - Partial Namespace In Memory. –Proposal to keep active working set of namespace in memory. ⬢ HDFS-5477 - Block Management as a Service. –Proposed solution for block space scalability issue. ⬢ Ozone borrows many ideas from these and is a super set of these approaches.
  • 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Presentation Outline Ozone Introduction Ozone Architectural Overview Containers Ozone - Bringing it all together Bonus Slides - if we have time.
  • 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Introduction ⬢ An Ozone URL –http://hostname/myvolume/mybucket/mykey ⬢ An S3 URL –http://hostname/mybucket/mykey ⬢ An Azure URL –http://hostname/myaccount/mybucket/key
  • 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Definitions ⬢ Storage Volume –A notion similar to an account –Allows admin controls on usage of the object store e.g. storage quota –Created and managed by admins only ⬢ Bucket –Consists of keys and objects –Similar to a bucket in S3 or a container in Azure –ACLs
  • 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Definitions (continued) ⬢ Storage Key –Unique in a bucket ⬢ Object –Values in a bucket –Each corresponds to a unique key within a bucket
  • 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - REST API ⬢ POST - Creates Volumes and Buckets –Only Admin creates volumes –Bucket can be created by owner of the volume ⬢ PUT - Updates Volumes , Buckets and creates keys –Only admin can change some volume settings –Buckets have ACLs –Creates Keys
  • 10. 1 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - REST API (continued) ⬢ GET - Lists volumes and buckets and allows reading of keys –Lists Volumes –List Buckets –Get Keys ⬢ DELETE - Deletes volumes, buckets and keys. –Delete Volumes –Delete Buckets –Removes the Key
  • 11. 1 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Components ⬢ Containers – Actual storage locations on Datanodes. –We acknowledge the term container is overloaded. No relation to YARN containers or LXC. –Assume container means a collection of blocks on a datanode for now. –Containers deep dive to follow. ⬢ Ozone Handler - REST frontend for the Ozone rest protocol - deployed on datanodes. ⬢ Storage Container Manager (SCM) - Manages the container life cycle. ⬢ Ozone Key Space Manager (KSM) - Maps Ozone entities to Containers. ⬢ Container Client - Talks to KSM to discover the location of a container and sends IO requests to the appropriate container.
  • 12. 1 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Overview
  • 13. 1 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager
  • 14. 1 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager ⬢ Key to container mapping service. ⬢ Keeps the key ranges to containers mapping in memory. –Θ(number of containers) - 1 Exabyte cluster = 200M containers x 5GB each. –Memory usage scales with number of containers and not number of keys. ⬢ KSM does NOT know about all the keys in the system. ⬢ KSM state is replicated via RAFT, NameNode-like snapshots.
  • 15. 1 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager ⬢ KSM knows about Ozone Volumes and Buckets. ⬢ KSM keeps a map of Volumes to container and buckets to containers. ⬢ KSM performs longest prefix match on a given string. ⬢ Example: The user wants to lookup a key - “/volume1/bucket1/key1” –KSM authenticates the user, maps this key to a container and looks up the container location. –Container client gets a token from the KSM and talks to the container on the data node. –Container client makes a getKey call to the datanode container with the full key path. –DataNode validates the access token and serves the value. ⬢ Contents of a bucket may span multiple containers.
  • 16. 1 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved KSM - Bucket spanning multiple containers
  • 17. 1 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved1 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers
  • 18. 1 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Container Framework ⬢ A shareable generic block service that can be used by distributed storage services. ⬢ Make it easier to develop new storage services - BYO storage format and naming scheme. ⬢ Design Goals –No single points of failure. All services are replicated. –Avoid bottlenecks •Minimize central state •No verbose Block Reports ⬢ Lessons learned from large scale HDFS clusters. ⬢ Ideas from earlier proposals in HDFS community.
  • 19. 1 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Container Framework Components
  • 20. 2 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers
  • 21. 2 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers ⬢ A container is the unit of replication –Size bounded by how quickly it can be re-replicated after a loss. ⬢ Each container is an independent key-value store. –No requirements on the structure or format of keys/values. –Keys are unique only within a container. ⬢ E.g. key-value pair could be one of –An Ozone Key-Value pair –An HDFS block ID and block contents •Or part of a block, when a block spans containers.
  • 22. 2 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers (continued) ⬢ Each container has metadata – Metadata consistency maintained via the RAFT protocol – Metadata consists of keys and references to chunks. ⬢ Container metadata stored in LevelDB. – Exact choice of KV store unimportant. LevelDB is already used by other Hadoop components. ⬢ A chunk is a piece of user data. – Chunks are replicated via a data pipeline. – Chunks can be of arbitrary sizes e.g. a few KB to GBs. – Each chunk reference is a (file, offset, length) triplet. ⬢ Containers may garbage collect unreferenced chunks. ⬢ Each container independently decides how to map chunks to files – Allow reauthoring files for performance, compaction and overwrites.
  • 23. 2 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers (continued)
  • 24. 2 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers support simple client operations ⬢ Write chunks - streaming writes ⬢ Put(key, List<ChunkReference>) –The value is a list of chunk references. –Putting a key makes previously written chunk data visible to readers. –Put overwrites previous versions of the key. ⬢ Get(key) –Returns a list of chunk references ⬢ Read chunks - streaming reads ⬢ Delete(key) ⬢ List Keys
  • 25. 2 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Container Manager
  • 26. 2 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Container Manager ⬢ A fault-tolerant replicated service ⬢ Replicates its own state using RAFT protocol ⬢ Provides Container Location Service to clients –Given a container ID, return a list of nodes with replicas –Mapping a container ID to Data Nodes (and vice versa) ⬢ Provides Cluster Membership Management –Maintain list of live Data Nodes in the cluster –Handle heartbeats from DataNodes
  • 27. 2 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Container Manager (continued) ⬢ Provides Replication services –Detect lost container replicas and initiate re-replication –Containers send a container report. •Unlike HDFS block reports which include details about each block , a container report is a summary of information. •This is used by KSM for placement of containers ⬢ If a node suffers from disk failure or if a node is lost, the reconstruction is a local activity which is coordinated via RAFT running on the data nodes.
  • 28. 2 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Container Manager ⬢ Maintains pre-created containers ⬢ Collects container operation statistics ⬢ Decides which Data Nodes form the replication set for a given container. –The number of replication sets in a cluster is bounded –Borrowing the work done by Facebook and RAMCloud (Copysets, Cidon et al. 2013). ⬢ Important - Knows nothing about keys –Does NOT provide Naming Service (mapping keys to containers) –e.g. KSM provides naming for Ozone.
  • 29. 2 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Conceptual Representation of Ozone and Container State
  • 30. 3 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Bringing it all together
  • 31. 3 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bringing it all together - Ozone createVolume operations Key Space Manager Replicated containers 1: createVolume Container Manager 2: Lookup(volName, Operation) 3: getContainer 4: putMetdata(VolumeName, Properties) Ozone HandlerClient Heartbeats & Reports DataNodes
  • 32. 3 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tracing an Ozone PutKey Key Space Manager Replicated containers 1: Ozone - putKey 2: Lookup(keyName, Operation) 4: putData(File, offset, Length, data) Client 5: putMetadata(key, List<chunks>) Ozone Handler DataNodes
  • 33. 3 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tracing an Ozone createVolume OzoneVolume vol = (new VolumeBuilder(pipeLine)) .setCreated(new Date()) .setOwnerName("bilbo") .setClient(client) .setName(“shire”) .build(); POST /shire keyData = {ContainerKeyData} keyName = "shire" containerName = "OzoneContainer" metadata = 0."Created" -> "1449533074362" 1."CreatedBy" -> "gandalf" 2."Key" -> "VOLUME" 3."Owner" -> "bilbo" Ozone REST Handler code Container wire and storage format
  • 34. 3 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Metadata operations ⬢ Any metadata write to a container is Replicated via RAFT. ⬢ Machines forming the replication set for a container comprise a pipeline. ⬢ A createVolume call reduces to putKey operation on the container. ⬢ putKey is consistent, atomic and durable. ⬢ All metadata data operations are done via putKey, getKey and deleteKey. ⬢ Data is written to one or more chunks and a key is updated to point to those chunks. ⬢ Updating the key makes the data visible in the namespace.
  • 35. 3 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Current State of Ozone ⬢ Stand alone container framework. ⬢ Single node ozone using container framework. ⬢ Full REST API -- Command Line Tools and Client Libs are fully functional. ⬢ Active development in branch HDFS-7240. ⬢ Work in progress: –SCM –KSM –Replication Pipeline –RAFT
  • 36. 3 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Acknowledgements ⬢ Ozone is being designed and developed by Jitendra Pandey, Chris Nauroth, Tsz Wo (Nicholas) Sze, Jing Zhao, Suresh Srinivas, Sanjay Radia, Anu Engineer and Arpit Agarwal. ⬢ The Apache community has been very helpful and we were supported by comments and contributions from Kanaka Kumar Avvaru, Edward Bortnikov, Thomas Demoor, Nick Dimiduk, Chris Douglas, Jian Fang, Lars Francke, Gautam Hegde, Lars Hofhansl, Jakob Homan, Virajith Jalaparti, Charles Lamb, Steve Loughran, Haohui Mai, Colin Patrick McCabe, Aaron Myers, Owen O’Malley, Liam Slusser, Jeff Sogolov, Enis Soztutar, Andrew Wang, Fengdong Yu, Zhe Zhang & khanderao.
  • 37. 3 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You
  • 38. 3 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bonus Slides
  • 39. 3 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ KSM deals with dynamic partitioning of containers. ⬢ Let us say that a user starts by uploading all his photographs to a bucket in ozone. ⬢ Since all the photographs are called IMG_* (thanks Apple), we will soon overflow the 5GB capacity of the container. ⬢ At this point we need to split the container.
  • 40. 4 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone KSM - Dynamic Container Partitioning ⬢ The container client attempts to write the Nth ozone key, IMG_N, gets a partition required error. ⬢ Container client will take that error and return that info to KSM. ⬢ That error contains the info -- about the proposed split -- That is IMG_0- IMG_200 will stay in this container and IMG_201-IMG_400 will move to next container. ⬢ Note: KSM initiates container partitioning but mechanics of the split are handled by the Container Layer
  • 41. 4 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ One of the assumptions we have made about a container split is that the splits are on the same datanode as the original container. ⬢ This allows us to reduce a split operation to a copy of Keys from one LevelDB to another LevelDB. ⬢ if we need to move actual file data from one datanode to another -- we do support container moves. However they are slow. ⬢ A split on the other hand will complete in seconds in most cases. ⬢ The split point is chosen by the container so that we are able to pick the 50th percentile position that gives us reasonable chance at an equal partition of a container. ⬢ KSM does not know about the keys or the actual data sizes until much later. ⬢ So always relies on the container to tell it where the split should be.
  • 42. 4 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ A container split is done in KSM via updating the Tree. The range partition key we maintain gets updated to reflect the fact that Keys - {IMG_0 - IMG_200} are on container C1, and keys {IMG_201-IMG_Z} are on C2. ⬢ Container will update the SCM when the split is done. ⬢ This information is learned and maintained by KSM.
  • 43. 4 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Soft Quotas ⬢ In the HDFS world, a Quota is hard limit. It is actually conservative in quota management. ⬢ In the ozone world, Quotas are soft quotas. That is users can and will be able to violate it, but KSM/SCM will eventually learn about it and lock the volume out. ⬢ The key reason why this is different is because KSM/SCM is not involved in the allocation of chunks. ⬢ The containers have a partial -- that is an isolated view of the data in a volume. Since volumes can span many containers, it is possible for users to allocate chunks that violate the volume quota, but eventually KSM will learn and disable further writes to a volume.
  • 44. 4 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Missing namespace problem ⬢ One great thing about HDFS is Namenode. –Despite scalability issues, in most cases Namenode does a wonderful job. ⬢ Ozone - Subtle problem if we lose the all replicas of a container. ⬢ We will not only lose data -- just as if HDFS lost its all 3 replicas, but we will also lose information about which keys have been lost. ⬢ To solve this issue, we propose to have a on-disk eventually consistent log maintained by a separate service. –Records information about the keys that exist in the cluster. ⬢ This Scribe service logs the state of the cluster.
  • 45. 4 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Range reads ⬢ Ozone supports range reads and might support range writes like part upload in S3. –Ozone achieves this by using the chunk mechanism. –Chunks offer a stream like interface. –You can seek to any location and read as many bytes as you want. –This is used by ozone to support range reads ⬢ Periodic Scanner can reclaim unreferenced chunks.
  • 46. 4 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Scalability - What HDFS does well ⬢ HDFS NN stores all namespace metadata in memory (as per GFS) –Scales to large clusters (5K) since all metadata in memory •60K-100K tasks can share the Namenode •Low latency –Large data if files are large ⬢ Proof points of large data and large clusters –Single Organizations have over 600PB in HDFS –Single clusters with over 200PB using federation –Large clusters of over 4K multi-core nodes hitting a single NN