Zookeeper is a distributed coordination service that provides naming, configuration, synchronization, and group services. It allows distributed processes to coordinate with each other through a shared hierarchical namespace of data registers called znodes. Zookeeper follows a leader-elected consensus protocol to guarantee atomic broadcast of state updates from the leader to followers. It uses a hierarchical namespace of znodes similar to a file system to store configuration data and other application-defined metadata. Zookeeper provides services like leader election, group membership, synchronization, and configuration management that are essential for distributed systems.
So we're running Apache ZooKeeper. Now What? By Camille Fournier Hakka Labs
The ZooKeeper framework was originally built at Yahoo! to make it easy for the company’s applications to access configuration information in a robust and easy-to-understand way, but it has since grown to offer a lot of features that help coordinate work across distributed clusters. Apache Zookeeper became a de-facto standard for coordination service and used by Storm, Hadoop, HBase, ElasticSearch and other distributed computing frameworks.
Introduction to ZooKeeper - TriHUG May 22, 2012mumrah
Presentation given at TriHUG (Triangle Hadoop User Group) on May 22, 2012. Gives a basic overview of Apache ZooKeeper as well as some common use cases, 3rd party libraries, and "gotchas"
Demo code available at https://github.com/mumrah/trihug-zookeeper-demo
Centralized Application Configuration with Spring and Apache ZookeeperRyan Gardner
From talk given at Spring One 2gx Dallas, 2014
Application configuration is an evolution. It starts as a hard-coded strings in your application and hopefully progresses to something external, such as a file or system property that can be changed without deployment. But what happens when other enterprise concerns enter the mix, such as audit requirements or access control around who can make changes? How do you maintain the consistency of values across too many application servers to manage at one time from a terminal window? The next step in the application configuration evolution is centralized configuration that can be accessed by your applications as they move through your various environments on their way to production. Such a service transfers the ownership of configuration from the last developer who touched the code to a well-versed application owner who is responsible for the configuration of the application across all environments. At Dealer.com, we have created one such solution that relies on Apache ZooKeeper to handle the storage and coordination of the configuration data and Spring to handle to the retrieval, creation and registration of configured objects in each application. The end result is a transparent framework that provides the same configured objects that could have been created using a Spring configuration, configuration file and property value wiring. This talk will cover both the why and how of our solution, with a focus on how we leveraged the powerful attributes of both Apache ZooKeeper and Spring to rid our application of local configuration files and provide a consistent mechanism for application configuration in our enterprise.
This is Apache ZooKeeper session.
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
By the end of this presentation you should be fairly clear about Apache ZooKeeper.
To watch the video or know more about the course, please visit
http://www.knowbigdata.com/page/big-data-and-hadoop-online-instructor-led-training
So we're running Apache ZooKeeper. Now What? By Camille Fournier Hakka Labs
The ZooKeeper framework was originally built at Yahoo! to make it easy for the company’s applications to access configuration information in a robust and easy-to-understand way, but it has since grown to offer a lot of features that help coordinate work across distributed clusters. Apache Zookeeper became a de-facto standard for coordination service and used by Storm, Hadoop, HBase, ElasticSearch and other distributed computing frameworks.
Introduction to ZooKeeper - TriHUG May 22, 2012mumrah
Presentation given at TriHUG (Triangle Hadoop User Group) on May 22, 2012. Gives a basic overview of Apache ZooKeeper as well as some common use cases, 3rd party libraries, and "gotchas"
Demo code available at https://github.com/mumrah/trihug-zookeeper-demo
Centralized Application Configuration with Spring and Apache ZookeeperRyan Gardner
From talk given at Spring One 2gx Dallas, 2014
Application configuration is an evolution. It starts as a hard-coded strings in your application and hopefully progresses to something external, such as a file or system property that can be changed without deployment. But what happens when other enterprise concerns enter the mix, such as audit requirements or access control around who can make changes? How do you maintain the consistency of values across too many application servers to manage at one time from a terminal window? The next step in the application configuration evolution is centralized configuration that can be accessed by your applications as they move through your various environments on their way to production. Such a service transfers the ownership of configuration from the last developer who touched the code to a well-versed application owner who is responsible for the configuration of the application across all environments. At Dealer.com, we have created one such solution that relies on Apache ZooKeeper to handle the storage and coordination of the configuration data and Spring to handle to the retrieval, creation and registration of configured objects in each application. The end result is a transparent framework that provides the same configured objects that could have been created using a Spring configuration, configuration file and property value wiring. This talk will cover both the why and how of our solution, with a focus on how we leveraged the powerful attributes of both Apache ZooKeeper and Spring to rid our application of local configuration files and provide a consistent mechanism for application configuration in our enterprise.
This is Apache ZooKeeper session.
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
By the end of this presentation you should be fairly clear about Apache ZooKeeper.
To watch the video or know more about the course, please visit
http://www.knowbigdata.com/page/big-data-and-hadoop-online-instructor-led-training
Deploying and managing SolrCloud in the cloud using the Solr Scale Toolkitthelabdude
SolrCloud is a set of features in Apache Solr that enable elastic scaling of search indexes using sharding and replication. In this presentation, Tim Potter will demonstrate how to provision, configure, and manage a SolrCloud cluster in Amazon EC2, using a Fabric/boto based solution for automating SolrCloud operations. Attendees will come away with a solid understanding of how to operate a large-scale Solr cluster, as well as tools to help them do it. Tim will also demonstrate these tools live during his presentation. Covered technologies, include: Apache Solr, Apache ZooKeeper, Linux, Python, Fabric, boto, Apache Kafka, Apache JMeter.
Container monitoring for resource and application metrics with cAdvisor. Shipping monitoring information with the container so it is monitored irrespective of the host it runs on.
Intro to monitoring in distributed systems, cAdvisor, heapster, kubedash, kubernetes
In the big data world, our data stores communicate over an asynchronous, unreliable network to provide a facade of consistency. However, to really understand the guarantees of these systems, we must understand the realities of networks and test our data stores against them.
Jepsen is a tool which simulates network partitions in data stores and helps us understand the guarantees of our systems and its failure modes. In this talk, I will help you understand why you should care about network partitions and how can we test datastores against partitions using Jepsen. I will explain what Jepsen is and how it works and the kind of tests it lets you create. We will try to understand the subtleties of distributed consensus, the CAP theorem and demonstrate how different data stores such as MongoDB, Cassandra, Elastic and Solr behave under network partitions. Finally, I will describe the results of the tests I wrote using Jepsen for Apache Solr and discuss the kinds of rare failures which were found by this excellent tool.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein
In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
How to make a simple cheap high availability self-healing solr clusterlucenerevolution
Presented by Stephane Gamard, Chief Technology Officer, Searchbox
In this presentation we aim to show how to make a high availability Solr cloud with 4.1 using only Solr and a few bash scripts. The goal is to present an infrastructure which is self healing using only cheap instances based on ephemeral storage. We will start by providing a comprehensive overview of the relation between collections, Solr cores, shardes, and cluster nodes. We continue by an introduction to Solr 4.x clustering using zookeeper with a particular emphasis on cluster state status/monitoring and solr collection configuration. The core of our presentation will be demonstrated using a live cluster.
We will show how to use cron and bash to monitor the state of the cluster and the state of its nodes. We will then show how we can extend our monitoring to auto generate new nodes, attach them to the cluster, and assign them shardes (selecting between missing shardes or replication for HA). We will show that using a high replication factor it is possible to use ephemeral storage for shards without the risk of data loss, greatly reducing the cost and management of the architecture. Future work discussions, which might be engaged using an open source effort, include monitoring activity of individual nodes as to scale the cluster according to traffic and usage.
Sanger OpenStack presentation March 2017Dave Holland
A description of the Sanger Institute's journey with OpenStack to date, covering RHOSP, Ceph, S3, user applications, and future plans. Given at the Sanger Institute's OpenStack Day.
Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but can also be used for custom internal R&D built software applications too.
Solr Exchange: Introduction to SolrCloudthelabdude
SolrCloud is a set of features in Apache Solr that enable elastic scaling of search indexes using sharding and replication. In this presentation, Tim Potter will provide an architectural overview of SolrCloud and highlight its most important features. Specifically, Tim covers topics such as: sharding, replication, ZooKeeper fundamentals, leaders/replicas, and failure/recovery scenarios. Any discussion of a complex distributed system would not be complete without a discussion of the CAP theorem. Mr. Potter will describe why Solr is considered a CP system and how that impacts the design of a search application.
EFK Stack이란 ElasticSearch, Fluentd, Kibana라는 오픈소스의 조합으로, 방대한 양의 데이터를 신속하고 실시간으로 수집/저장/분석/시각화 할 수 있는 솔루션입니다. 특히 컨테이너 환경에서 로그 수집을 위해 주로 사용되는 기술 스택입니다.
Elasitc Stack에 대한 소개와 EFK Stack 설치 방법에 대해 설명합니다.
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)Docker, Inc.
Operating apps at web scale has become the new normal, but has been out of reach for most companies. Join us as we show you how to deploy and manage your Docker containers at scale. See how easy it is to build highly-available, fault-tolerant web scale apps using Docker with the Mesos cluster scheduler. Docker plus Mesos is a new way to scale applications. Together they give you capabilities similar to Google’s Borg, the Googleplex’s secret weapon of scalability and fault tolerance.
You have amazing content and you want to get it to your users as fast as possible. In today’s industry, milliseconds matter and slow websites will never keep up. You can use a CDN but they are expensive, make you dependent on a third party to deliver your content, and can be notoriously inflexible. Enter Varnish, a powerful, open-source caching reverse proxy that lives in your network and lets you take control of how your content is managed and delivered. We’ll discuss how to install and configure Varnish in front of a typical web application, how to handle sessions and security, and how you can customize Varnish to your unique needs. This session will teach you how Varnish can help you give your users a better experience while saving your company and clients money at the same time.
[Download the slide to get the entire talk in the form of presentation note embedded in the ppt] Apache ZooKeeper is the chosen leader in distributed coordination. In this talk, I have explored the atomic elements of Apache ZooKeeper, how it fits everything together and some of its popular use cases. For ZooKeeper simplicity is the key and as a consumer of the API, our imagination enables us to push the limits of the ZooKeeper world.
Deploying and managing SolrCloud in the cloud using the Solr Scale Toolkitthelabdude
SolrCloud is a set of features in Apache Solr that enable elastic scaling of search indexes using sharding and replication. In this presentation, Tim Potter will demonstrate how to provision, configure, and manage a SolrCloud cluster in Amazon EC2, using a Fabric/boto based solution for automating SolrCloud operations. Attendees will come away with a solid understanding of how to operate a large-scale Solr cluster, as well as tools to help them do it. Tim will also demonstrate these tools live during his presentation. Covered technologies, include: Apache Solr, Apache ZooKeeper, Linux, Python, Fabric, boto, Apache Kafka, Apache JMeter.
Container monitoring for resource and application metrics with cAdvisor. Shipping monitoring information with the container so it is monitored irrespective of the host it runs on.
Intro to monitoring in distributed systems, cAdvisor, heapster, kubedash, kubernetes
In the big data world, our data stores communicate over an asynchronous, unreliable network to provide a facade of consistency. However, to really understand the guarantees of these systems, we must understand the realities of networks and test our data stores against them.
Jepsen is a tool which simulates network partitions in data stores and helps us understand the guarantees of our systems and its failure modes. In this talk, I will help you understand why you should care about network partitions and how can we test datastores against partitions using Jepsen. I will explain what Jepsen is and how it works and the kind of tests it lets you create. We will try to understand the subtleties of distributed consensus, the CAP theorem and demonstrate how different data stores such as MongoDB, Cassandra, Elastic and Solr behave under network partitions. Finally, I will describe the results of the tests I wrote using Jepsen for Apache Solr and discuss the kinds of rare failures which were found by this excellent tool.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein
In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
How to make a simple cheap high availability self-healing solr clusterlucenerevolution
Presented by Stephane Gamard, Chief Technology Officer, Searchbox
In this presentation we aim to show how to make a high availability Solr cloud with 4.1 using only Solr and a few bash scripts. The goal is to present an infrastructure which is self healing using only cheap instances based on ephemeral storage. We will start by providing a comprehensive overview of the relation between collections, Solr cores, shardes, and cluster nodes. We continue by an introduction to Solr 4.x clustering using zookeeper with a particular emphasis on cluster state status/monitoring and solr collection configuration. The core of our presentation will be demonstrated using a live cluster.
We will show how to use cron and bash to monitor the state of the cluster and the state of its nodes. We will then show how we can extend our monitoring to auto generate new nodes, attach them to the cluster, and assign them shardes (selecting between missing shardes or replication for HA). We will show that using a high replication factor it is possible to use ephemeral storage for shards without the risk of data loss, greatly reducing the cost and management of the architecture. Future work discussions, which might be engaged using an open source effort, include monitoring activity of individual nodes as to scale the cluster according to traffic and usage.
Sanger OpenStack presentation March 2017Dave Holland
A description of the Sanger Institute's journey with OpenStack to date, covering RHOSP, Ceph, S3, user applications, and future plans. Given at the Sanger Institute's OpenStack Day.
Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but can also be used for custom internal R&D built software applications too.
Solr Exchange: Introduction to SolrCloudthelabdude
SolrCloud is a set of features in Apache Solr that enable elastic scaling of search indexes using sharding and replication. In this presentation, Tim Potter will provide an architectural overview of SolrCloud and highlight its most important features. Specifically, Tim covers topics such as: sharding, replication, ZooKeeper fundamentals, leaders/replicas, and failure/recovery scenarios. Any discussion of a complex distributed system would not be complete without a discussion of the CAP theorem. Mr. Potter will describe why Solr is considered a CP system and how that impacts the design of a search application.
EFK Stack이란 ElasticSearch, Fluentd, Kibana라는 오픈소스의 조합으로, 방대한 양의 데이터를 신속하고 실시간으로 수집/저장/분석/시각화 할 수 있는 솔루션입니다. 특히 컨테이너 환경에서 로그 수집을 위해 주로 사용되는 기술 스택입니다.
Elasitc Stack에 대한 소개와 EFK Stack 설치 방법에 대해 설명합니다.
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)Docker, Inc.
Operating apps at web scale has become the new normal, but has been out of reach for most companies. Join us as we show you how to deploy and manage your Docker containers at scale. See how easy it is to build highly-available, fault-tolerant web scale apps using Docker with the Mesos cluster scheduler. Docker plus Mesos is a new way to scale applications. Together they give you capabilities similar to Google’s Borg, the Googleplex’s secret weapon of scalability and fault tolerance.
You have amazing content and you want to get it to your users as fast as possible. In today’s industry, milliseconds matter and slow websites will never keep up. You can use a CDN but they are expensive, make you dependent on a third party to deliver your content, and can be notoriously inflexible. Enter Varnish, a powerful, open-source caching reverse proxy that lives in your network and lets you take control of how your content is managed and delivered. We’ll discuss how to install and configure Varnish in front of a typical web application, how to handle sessions and security, and how you can customize Varnish to your unique needs. This session will teach you how Varnish can help you give your users a better experience while saving your company and clients money at the same time.
[Download the slide to get the entire talk in the form of presentation note embedded in the ppt] Apache ZooKeeper is the chosen leader in distributed coordination. In this talk, I have explored the atomic elements of Apache ZooKeeper, how it fits everything together and some of its popular use cases. For ZooKeeper simplicity is the key and as a consumer of the API, our imagination enables us to push the limits of the ZooKeeper world.
Slides for presentation on ZooKeeper I gave at Near Infinity (www.nearinfinity.com) 2012 spring conference.
The associated sample code is on GitHub at https://github.com/sleberknight/zookeeper-samples
Distributed system coordination by zookeeper and introduction to kazoo python...Jimmy Lai
Zookeeper is a coordination tool to let people build distributed systems easier. In this slides, the author summarizes the usage of zookeeper and provides Kazoo Python library as example.
ZooKeeper - wait free protocol for coordinating processesJulia Proskurnia
ZooKeeper is a service for coordinating processes within distributed systems. Stress test of the tool was applied. Reliable Multicast and Dynamic LogBack system Configuration management were implemented with ZooKeeper.
More details: http://proskurnia.in.ua/wiki/zookeeper_research
Sunni Brown's Ted Talk on Doodling points our attention to how the negative connotation given to the word through history influences the way we react to doodling in today's society. At work, in schools, in court, if someone is "caught" doodling, it is treated as an offense, when in reality there are many learning benefits of doodling, because it helps us think! Khadeidra Le Gendre shares her interpretation through a visual: graphic recording of the TedTalk.
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab
Big Data with Hadoop & Spark Training: http://bit.ly/2kvXlPd
This CloudxLab Introduction to Apache ZooKeeper tutorial helps you to understand ZooKeeper in detail. Below are the topics covered in this tutorial:
1) Data Model
2) Znode Types
3) Persistent Znode
4) Sequential Znode
5) Architecture
6) Election & Majority Demo
7) Why Do We Need Majority?
8) Guarantees - Sequential consistency, Atomicity, Single system image, Durability, Timeliness
9) ZooKeeper APIs
10) Watches & Triggers
11) ACLs - Access Control Lists
12) Usecases
13) When Not to Use ZooKeeper
Presentation done at the November meeting of the Sudoers Barcelona group (https://www.meetup.com/sudoersbcn/).
HashiCorp Vault (https://www.vaultproject.io/)
"Vault és una eina per emmagatzemar i gestionar secrets. Veurem què ofereix, com instal·lar-la, utilitzar-la i operar-la, i la nostra experiència."
Organizations continue to adopt Solr because of its ability to scale to meet even the most demanding workflows. Recently, LucidWorks has been leading the effort to identify, measure, and expand the limits of Solr. As part of this effort, we've learned a few things along the way that should prove useful for any organization wanting to scale Solr. Attendees will come away with a better understanding of how sharding and replication impact performance. Also, no benchmark is useful without being repeatable; Tim will also cover how to perform similar tests using the Solr-Scale-Toolkit in Amazon EC2.
You can find the first part of this presentation here: https://www.slideshare.net/secret/pAvK8Qd9f07oa
This presentation takes a deep dive into how the Million Song Library, a microservices-based application, was built using the Netflix Stack, Cassandra and Datastax.
To learn more about Million Song Library and its components visit the project on GitHub: https://github.com/kenzanlabs/million-song-library
Lea
DCSF19 Container Security: Theory & Practice at NetflixDocker, Inc.
Michael Wardrop, Netflix
Usage of containers has undergone rapid growth at Netflix and it is still accelerating. Our container story started organically with developers downloading Docker and using it to improve their developer experience. The first production workloads were simple batch jobs, pioneering micro-services followed, then status as a first class platform running critical workloads.
As the types of workloads changed and their importance increased, the security of our container ecosystem needed to evolve and adapt. This session will cover some security theory, architecture, along with practical considerations, and lessons we learnt along the way.
Scaling SolrCloud to a Large Number of Collections - Fifth Elephant 2014Shalin Shekhar Mangar
The traditional and typical search use case is the one large search collection distributed among many nodes and shared by all users. However, there is a class of applications which need a large number of small or medium collections which can be used, managed and scaled separately. This talk will cover our effort in helping a client set up a large scale SolrCloud setup with thousands of collections running on hundreds of nodes. I will describe the bottlenecks that we found in SolrCloud when running a large number of collections. I will also take you through the multiple features and optimizations that we contributed to Apache Solr to reduce or remove the choke points in the system. Finally, I will talk about the benchmarking process and the lessons learned from supporting such an installation in production.
Flying to clouds - can it be easy? Cloud Native ApplicationsJacek Bukowski
Nowadays "cloud" and "microservice" terms are used all the time, even overused. Does any system must be the "microservices" deployed in the "cloud"? Definitely not! However once you see that your system may benefit from that architecture, the next question is how to get there - how to fly to the clouds?
Spring was always about simplifying the complicated aspects of your enterprise system. Netflix went to microservice architecture long before this term even was created. Both are very much contributed to open source software. How can you benefit from joint forces of the both?
JDD 2016 - Jacek Bukowski - "Flying To Clouds" - Can It Be Easy?PROIDEA
Nowadays "cloud" and "microservice" terms are used all the time, even overused. Does any system must be the "microservices" deployed in the "cloud"? Definitely not! However once you see that your system may benefit from that architecture, the next question is how to get there - how to fly to the clouds?
Spring was always about simplifying the complicated aspects of your enterprise system. Netflix went to microservice architecture long before this term even was created. Both are very much contributed to open source software. How can you benefit from joint forces of the both?
Multi-cluster Kubernetes Networking- Patterns, Projects and GuidelinesSanjeev Rampal
Talk presented at Kubernetes Community Day, New York, May 2024.
Technical summary of Multi-Cluster Kubernetes Networking architectures with focus on 4 key topics.
1) Key patterns for Multi-cluster architectures
2) Architectural comparison of several OSS/ CNCF projects to address these patterns
3) Evolution trends for the APIs of these projects
4) Some design recommendations & guidelines for adopting/ deploying these solutions.
This 7-second Brain Wave Ritual Attracts Money To You.!nirahealhty
Discover the power of a simple 7-second brain wave ritual that can attract wealth and abundance into your life. By tapping into specific brain frequencies, this technique helps you manifest financial success effortlessly. Ready to transform your financial future? Try this powerful ritual and start attracting money today!
ER(Entity Relationship) Diagram for online shopping - TAEHimani415946
https://bit.ly/3KACoyV
The ER diagram for the project is the foundation for the building of the database of the project. The properties, datatypes, and attributes are defined by the ER diagram.
1.Wireless Communication System_Wireless communication is a broad term that i...JeyaPerumal1
Wireless communication involves the transmission of information over a distance without the help of wires, cables or any other forms of electrical conductors.
Wireless communication is a broad term that incorporates all procedures and forms of connecting and communicating between two or more devices using a wireless signal through wireless communication technologies and devices.
Features of Wireless Communication
The evolution of wireless technology has brought many advancements with its effective features.
The transmitted distance can be anywhere between a few meters (for example, a television's remote control) and thousands of kilometers (for example, radio communication).
Wireless communication can be used for cellular telephony, wireless access to the internet, wireless home networking, and so on.
2. What is distributed system ?
• A distributed systems provides single coherent view of collection
of independent computing machines.
• Multiple software components on multiple computers, which run
as a single system.
• It can be locally or geographically connected.
4. Delusion of Distributed System
• Reliable Network
• Zero Latency
• Infinite Bandwidth
• Secure Network
• Topology Doesn’t change
• Homogeneous Network
• There is one administrator
8. Zookeeper Architecture
• Zookeeper follows client server architecture
• Collection of Zookeeper servers is called an ensemble
• At given time, a Zookeeper client is connected to one Zookeeper
server
• Client read requests are catered by connected Zookeeper server
• Client write requests are forwarded to Zookeeper leader.
• Write is considered successful, if quorum is achieved.
10. ZAB Protocol
• Zookeeper Atomic Broadcast
• Propagates state changes from leader to followers
• Guarantees the state change
11. Znodes
• Organized similar to standard Unix and Linux file system
• Similar to files and directories
• Both directory and leaf znode holds data
• Stores data <= 1MB
13. Time in Zookeeper
• Zxid : Every change to ZK state receives a stamp in form of zxid
• Version Number : Change to Znode will cause change in Version
• Ticks : Ticks define timing of events
• Real Time : To put timestamp of stat structure and znode
modification
14. Znode Stat Structure
• Czxid : Change that caused Znode to be created
• Mzxid : Change that last modified Znode
• Pzxid : Change that last modified children of the Znode
• Ctime : Millisecond from epoch, when this Znode was created
• Mtime : Millisecond from epoch, when this Znode was last modified
• Version : Number of changes to data of Znode
• Cversion : Number of changes to children of Znode
• Aversion : Number of changes to ACL of Znode
• EphemeralOwner : SessionId of owner of this Znode
• DataLength : Length of data field
• Numchildren : Number of children
15. Zookeeper Operations
create Creates Znode
delete Deletes Znode
exists Does Znode exist?
getACL Get ACL permissions
setACL Set ACL permissions
17. Types of Znodes
• Ephemeral Znode : Deleted with termination of client session,
also can’t have any child znode.
• Persistent Znode : Can have child znode, It is explicitly required
to delete it.
• Sequential Znode : Append a monotonicly increasing counter to
the end of path. It is unique to parent znode.
18. Znode Watchers
• One time triggers
• Sent to client which sets the watch
• Maintained locally at zookeeper
• Client won’t receive watches after getting disconnected
• Two lists of watches
– Data watches : create(),setData(), delete()
– Child watches : create(), delete()
22. Netflix/Soabase Exhibitor
• Zookeeper server is fail fast
• Zookeeper is self-healing, (i.e. crashed/failed server if brought
online, will automatically join the cluster)
• Instance Monitoring
• Log Cleanup
• Backup/Restore
• Cluster-wide configuration
• Rolling Ensemble Change
• Rich REST API
23. Backup/Exhibitor
• Online Backup - Use Netflix Exhibitor
• Offline Backup - Bring down the old ensemble cluster and copy
the zookeeper data directory to the new ensemble cluster
27. Data Directory Cleanup
• autopurge.snapRetainCount : Maintains n number of recent
snapshots along with transaction log
• autopurge.purgeInterval : Time Interval at which purge event
will trigger
28. Zookeeper Future Releases - 3.5.X
• Dynamic Reconfiguration w/o Rolling restarts
• Separate Dynamic configuration file
• Adding/Removing Servers
• Modifying server parameters
• Rebalance Client Connections
• Backward Compatible
• Modifying Server Roles[Obersver/Participants]
29. Zookeeper in Real World
• Apache Kafka
• Apache Storm
• Apache HBase
• Apache Solr
• Apache Mesos
• Apache Hive