This document provides an overview of joining a peer-to-peer (p2p) conversation about Apache Cassandra. It begins with introductions from Joaquin Casares of The Last Pickle consulting firm. It then covers definitions of p2p networks and examples like KaZaA and BitTorrent. Specific Cassandra topics discussed include token ownership and virtual nodes, seed nodes, bootstrapping, replacing addresses, and rebuilding nodes. Throughout, analogies are used to explain concepts in conversational terms between network "peers".
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...DataStax
Successfully running Apache Cassandra in production often means knowing what configuration settings to change and which ones to leave as default. Over the years the cassandra.yaml file has grown to provide a number of settings that can improve stability and performance. While the file contains plenty of helpful comments, there is more to be said about the settings and when to change them.
In this talk Edward Capriolo, Consultant at The Last Pickle, will break down the parameters in the configuration files. Looking at those that are essential to getting started, those that impact performance, those that improve availability, the exotic ones, and the ones that should not be played with. This talk is ideal for someone someone setting up Cassandra for the first time up to people with deployments in productions and wondering what the more exotic configuration options do.
About the Speaker
Edward Capriolo Consultant, The Last Pickle
Long time Apache Cassandra user, big data enthusiast.
Orchestrating Big Data pipelines @ Fandom - Krystian Mistrzak Thejas MurthyEvention
Fandom is the largest entertainment fan site in the world. With more than 360,000 fan communities and a global
audience of over 190 million monthly uniques, we are the fan’s voice in entertainment. Being the largest entertainment site, wikia generates massive volumes of data, which varies from clickstream, user activities, api requests, ad delivery, A/B testing and much more. The big challenge is not just the volume but the orchestration involved in combining various sources of data with various periodicity, volumes. And Making sure the processed data is available for the consumers within the expected time. Thus helping gain the right insights well within the right time. A conscious decision was made to choose the right open source tool to solve the problem of orchestration, after evaluating various tools we decided to use Apache airflow. This presentation will give an overview of comparisons of existing tools and emphasize on why we choose airflow. And how Airflow is being used to create a stable reliable orchestration platform to enable non data engineers to seamlessly access data by democratizing data. We will focus on some tricks and best practises of developing workflows with Airflow and show how we are using some of the features of airflow.
Things YouShould Be Doing When Using Cassandra DriversRebecca Mills
Did you know there are some things you should doing when writing your data driven application using Apache Cassandra? Let’s talk about that. There are features in almost every driver that will keep your application online and running fast. You should know what they are. Even if you know these features, I’m going to tell why they work and why they’re a good idea. This will not be language specific; I will be using multiple languages and drivers. This talk should appeal to programmers in general.
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...DataStax
Successfully running Apache Cassandra in production often means knowing what configuration settings to change and which ones to leave as default. Over the years the cassandra.yaml file has grown to provide a number of settings that can improve stability and performance. While the file contains plenty of helpful comments, there is more to be said about the settings and when to change them.
In this talk Edward Capriolo, Consultant at The Last Pickle, will break down the parameters in the configuration files. Looking at those that are essential to getting started, those that impact performance, those that improve availability, the exotic ones, and the ones that should not be played with. This talk is ideal for someone someone setting up Cassandra for the first time up to people with deployments in productions and wondering what the more exotic configuration options do.
About the Speaker
Edward Capriolo Consultant, The Last Pickle
Long time Apache Cassandra user, big data enthusiast.
Orchestrating Big Data pipelines @ Fandom - Krystian Mistrzak Thejas MurthyEvention
Fandom is the largest entertainment fan site in the world. With more than 360,000 fan communities and a global
audience of over 190 million monthly uniques, we are the fan’s voice in entertainment. Being the largest entertainment site, wikia generates massive volumes of data, which varies from clickstream, user activities, api requests, ad delivery, A/B testing and much more. The big challenge is not just the volume but the orchestration involved in combining various sources of data with various periodicity, volumes. And Making sure the processed data is available for the consumers within the expected time. Thus helping gain the right insights well within the right time. A conscious decision was made to choose the right open source tool to solve the problem of orchestration, after evaluating various tools we decided to use Apache airflow. This presentation will give an overview of comparisons of existing tools and emphasize on why we choose airflow. And how Airflow is being used to create a stable reliable orchestration platform to enable non data engineers to seamlessly access data by democratizing data. We will focus on some tricks and best practises of developing workflows with Airflow and show how we are using some of the features of airflow.
Things YouShould Be Doing When Using Cassandra DriversRebecca Mills
Did you know there are some things you should doing when writing your data driven application using Apache Cassandra? Let’s talk about that. There are features in almost every driver that will keep your application online and running fast. You should know what they are. Even if you know these features, I’m going to tell why they work and why they’re a good idea. This will not be language specific; I will be using multiple languages and drivers. This talk should appeal to programmers in general.
Flowr offers a scatter-gather approach, and works by submitting a web of jobs to the computing cluster using dependencies. It requires two simple data.frames as inputs and supports platforms like LSF, SGE, Torque and SLURM.
2017-10-24 All Day DevOps - Disposable Development EnvironmentsBoyd Hemphill
Why let your developers suffer in their own private, bespoke hell when you can standardize to improve non-functional requirements with Vagrant and Docker?
Basic stuff You Need to Know about CassandraYu-Chang Ho
This slide is intended to instruct the backend service team members of PM2.5 Open Data Service (pm25.lass-net.org) to learn the basic stuff about Apache Cassandra.
Rafał Ostrowski: Node.js jest znany ze swojej wydajności przy obsługiwaniu wielu operacji I/O. Ale co w momencie, gdy przychodzi potrzeba obsługi zadań przy których wiatraki naszych CPU zaczynają podkręcać obroty? Czy nasze API jest wtedy skazane na czkawkę? Na prezentacji co nieco o Worker Threads oraz o klastrach i procesach, które ułatwią nam budowanie skalowalnych aplikacji.
Going Multiplayer With Kafka With Ben Gamble | Current 2022HostedbyConfluent
Going Multiplayer With Kafka With Ben Gamble | Current 2022
Today we’ll walk through building multi-user and multiplayer spaces for games, collaboration, and for creation, leveraging Apache Kafka® for state management, and stream processing to handle conflicts and atomic edits.
We’re not building the metaverse! But as technology matures, ideas jump between disciplines, from ReactJS using ideas from game rendering, to the recent innovations in ECS patterns that borrow heavily from column stores in databases - there’s never been a better time to bring ideas from one sector of software engineering to another. Apache Kafka® makes event management simple so what can we borrow to make it collaborative?
Starting with a simple chat application and working into cursor sharing, collaborative editing and even a multiplayer game, we’ll walk through how to collect and manage user inputs,, and how backing onto an event log allows for version control, undos and time travel. We’ll also explore the various ways you can build up a canonical source of truth in a distributed system, from snapshots, to lockstep sync, to eventual consistency. Along the way we’ll learn a bit about CRDTs (conflict-free replicated data types), mergeable data structures and some of the ways to manage this complexity effectively.
OSDC 2019 | Storage Wars – Using Ceph since Firefly by Achim LedermüllerNETWAYS
Ceph claims to be a perfect unified file system for almost every purpose. In theory this can be true but in reality there are of course limitations. Following our way from Firefly (2014) to Luminous (2018) I will show our use cases, the shortcomings and the pitfalls we fell into! This will show Ceph in use for typical cloud and container stacks, simple media archives for applications and the devilish high available shared file systems.
C* Summit 2013: Practice Makes Perfect: Extreme Cassandra Optimization by Alb...DataStax Academy
Ooyala has been using Apache Cassandra since version 0.4. Our data ingest volume has exploded since 0.4 and Cassandra has scaled along with us. Al will cover many topics from an operational perspective on how to manage, tune, and scale Cassandra in a production environment.
It's easy enough to test the correctness of the infracode we write with unit-tests and parsers, but testing it does what it's supposed to do in the enviornment itself is a little more challenging. In this talk, I'm going to talk about some of the tools and approaches to use to test your configuration automation tool of choice.
C* Summit EU 2013: Practice Makes Perfect: Extreme Cassandra OptimizationDataStax Academy
Speaker: Al Tobey, Open Source Mechanic at DataStax
Video: http://www.youtube.com/watch?v=AcPME94F13U&list=PLqcm6qE9lgKLoYaakl3YwIWP4hmGsHm5e&index=24
Ooyala has been using Apache Cassandra since version 0.4. Our data ingest volume has exploded since 0.4 and Cassandra has scaled along with us. Al will cover many topics from an operational perspective on how to manage, tune, and scale Cassandra in a production environment.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
More Related Content
Similar to Joining a p2p Conversation - 2017-06 Meetup
Flowr offers a scatter-gather approach, and works by submitting a web of jobs to the computing cluster using dependencies. It requires two simple data.frames as inputs and supports platforms like LSF, SGE, Torque and SLURM.
2017-10-24 All Day DevOps - Disposable Development EnvironmentsBoyd Hemphill
Why let your developers suffer in their own private, bespoke hell when you can standardize to improve non-functional requirements with Vagrant and Docker?
Basic stuff You Need to Know about CassandraYu-Chang Ho
This slide is intended to instruct the backend service team members of PM2.5 Open Data Service (pm25.lass-net.org) to learn the basic stuff about Apache Cassandra.
Rafał Ostrowski: Node.js jest znany ze swojej wydajności przy obsługiwaniu wielu operacji I/O. Ale co w momencie, gdy przychodzi potrzeba obsługi zadań przy których wiatraki naszych CPU zaczynają podkręcać obroty? Czy nasze API jest wtedy skazane na czkawkę? Na prezentacji co nieco o Worker Threads oraz o klastrach i procesach, które ułatwią nam budowanie skalowalnych aplikacji.
Going Multiplayer With Kafka With Ben Gamble | Current 2022HostedbyConfluent
Going Multiplayer With Kafka With Ben Gamble | Current 2022
Today we’ll walk through building multi-user and multiplayer spaces for games, collaboration, and for creation, leveraging Apache Kafka® for state management, and stream processing to handle conflicts and atomic edits.
We’re not building the metaverse! But as technology matures, ideas jump between disciplines, from ReactJS using ideas from game rendering, to the recent innovations in ECS patterns that borrow heavily from column stores in databases - there’s never been a better time to bring ideas from one sector of software engineering to another. Apache Kafka® makes event management simple so what can we borrow to make it collaborative?
Starting with a simple chat application and working into cursor sharing, collaborative editing and even a multiplayer game, we’ll walk through how to collect and manage user inputs,, and how backing onto an event log allows for version control, undos and time travel. We’ll also explore the various ways you can build up a canonical source of truth in a distributed system, from snapshots, to lockstep sync, to eventual consistency. Along the way we’ll learn a bit about CRDTs (conflict-free replicated data types), mergeable data structures and some of the ways to manage this complexity effectively.
OSDC 2019 | Storage Wars – Using Ceph since Firefly by Achim LedermüllerNETWAYS
Ceph claims to be a perfect unified file system for almost every purpose. In theory this can be true but in reality there are of course limitations. Following our way from Firefly (2014) to Luminous (2018) I will show our use cases, the shortcomings and the pitfalls we fell into! This will show Ceph in use for typical cloud and container stacks, simple media archives for applications and the devilish high available shared file systems.
C* Summit 2013: Practice Makes Perfect: Extreme Cassandra Optimization by Alb...DataStax Academy
Ooyala has been using Apache Cassandra since version 0.4. Our data ingest volume has exploded since 0.4 and Cassandra has scaled along with us. Al will cover many topics from an operational perspective on how to manage, tune, and scale Cassandra in a production environment.
It's easy enough to test the correctness of the infracode we write with unit-tests and parsers, but testing it does what it's supposed to do in the enviornment itself is a little more challenging. In this talk, I'm going to talk about some of the tools and approaches to use to test your configuration automation tool of choice.
C* Summit EU 2013: Practice Makes Perfect: Extreme Cassandra OptimizationDataStax Academy
Speaker: Al Tobey, Open Source Mechanic at DataStax
Video: http://www.youtube.com/watch?v=AcPME94F13U&list=PLqcm6qE9lgKLoYaakl3YwIWP4hmGsHm5e&index=24
Ooyala has been using Apache Cassandra since version 0.4. Our data ingest volume has exploded since 0.4 and Cassandra has scaled along with us. Al will cover many topics from an operational perspective on how to manage, tune, and scale Cassandra in a production environment.
Similar to Joining a p2p Conversation - 2017-06 Meetup (20)
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Knowledge engineering: from people to machines and back
Joining a p2p Conversation - 2017-06 Meetup
1. JOINING A p2p CONVERSATION
JOAQUIN CASARES, THE LAST PICKLE
2. JOINING A p2p CONVERSATION
JOAQUIN CASARES
▸ The Last Pickle
▸ Consultant
▸ Previously:
▸ Umbel
▸ Software Engineer
▸ Riptano/DataStax
▸ Support Engineer
▸ Software Engineer-in-Test
▸ Demo Engineer
3. JOINING A p2p CONVERSATION
THE LAST PICKLE
▸ 50+ years combined experience with Apache Cassandra.
▸ We communicate ideas.
▸ We are committed to doing the right thing for both our team of experts and our clients.
▸ Our passion for sharing our knowledge is present in all that we do.
▸ Consider us a member of your team.
▸ Ultimately:
▸ We want you to be successful and have all the information to do so.
5. OVERVIEW
Overview
▸ p2p Networks.
▸ Cassandra fundamentals.
▸ How to add capacity.
▸ How to check on the status.
▸ Things you shouldn't forget, I think.
▸ How to forget.
41. 3 2
BOOTSTRAP
POST-BOOTSTRAP: CLEANUP
B D G J
3
2
1
4
M Q T W
5 3 2 1 3 2 4 4
5
P
5
A
1
B
C
D
A
Q
P
O
N
P
O
N
B
I REALLY DON'T NEED THIS
EXTRA PRESSURE P, O, & N.
SEE YOU, C!
M
43. BOOTSTRAP
WHEN TO USE THE BOOTSTRAP PROCESS
▸ Prerequisite: Is everything UN (Up/Normal)?
▸ nodetool status
44. BOOTSTRAP
WHEN TO USE THE BOOTSTRAP PROCESS
▸ Are you hitting a disk capacity issue?
▸ df -h
45. BOOTSTRAP
WHEN TO USE THE BOOTSTRAP PROCESS
▸ Are you hitting CPU capacity limits?
▸ top/htop
46. BOOTSTRAP
WHEN TO USE THE BOOTSTRAP PROCESS
▸ Does request latency have room for improvement?
▸ nodetool cfstats
▸ nodetool tablestats on newer versions of Cassandra.
47. BOOTSTRAP
WHEN TO USE THE BOOTSTRAP PROCESS
▸ Do you want to split up token hot spots?
▸ nodetool status $KEYSPACE
48. BOOTSTRAP
WHEN TO USE THE BOOTSTRAP PROCESS
▸ Is the prerequisite met and YES to any of the other questions? Then bootstrap!
49. BOOTSTRAP
PLAY IT SAFE
▸ UJ (Up/Joining) is an ephemeral state after ~2 minutes, but wait 5 minutes.
▸ UN (Up/Normal) is a persistent state.
▸ With Cassandra 2.2+, we have nodetool bootstrap resume, or simply restarting
the node.
▸ With Cassandra pre-2.2, we must clear:
▸ data_file_directories
▸ commitlog_directory
▸ saved_caches_directory
51. BOOTSTRAP
DON'TS
▸ Seed nodes cannot be bootstrapped.
▸ No need to include auto_bootstrap parameter in the cassandra.yaml.
▸ Do not join more than one node per rack concurrently.
▸ Do not start two bootstrap processes within 2 minutes of each other.
▸ Do not bootstrap a node when there are more than 0 nodes offline.
▸ The JVM option -Dcassandra.consistent.rangemovement=false can be used to override the default behavior.
▸ Will require a follow-up rolling repair.
▸ Do not bootstrap a node running a different version of Cassandra.
▸ Do not bootstrap new nodes into a mixed-version cluster.
▸ Don't forget about your racks! More on that later...
52. BOOTSTRAP
TIDBIT: CLEANUP
▸ Cleanup removes stale replicas off a node.
▸ Acts like a single-SSTable compaction.
▸ Does not need to be run manually, since any followup compactions will remove
stale data.
▸ Is useful when disk capacity is at it's limit.
53. KEEP FREE DISK SPACE UNDER 50% TO
ALLOW FOR NORMAL COMPACTIONS TO
COMPLETE SUCCESSFULLY.
The Last Pickle
BOOTSTRAP
57. FOR IMPORTANT DATA:
USE AN ODD-NUMBERED
REPLICATION_FACTOR > 1.
The Last Pickle
REPLACE_ADDRESS
58. FOR IMPORTANT DATA:
WRITE AT A CONSISTENCY LEVEL OF
LOCAL_QUORUM OR HIGHER.
The Last Pickle
REPLACE_ADDRESS
59. TO AVOID STALE DATA:
READ AT A CONSISTENCY LEVEL OF
LOCAL_QUORUM OR HIGHER.
The Last Pickle
REPLACE_ADDRESS
60. REPLACE_ADDRESS
TIDBITS: REPLACE_ADDRESS
▸ If using cl.ONE, you might have a bad time.
▸ But:
▸ Remember to respect max_hint_window_in_ms, which defaults to 3 hours
and starts as soon as the original node is knocked offline.
▸ If the hinted handoff window is missed, a rolling repair may be needed.
▸ Use -Dcassandra.replace_address_first_boot=<IP_ADDRESS> to
prevent possible issues if you forget to remove the flag.
61. REPLACE_ADDRESS
TIDBITS: REPLACE_ADDRESS - EXPANDED
▸ If not using a consistency level higher than ONE, stale data or data loss is possible and likely in the event of a node
failure.
▸ But:
▸ Hinted handoff may prevent stale data or data loss if a new node is in the UN (Up/Normal) state within the
max_hint_window_in_ms, which defaults to 3 hours and starts as soon as the original node is knocked offline.
▸ If the hinted handoff window is missed, and if running with a replication factor > 1, and other replicas
received the newer mutation, a rolling repair will make the new replica consistent.
▸ Read-repair may prevent stale data from being returned and repair stale partitions, but the
dclocal_read_repair_chance table schema parameter defaults to 10% of all requests.
▸ Use -Dcassandra.replace_address_first_boot=<IP_ADDRESS> to prevent possible issues if you forget to
remove the flag.
62. REPLACE_ADDRESS
WHEN TO USE THE REPLACE_ADDRESS PROCESS
▸ Prerequisite: Is the node unrecoverable/inaccessible?
▸ Are the current snapshots too stale?
▸ Most times snapshots are used for disaster scenarios.
▸ Is there enough time to replace the old node and run a repair before
gc_grace_seconds, which defaults to 10 days?
▸ If not, use nodetool removetoken.
▸ Is YES the response to all of the above questions? Then replace_address.
68. REBUILD
PROCESS: GETTING THINGS SORTED
▸ Prerequisite: Use the NetworkTopologyStrategy, instead of the SimpleStrategy.
▸ Prerequisite: Use a DCAwarePolicy with the Cassandra driver to restrict the contact points.
▸ Prerequisite: Use LOCAL_QUORUM and LOCAL_ONE consistency levels to restrict
requests to the specific data center.
▸ Do not define a replication strategy to the new new data center at first.
▸ Bootstrap all intended nodes into the new data center.
▸ Because there is no replication strategies that mention the new data center, this should
cause almost no streaming tasks.
69. REBUILD
PROCESS, PART 2: EXECUTING THE REBUILD
▸ Once all new nodes are in the new data center, continue.
▸ Modify the NTS settings to include the new data center.
▸ Run the rebuild process on as many concurrent nodes as latency metrics allow.
▸ To mitigate load on existing nodes, you may be able to use multiple data
center sources concurrently by using different data center parameters for the
nodetool rebuild command.
70. REBUILD
PROCESS, PART 3: USING THE NEW DATA CENTER
▸ Once all nodes have been completed the rebuild process, continue.
▸ If the intent was to remove a deprecated data center, update the
DCAwarePolicy for the Cassandra driver to point to the new data center and
restart the application. Then update NTS, and remove deprecated nodes.
▸ If the intent was to add a new data center, launch new application servers
within the same data center and modify the DCAwarePolicy to reference the
new data center.
71. REBUILD
WHEN TO USE THE REBUILD PROCESS
▸ Are you attempting to add or deprecate an entire data center?
▸ Migrate to new hardware?
▸ Moving to the cloud?
▸ Moving to a different cloud?
▸ Moving back from the cloud? :)
▸ Is YES the response to any of the above questions? Then rebuild.
72. THIS IS A WELL TRODDEN PATH.
DON'T WORRY.
BE HAPPY.
The Last Pickle
REBUILD
74. STATUS
CHECK PROGRESS - SHOW OUTPUT
▸ nodetool compactionstats
▸ Monitor pending compactions, which are a byproduct of:
▸ Streaming data.
▸ nodetool cleanup
▸ Use nodetool setcompactionthroughput to throttle disk load.
▸ nodetool netstats
▸ Monitor active streaming tasks.
▸ nodetool status
▸ Monitor node's joining and up status.
76. RACKS
PROPER BALANCE
▸ Balance is important when considering: disk load, token distribution, data center
load, and rack load.
▸ While other setups may be valid, Keep It Super Simple.
▸ Use 1 rack, or enough racks to equal the replication_strategy, for the data center.
▸ Ensure each rack always has an equal number of nodes.
▸ Each rack splits up the token range amongst themselves.
▸ Each data center will store its copies across racks, if available.
78. INTERNODE SECURITY
SSL ENCRYPTION
▸ Cassandra supports the following types of internode encryption:
▸ None.
▸ Inter-data center.
▸ Intra-data center, or inter-node.
▸ If data centers are separated by a public network, TLP recommends using inter-data
center encryption.
▸ If running with paranoid security settings, encryption can be used between each node,
regardless of topology settings.
80. EXPOSED JMX PORT
LOCKING DOWN YOUR JMX PORT
▸ Cassandra allows access to system metrics, system commands, and potentially
destructive commands via a JMX port.
▸ By default, Cassandra 2.1.4+ restricts access to the JMX port to localhost.
▸ If remote access to JMX is required, edit cassandra-env.sh to change access
or authentication settings.
▸ TLP still recommends proper firewall and security settings be used to restrict
access to Cassandra from only verified machines.
82. REMOVING NODES
DECOMMISSION
▸ nodetool decommission should be used for nodes that are still operational, but will no
longer be part of the rack.
▸ When decommissioning a node, the node's replica ranges are redistributed amongst the
surviving nodes in the rack.
▸ A "reverse bootstrap" occurs in which all replicas that the node is responsible for are
streamed to the new replica owners.
▸ Once all new replica owners hold all of the deprecated node's data, the node is removed
from the ring.
▸ After 72 hours, the node is removed from the gossip state.
83. REMOVING NODES
REMOVENODE
▸ nodetool removenode can be used when a node has died and will not be
replaced.
▸ The removenode command does not handle any streaming tasks, so follow up
repairs are required to ensure the cluster is in a consistent state.
▸ The removenode command simply removes a node from the gossip state,
forcing surviving nodes within the rack into being responsible for the
deprecated node's token ranges.
84. REMOVING NODES
ASSASSINATE
▸ Note: This is NOT a hammer.
▸ Sometimes gossip state can become wonky with echoes of previously removed nodes. In
these cases, and only in these cases, should nodetool assassinate be used.
▸ Much like nodetool removetoken, this command modifies the gossip state, but instead
of marking the node as being REMOVED, the entry is removed in its entirety.
▸ Sometimes a single command may not remove a stubborn gossip state. In these cases,
running nodetool assassinate across all nodes, in parallel, multiple times, may be
needed to remove any culprit gossip echoed states.
▸ Repeated note: This is NOT a hammer.
86. RECAP
p2p NETWORKS
▸ KaZaA: Not really p2p.
▸ Bittorrent: Decentralized p2p.
▸ Cassandra: Stateful p2p, via Gossip.
87. RECAP
Fundamentals
▸ Nodes own multiple token ranges, when using Vnodes.
▸ Seed nodes allow new nodes to enter the cluster.
▸ Seed nodes also help keep a "consistent" topological state.
88. RECAP
ADDING CAPACITY
▸ Bootstrap
▸ The normal process to adding a node.
▸ Follow up with nodetool cleanup.
▸ replace_address
▸ Useful when a node is completely lost.
▸ nodetool rebuild
▸ Used to add an entire data center.
90. RECAP
BE MINDFUL
▸ Ensure racks remain balanced upon topological changes.
▸ Ensure inter-node encryption is considered, especially when communicating
over an open network.
▸ Ensure JMX access is not accidentally exposed to public access.
91. RECAP
REMOVING NODES
▸ nodetool decommission
▸ Deprecate a live node.
▸ nodetool removetoken
▸ Remove a downed node and reassign token ranges.
▸ nodetool assassinate
▸ Remove a gossip entry entirely.
▸ Note: This is still NOT a hammer.
93. JOAQUIN@THELASTPICKLE.COM
Joaquin Casares, The Last Pickle
I'M BOB.
BON VOYAGE!
DIGIORNO
HASTA LA VISTA, BABY.
TTYL
TTYS
WELL THAT WAS FUN.
DON'T FORGET ABOUT ME!
WE'LL ALWAYS HAVE PARIS.
HERE'S LOOKING AT YOUR, KID.
I KNOW NOW WHY YOU CRY.
CYA