SlideShare a Scribd company logo
1 of 60
Download to read offline
Consistency or Bust
                      Breaking a Riak Cluster




Sunday, July 31, 11
Jeff Kirkell
      email: jeff.kirkell@gmail.com
      twitter: @jeffkirkell




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
NoSQL Primer




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Riak Sales Pitch
      Dynamo inspired
      AP of CAP
      key / value store
      scales consistently
      fault tolerance
      true high availability
      built-in replication
      lot’s of clients
      simple to implement
      low $$$ to use
      multiple license options


      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Riak Deep Dive




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Cluster Defined
      cluster
      # nodes
      64 / # nodes
      vnodes




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Important Bits
      gossiping
      hintful handoff
      n # copies of data
      r # nodes that must respond for read
      w # nodes that must respond for write
      dw # replicas written on write for success
      rw # replicas deleted for success




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Tools of the Trade
      terminal
      curl
      rebar
      rekon




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
The Workshop
                                   setup instructions for cluster
                            https://github.com/jkirkell/riak_workshop




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Start Your Engines
     $ ./dev/dev1/bin/riak start
     $ ./dev/dev2/bin/riak start
     $ ./dev/dev3/bin/riak start
     $ ./dev/dev4/bin/riak start
     $ ./dev/dev5/bin/riak start
     $ ps aux | grep erl




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Come Together
     $ ./dev/dev2/bin/riak-admin join dev1@127.0.0.1
     $ ./dev/dev3/bin/riak-admin join dev1@127.0.0.1
     $ ./dev/dev4/bin/riak-admin join dev1@127.0.0.1
     $ ./dev/dev5/bin/riak-admin join dev1@127.0.0.1
     $ ./dev/dev3/bin/riak-admin status




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Default Settings




                                           quorum = floor(5 / 2 + 1) = 3

      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Add Data
     $ curl -v -d 'inserted data' -H "Content-Type: text/plain"
       http://127.0.0.1:8091/riak/default




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Confirm Data Exists
     $ curl -i http://127.0.0.1:8091/riak/default?keys=true




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Find Your Data
     $ find ./dev/dev*/data/bitcask/*/*.data




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Stop Primary Node
     $ dev[n]/bin/riak stop

     $ curl -i http://127.0.0.1:8091/riak/default/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Stop Another Primary Node
     $ dev[n]/bin/riak stop

     $ curl -i http://127.0.0.1:8091/riak/default/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Query Again
     $ curl -i http://127.0.0.1:8091/riak/default/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Check What’s Happening
     $ find ./dev/dev*/data/bitcask/*/*.data




     $ ./dev/dev[n]/bin/riak-admin transfers




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Update Data
     $ curl -v -d 'updated data' -H "Content-Type: text/plain"
       http://127.0.0.1:8091/riak/defaults/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Start Node
     $ dev[n]/bin/riak start

     $ ./dev/dev[n]/bin/riak-admin transfers




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Review Logs
     $ less ./dev2/log/*.1




     $ less ./dev5/log/*.1




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Start Other Node
     $ dev[n]/bin/riak start

     $ curl -i http://127.0.0.1:8091/riak/default/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Summary
      consistently fail 1st read with 2 node down
      2 node outage max.
      hintful-handoff takes time
      default settings good for general use




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Clean Your Cluster
     $        dev1/bin/riak stop
     $        dev2/bin/riak stop
     $        dev3/bin/riak stop
     $        dev4/bin/riak stop
     $        dev5/bin/riak stop
     $        cd .. && rm -rf ./dev/*
     $        make devrel
     $        dev1/bin/riak start
     $        dev2/bin/riak start
     $        dev3/bin/riak start
     $        dev4/bin/riak start
     $        dev5/bin/riak start

      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Low Value Data




                                                     n = 2, r = 1, w = 1

      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Set Bucket Properties
     $ curl -v -X PUT -H "Content-Type: application/json" -d
       '{"props":{"n_val":2,"r":1,"w":1}}' http://127.0.0.1:8091/
       riak/low_value




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Add Data
     $ curl -v -d 'inserted data' -H "Content-Type: text/plain"
       http://127.0.0.1:8091/riak/low_value




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Confirm Data Exists
     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Find Your Data
     $ find ./dev/dev*/data/bitcask/*/*.data




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Stop Primary Node
     $ dev[n]/bin/riak stop

     $ curl -i http://127.0.0.1:8091/riak/default/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Find Your Data Again
     $ find ./dev/dev*/data/bitcask/*/*.data




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Stop Another Primary Node
     $ dev[n]/bin/riak stop

     $ curl -i http://127.0.0.1:8091/riak/default/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Query Again
     $ curl -i http://127.0.0.1:8091/riak/default/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Check What’s Happening
     $ find ./dev/dev*/data/bitcask/*/*.data




     $ ./dev/dev[n]/bin/riak-admin transfers




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Add More Data
     $ curl -v -d 'new data' -H "Content-Type: text/plain" http://
       127.0.0.1:8091/riak/low_value




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Confirm Data Exists
     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Start Down Nodes
     $ dev[n]/bin/riak start

     $ dev[n]/bin/riak start

     $ ./dev/dev1/bin/riak-admin transfers




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Confirm Data Exists
     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Summary
      hintful-handoff does not happen *
      1 node outage max for reads
      4 node outage max for writes
      great for fire and forget data like logs


                                 * Look at the logs for the nodes you brought down. It will
                                   show data transfer from hand-off even though the available
                                   nodes do not serve the read requests. This may or may not
                                   happen randomly due to Eventual Consistency. One other
                                   note is that R + W = N cannot guarantee consistency.




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Clean Your Cluster
     $        dev1/bin/riak stop
     $        dev2/bin/riak stop
     $        dev3/bin/riak stop
     $        dev4/bin/riak stop
     $        dev5/bin/riak stop
     $        cd .. && rm -rf ./dev/*
     $        make devrel
     $        dev1/bin/riak start
     $        dev2/bin/riak start
     $        dev3/bin/riak start
     $        dev4/bin/riak start
     $        dev5/bin/riak start

      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Mission Critical Data




                                              n = 5, r = 1, w = 5, dw = 5

      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Set Bucket Properties




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Set Bucket Properties
     $ curl -v -X PUT -H "Content-Type: application/json" -d
       '{"props":{"n_val":5,"r":1,"w":5,"dw":5}}' http://
       127.0.0.1:8091/riak/high_value




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Add Data
     $ curl -v -d 'inserted data' -H "Content-Type: text/plain"
       http://127.0.0.1:8091/riak/high_value




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Confirm Data Exists
     $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Find Your Data
     $ find ./dev/dev*/data/bitcask/*/*.data




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Stop Any 2 Nodes
     $ dev[n]/bin/riak stop

     $ dev[n]/bin/riak stop

     $ curl -i http://127.0.0.1:8091/riak/high_value/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Check What’s Happening
     $ find ./dev/dev*/data/bitcask/*/*.data




     $ ./dev/dev[n]/bin/riak-admin transfers




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Stop Another Node
     $ dev[n]/bin/riak stop

     $ curl -i http://127.0.0.1:8091/riak/high_value/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Check What’s Happening
     $ find ./dev/dev*/data/bitcask/*/*.data




     $ ./dev/dev[n]/bin/riak-admin transfers




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Update Data
     $ curl -v -d 'updated data' -H "Content-Type: text/plain"
       http://127.0.0.1:8091/riak/high_value/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Query Data
     $ curl -i http://127.0.0.1:8091/riak/high_value/[key]




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Start Down Nodes
     $ dev[n]/bin/riak start

     $ dev[n]/bin/riak start

     $ dev[n]/bin/riak start

     $ ./dev/dev1/bin/riak-admin transfers




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Confirm Data Exists
     $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Summary
      data will duplicate on available nodes to ensure 5 copies
      4 node outage max for reads
      4 node outage max for writes
      slower due to all data writes
      great for high availability data




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Total Destruction




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Total Destruction




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Summary
      easy to get up and going
      scales as you need it
      flexible configuration for consistency sake
      eventual consistency doesn’t always do what you expect
      just scratched the surface




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11
Great Reading
      http://www.basho.com
      http://bit.ly/eG08u0
      http://bit.ly/eFfcjb
      http://bit.ly/ejCClJ
      http://slidesha.re/g3hQfg
      http://bit.ly/n6hWMK




      Consistency or Bust: Breaking a Riak Cluster
Sunday, July 31, 11

More Related Content

Similar to Breaking a riak cluster

Perl Intro 7 Subroutines
Perl Intro 7 SubroutinesPerl Intro 7 Subroutines
Perl Intro 7 SubroutinesShaun Griffith
 
2019 brkdwn spvrk
2019 brkdwn spvrk2019 brkdwn spvrk
2019 brkdwn spvrkBRKDWN
 
How we setup Rsync-powered Incremental Backups
How we setup Rsync-powered Incremental BackupsHow we setup Rsync-powered Incremental Backups
How we setup Rsync-powered Incremental Backupsnicholaspaun
 
Always On, Multi-Site Design Considerations
Always On, Multi-Site Design ConsiderationsAlways On, Multi-Site Design Considerations
Always On, Multi-Site Design ConsiderationsJohn Martin
 
What's new in Scala 2.13?
What's new in Scala 2.13?What's new in Scala 2.13?
What's new in Scala 2.13?Hermann Hueck
 
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Running High Performance and Fault Tolerant Elasticsearch Clusters on DockerRunning High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Running High Performance and Fault Tolerant Elasticsearch Clusters on DockerSematext Group, Inc.
 
Perl Intro 9 Command Line Arguments
Perl Intro 9 Command Line ArgumentsPerl Intro 9 Command Line Arguments
Perl Intro 9 Command Line ArgumentsShaun Griffith
 
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Running High Performance & Fault-tolerant Elasticsearch Clusters on DockerRunning High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Running High Performance & Fault-tolerant Elasticsearch Clusters on DockerSematext Group, Inc.
 
Tackling Big Data with Hadoop
Tackling Big Data with HadoopTackling Big Data with Hadoop
Tackling Big Data with Hadooppoorlytrainedape
 
Distributed Search in Riak - Integrating Search in a NoSQL Database: Presente...
Distributed Search in Riak - Integrating Search in a NoSQL Database: Presente...Distributed Search in Riak - Integrating Search in a NoSQL Database: Presente...
Distributed Search in Riak - Integrating Search in a NoSQL Database: Presente...Lucidworks
 
Riak a successful failure
Riak   a successful failureRiak   a successful failure
Riak a successful failureGiltTech
 
Rugged Driven Development with Gauntlt
Rugged Driven Development with GauntltRugged Driven Development with Gauntlt
Rugged Driven Development with GauntltJames Wickett
 

Similar to Breaking a riak cluster (17)

Perl Intro 7 Subroutines
Perl Intro 7 SubroutinesPerl Intro 7 Subroutines
Perl Intro 7 Subroutines
 
RAC.docx
RAC.docxRAC.docx
RAC.docx
 
2019 brkdwn spvrk
2019 brkdwn spvrk2019 brkdwn spvrk
2019 brkdwn spvrk
 
How we setup Rsync-powered Incremental Backups
How we setup Rsync-powered Incremental BackupsHow we setup Rsync-powered Incremental Backups
How we setup Rsync-powered Incremental Backups
 
Always On, Multi-Site Design Considerations
Always On, Multi-Site Design ConsiderationsAlways On, Multi-Site Design Considerations
Always On, Multi-Site Design Considerations
 
What's new in Scala 2.13?
What's new in Scala 2.13?What's new in Scala 2.13?
What's new in Scala 2.13?
 
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Running High Performance and Fault Tolerant Elasticsearch Clusters on DockerRunning High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
 
Perl Intro 9 Command Line Arguments
Perl Intro 9 Command Line ArgumentsPerl Intro 9 Command Line Arguments
Perl Intro 9 Command Line Arguments
 
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Running High Performance & Fault-tolerant Elasticsearch Clusters on DockerRunning High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
 
Mission impossible
Mission impossibleMission impossible
Mission impossible
 
Tackling Big Data with Hadoop
Tackling Big Data with HadoopTackling Big Data with Hadoop
Tackling Big Data with Hadoop
 
Zfs intro v2
Zfs intro v2Zfs intro v2
Zfs intro v2
 
Riak Search 2: Yokozuna
Riak Search 2: YokozunaRiak Search 2: Yokozuna
Riak Search 2: Yokozuna
 
Distributed Search in Riak - Integrating Search in a NoSQL Database: Presente...
Distributed Search in Riak - Integrating Search in a NoSQL Database: Presente...Distributed Search in Riak - Integrating Search in a NoSQL Database: Presente...
Distributed Search in Riak - Integrating Search in a NoSQL Database: Presente...
 
Unit Testing Lots of Perl
Unit Testing Lots of PerlUnit Testing Lots of Perl
Unit Testing Lots of Perl
 
Riak a successful failure
Riak   a successful failureRiak   a successful failure
Riak a successful failure
 
Rugged Driven Development with Gauntlt
Rugged Driven Development with GauntltRugged Driven Development with Gauntlt
Rugged Driven Development with Gauntlt
 

Recently uploaded

VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 

Recently uploaded (20)

VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 

Breaking a riak cluster

  • 1. Consistency or Bust Breaking a Riak Cluster Sunday, July 31, 11
  • 2. Jeff Kirkell email: jeff.kirkell@gmail.com twitter: @jeffkirkell Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 3. NoSQL Primer Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 4. Riak Sales Pitch  Dynamo inspired  AP of CAP  key / value store  scales consistently  fault tolerance  true high availability  built-in replication  lot’s of clients  simple to implement  low $$$ to use  multiple license options Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 5. Riak Deep Dive Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 6. Cluster Defined  cluster  # nodes  64 / # nodes  vnodes Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 7. Important Bits  gossiping  hintful handoff  n # copies of data  r # nodes that must respond for read  w # nodes that must respond for write  dw # replicas written on write for success  rw # replicas deleted for success Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 8. Tools of the Trade  terminal  curl  rebar  rekon Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 9. The Workshop setup instructions for cluster https://github.com/jkirkell/riak_workshop Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 10. Start Your Engines $ ./dev/dev1/bin/riak start $ ./dev/dev2/bin/riak start $ ./dev/dev3/bin/riak start $ ./dev/dev4/bin/riak start $ ./dev/dev5/bin/riak start $ ps aux | grep erl Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 11. Come Together $ ./dev/dev2/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev3/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev4/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev5/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev3/bin/riak-admin status Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 12. Default Settings quorum = floor(5 / 2 + 1) = 3 Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 13. Add Data $ curl -v -d 'inserted data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/default Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 14. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/default?keys=true Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 15. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 16. Stop Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 17. Stop Another Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 18. Query Again $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 19. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 20. Update Data $ curl -v -d 'updated data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/defaults/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 21. Start Node $ dev[n]/bin/riak start $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 22. Review Logs $ less ./dev2/log/*.1 $ less ./dev5/log/*.1 Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 23. Start Other Node $ dev[n]/bin/riak start $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 24. Summary  consistently fail 1st read with 2 node down  2 node outage max.  hintful-handoff takes time  default settings good for general use Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 25. Clean Your Cluster $ dev1/bin/riak stop $ dev2/bin/riak stop $ dev3/bin/riak stop $ dev4/bin/riak stop $ dev5/bin/riak stop $ cd .. && rm -rf ./dev/* $ make devrel $ dev1/bin/riak start $ dev2/bin/riak start $ dev3/bin/riak start $ dev4/bin/riak start $ dev5/bin/riak start Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 26. Low Value Data n = 2, r = 1, w = 1 Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 27. Set Bucket Properties $ curl -v -X PUT -H "Content-Type: application/json" -d '{"props":{"n_val":2,"r":1,"w":1}}' http://127.0.0.1:8091/ riak/low_value Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 28. Add Data $ curl -v -d 'inserted data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/low_value Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 29. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 30. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 31. Stop Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 32. Find Your Data Again $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 33. Stop Another Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 34. Query Again $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 35. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 36. Add More Data $ curl -v -d 'new data' -H "Content-Type: text/plain" http:// 127.0.0.1:8091/riak/low_value Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 37. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 38. Start Down Nodes $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ ./dev/dev1/bin/riak-admin transfers Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 39. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 40. Summary  hintful-handoff does not happen *  1 node outage max for reads  4 node outage max for writes  great for fire and forget data like logs * Look at the logs for the nodes you brought down. It will show data transfer from hand-off even though the available nodes do not serve the read requests. This may or may not happen randomly due to Eventual Consistency. One other note is that R + W = N cannot guarantee consistency. Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 41. Clean Your Cluster $ dev1/bin/riak stop $ dev2/bin/riak stop $ dev3/bin/riak stop $ dev4/bin/riak stop $ dev5/bin/riak stop $ cd .. && rm -rf ./dev/* $ make devrel $ dev1/bin/riak start $ dev2/bin/riak start $ dev3/bin/riak start $ dev4/bin/riak start $ dev5/bin/riak start Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 42. Mission Critical Data n = 5, r = 1, w = 5, dw = 5 Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 43. Set Bucket Properties Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 44. Set Bucket Properties $ curl -v -X PUT -H "Content-Type: application/json" -d '{"props":{"n_val":5,"r":1,"w":5,"dw":5}}' http:// 127.0.0.1:8091/riak/high_value Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 45. Add Data $ curl -v -d 'inserted data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 46. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 47. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 48. Stop Any 2 Nodes $ dev[n]/bin/riak stop $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 49. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 50. Stop Another Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 51. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 52. Update Data $ curl -v -d 'updated data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 53. Query Data $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 54. Start Down Nodes $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ ./dev/dev1/bin/riak-admin transfers Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 55. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 56. Summary  data will duplicate on available nodes to ensure 5 copies  4 node outage max for reads  4 node outage max for writes  slower due to all data writes  great for high availability data Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 57. Total Destruction Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 58. Total Destruction Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 59. Summary  easy to get up and going  scales as you need it  flexible configuration for consistency sake  eventual consistency doesn’t always do what you expect  just scratched the surface Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11
  • 60. Great Reading  http://www.basho.com  http://bit.ly/eG08u0  http://bit.ly/eFfcjb  http://bit.ly/ejCClJ  http://slidesha.re/g3hQfg  http://bit.ly/n6hWMK Consistency or Bust: Breaking a Riak Cluster Sunday, July 31, 11