• Like
  • Save
NoSQL and Big Data for DevOps
Upcoming SlideShare
Loading in...5
×
 

NoSQL and Big Data for DevOps

on

  • 588 views

 

Statistics

Views

Total Views
588
Views on SlideShare
578
Embed Views
10

Actions

Likes
0
Downloads
5
Comments
0

2 Embeds 10

http://www.linkedin.com 8
https://www.linkedin.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    NoSQL and Big Data for DevOps NoSQL and Big Data for DevOps Presentation Transcript

    • NoSQL and Big Data for DevOps Gustavo FernandesSunday, 3 February 13
    • Agenda • DevOps? • Toolset • BigData • NoSQL • Demo • Q&ASunday, 3 February 13
    • DevOps - Motivations • Silos development/ops • Slow release cycles • Lack of awareness from either sideSunday, 3 February 13
    • Developers • Payed to add new features constantly • “Works in my laptop” • Usually IGNORE non-functional requirementsSunday, 3 February 13
    • Operations • Keep it stable • Reliable • Monitoring • Distance from codeSunday, 3 February 13
    • Devops - Enablers • Agile • Infrastructure as softwareSunday, 3 February 13
    • What is Devops? • Development + Operations • Discipline/Philosophy/Methodology • Role - coder of non-functional requirements • Faster, more reliable, continuous releases to productionSunday, 3 February 13
    • Devops - Principles • Automate everything: release, deployment, provision • Infrastructure as code - TDD, tags, branches, ... • Agile to opsSunday, 3 February 13
    • Open Source Toolset • Configuration management tool • Puppet, chef • Application Lifecycle management tools • Build tools: Maven, Gradle, Buildr, SBT, Rake • Maven repository: Nexus, Artifactory • Provisioning tools • Vagrant, Boxgrinder • CI servers: jenkinsSunday, 3 February 13
    • Puppet • Custom Declarative Language • Describe resources and states • Applies states to servers • Standalone/client-server/pub-sub • TestableSunday, 3 February 13
    • Puppet Resources file { "/my/file": user { opuser: ensure => present, password => $1$9VC1vFFa$GHKWgtdODti8eKqkQ7Ruv. source => "/path/to/", } backup => main, mode => “0644” } cron { logrotate: command => "/usr/sbin/logrotate", user => root, hour => 2, minute => 0 } exec { "tar -xf /Volumes/nfs02/important.tar": cwd => "/var/tmp", creates => "/var/tmp/myfile", path => ["/usr/bin", "/usr/sbin"] }Sunday, 3 February 13
    • Classes class mongodb($replicaset = ‘’, $disablenuma = ‘’) { $mongo_tgz = "mongodb-${arch}- ${version}.tgz"    $base_dir = "${base}"  group { "mongodb":         "ensure => present " } " user{ "mongodb":         "ensure => present,          gid => "mongodb",         "shell => "/sbin/nologin" " }    file { "$base_dir": " " ensure => "directory", " " owner => "mongodb", " " group => "mongodb", " " alias => "mongo-base" " } }Sunday, 3 February 13
    • File Server file { "/etc/sudoers": mode => 440, owner => root, group => root, source => "puppet:///modules/name/sudoers" } file { "$installdir/conf/mongo.conf": mode => 0744, ensure => present, content => template(mongodb/mongodb.conf.erb), }Sunday, 3 February 13
    • ERB Templates servers = [‘server1.domain’,‘server2.domain’,‘server3.domain’] file {/etc/foo.conf: ensure => file, content => template(foo/foo.conf.erb), } # foo.conf.erb # foo.conf.erb <% servers.each do |server| -%> server1.domain <%= server %> server2.domain <% end -%> server3.domainSunday, 3 February 13
    • Modules Static Files Module manifest ERB TemplatesSunday, 3 February 13
    • Site site.pp node "box1.domain" { include java include hadoop_master }Sunday, 3 February 13
    • Manifest • Puppet entry point • Nodes node "box1.domain" { include java include hadoop_master class { myapp: version => "1.5-SNAPSHOT", maven_repo => "http://devserver:8081/nexus/content/repositories/snapshots/", } } node “box2.domain” { include java include hadoop_slave }Sunday, 3 February 13
    • Facter Inventory • Companion ruby utility to puppet • Collect facts about one environment and expose as a map • Puppet uses it to decide what to deliver to a nodeSunday, 3 February 13
    • Facter - Example $ facter architecture => x86_64 domain => domain fqdn => devserver.domain hardwaremodel => x86_64 hostname => devserver ipaddress_eth1 => 192.168.95.15 ipaddress_lo => 127.0.0.1 is_virtual => true kernel => Linux memorytotal => 491.11 MB network_eth0 => 10.0.2.0 network_eth1 => 192.168.95.0 operatingsystem => OpenSuSE operatingsysrelease => 12.2 physicalprocessorcount => 1 processor0 => Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz processorcount => 1 ... ...Sunday, 3 February 13
    • Using facter values mapred-site.xml.erb ... <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value><%= $::processorcount %></value> </property> ...Sunday, 3 February 13
    • Puppet Extensions • The language itself • New types • New functions • New resourcesSunday, 3 February 13
    • The maven resource maven { "download-artifact": groupid => "com.gustavonalle", artifactid => "$artifact", version => "1.0-SNAPSHOT", repos => "http://devserver:8081/repo/snapshots/", directory => "/opt/myapp", classifier => “jar” require => File["basedir"], before => Exec["unzip"] }Sunday, 3 February 13
    • Maven resource • Support for SNAPSHOTS • Support for Releases • No need to install Maven in the server • Support Authentication • Support for http and https • Support for puppet:// protocolSunday, 3 February 13
    • Installing HadoopSunday, 3 February 13
    • Hadoop - Processes • Name Node running on at least one node • Secondary Name Name node running elsewhere • Data Node process running on nodes who are part of HDFS • Job Tracker running in the cluster • Task Tracker running in each node that can execute map or reduce taskSunday, 3 February 13
    • Hadoop HDFS • Name node knows all the slaves from <HADOOP_HOME>/conf/slaves slave1.domain.com slave2.domain.com ... • Slaves need to point at namenode in the file <HADOOP_HOME>/conf/core-site.xml <property> <name>fs.default.name</name> <value>hdfs://namenode:9000</value> </property>Sunday, 3 February 13
    • Hadoop Map Reduce • Job Tracker knows all the slaves from <HADOOP_HOME>/conf/slaves slave1.domain.com slave2.domain.com ... • Slaves need to point at jobtracker in the file <HADOOP_HOME>/conf/mapredsite.xml <property> <name>mapred.job.tracker</name> <value>master:9001</value> </property>Sunday, 3 February 13
    • Hadoop - SSH • SSH is required to do cluster-wide operations • ‘ssh localhost’ without asking password • su hadoop -c ‘ssh slave01’ without passwordSunday, 3 February 13
    • How puppet can help • Facter calculates optimal values for memory, cpu, number of maps • Generate ssh keys on the fly • Obtain hostnames automatically • Hide most of the complexity and expose only the bare minimal • Install in parallel high number of slavesSunday, 3 February 13
    • MongoDB - Replicaset Arbiter Primary (no data) SecondariesSunday, 3 February 13
    • MongoDB - Creating cluster • Replicaset must be done with all servers running • Using cmd tools and a bit of javascriptSunday, 3 February 13
    • How puppet can help • Ensure all servers are running and configured • Generate .js files and configuration files using templatesSunday, 3 February 13
    • Demo github.com/gustavonalle/ puppet Puppet Master Jenkins NexusSunday, 3 February 13
    • Demo ** Name Node Data Node Data Node Task Tracker * Job Tracker JVM Task Tracker M&R Job Mongo Secondary JVM box2.domain Mongo box1.domain Primary Mongo Arbiter * devserver.domain ** 2 Cpus * 1 CpuSunday, 3 February 13
    • site.pp class mongo_replicaset { class { mongodb: replicaset => fosdem, primary => box1.domain, secondaries => [box2.domain], arbiter => devserver.domain } } class customApp { class { myapp: version => "1.0-SNAPSHOT", maven_repo => "http://devserver.domain:8081/repo/snapshots/", } }Sunday, 3 February 13
    • site.pp (cont.) node "box1.domain" { include java class { hadoop: master => box1, slaves => [box1,box2] } include mongo_replicaset include customApp } node "box2.domain" { include java include mongo_replicaset class { hadoop: master => box1, } }Sunday, 3 February 13
    • Wrapping up • Server side software is not getting any simpler • But infrastructure is now “software” • Devops is here to staySunday, 3 February 13
    • Thank you github.com/gustavonalle/puppet gustavonalle@gmail.comSunday, 3 February 13