Your SlideShare is downloading. ×
NoSQL and Big Data                           for DevOps                             Gustavo FernandesSunday, 3 February 13
Agenda                    • DevOps?                    • Toolset                    • BigData                    • NoSQL  ...
DevOps - Motivations                    • Silos development/ops                    • Slow release cycles                  ...
Developers                    • Payed to add new features constantly                    • “Works in my laptop”            ...
Operations                    • Keep it stable                    • Reliable                    • Monitoring              ...
Devops - Enablers                    • Agile                    • Infrastructure as softwareSunday, 3 February 13
What is Devops?                    • Development + Operations                    • Discipline/Philosophy/Methodology      ...
Devops - Principles                    • Automate everything: release, deployment,                        provision       ...
Open Source Toolset                    •   Configuration management tool                        •   Puppet, chef           ...
Puppet                   •    Custom Declarative Language                   •    Describe resources and states            ...
Puppet Resources                                                                    file { "/my/file":        user { opuse...
Classes               class mongodb($replicaset = ‘’, $disablenuma = ‘’) {                  $mongo_tgz = "mongodb-${arch}-...
File Server        file { "/etc/sudoers":            mode => 440,            owner => root,            group => root,     ...
ERB Templates     servers = [‘server1.domain’,‘server2.domain’,‘server3.domain’]     file {/etc/foo.conf:           ensure...
Modules                                  Static Files                                  Module manifest                    ...
Site                                              site.pp                               node "box1.domain" {              ...
Manifest               • Puppet entry point               • Nodes            node "box1.domain" {              include jav...
Facter Inventory                    • Companion ruby utility to puppet                    • Collect facts about one enviro...
Facter - Example                        $ facter                        architecture        => x86_64                     ...
Using facter values        mapred-site.xml.erb           ...           <property>               <name>mapred.tasktracker.m...
Puppet Extensions                    • The language itself                    • New types                    • New functio...
The maven resource      maven { "download-artifact":         groupid => "com.gustavonalle",         artifactid => "$artifa...
Maven resource                    • Support for SNAPSHOTS                    • Support for Releases                    • N...
Installing HadoopSunday, 3 February 13
Hadoop - Processes                    • Name Node running on at least one node                    • Secondary Name Name no...
Hadoop HDFS                    •   Name node knows all the slaves from                        <HADOOP_HOME>/conf/slaves   ...
Hadoop Map Reduce                    •   Job Tracker knows all the slaves from                        <HADOOP_HOME>/conf/s...
Hadoop - SSH                 • SSH is required to do cluster-wide operations                 • ‘ssh localhost’ without ask...
How puppet can help                    • Facter calculates optimal values for                        memory, cpu, number o...
MongoDB - Replicaset                                  Arbiter                        Primary                              ...
MongoDB - Creating                             cluster                    • Replicaset must be done with all servers      ...
How puppet can help                    • Ensure all servers are running and                        configured              ...
Demo                        github.com/gustavonalle/                                 puppet                               ...
Demo                                          **          Name Node     Data Node                Data Node          Task T...
site.pp (cont.)              node "box1.domain" {                include java                class { hadoop:              ...
Wrapping up                    • Server side software is not getting any                        simpler                   ...
Thank you                        github.com/gustavonalle/puppet                           gustavonalle@gmail.comSunday, 3 ...
NoSQL and Big Data for DevOps
Upcoming SlideShare
Loading in...5
×

NoSQL and Big Data for DevOps

654

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
654
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "NoSQL and Big Data for DevOps"

  1. 1. NoSQL and Big Data for DevOps Gustavo FernandesSunday, 3 February 13
  2. 2. Agenda • DevOps? • Toolset • BigData • NoSQL • Demo • Q&ASunday, 3 February 13
  3. 3. DevOps - Motivations • Silos development/ops • Slow release cycles • Lack of awareness from either sideSunday, 3 February 13
  4. 4. Developers • Payed to add new features constantly • “Works in my laptop” • Usually IGNORE non-functional requirementsSunday, 3 February 13
  5. 5. Operations • Keep it stable • Reliable • Monitoring • Distance from codeSunday, 3 February 13
  6. 6. Devops - Enablers • Agile • Infrastructure as softwareSunday, 3 February 13
  7. 7. What is Devops? • Development + Operations • Discipline/Philosophy/Methodology • Role - coder of non-functional requirements • Faster, more reliable, continuous releases to productionSunday, 3 February 13
  8. 8. Devops - Principles • Automate everything: release, deployment, provision • Infrastructure as code - TDD, tags, branches, ... • Agile to opsSunday, 3 February 13
  9. 9. Open Source Toolset • Configuration management tool • Puppet, chef • Application Lifecycle management tools • Build tools: Maven, Gradle, Buildr, SBT, Rake • Maven repository: Nexus, Artifactory • Provisioning tools • Vagrant, Boxgrinder • CI servers: jenkinsSunday, 3 February 13
  10. 10. Puppet • Custom Declarative Language • Describe resources and states • Applies states to servers • Standalone/client-server/pub-sub • TestableSunday, 3 February 13
  11. 11. Puppet Resources file { "/my/file": user { opuser: ensure => present, password => $1$9VC1vFFa$GHKWgtdODti8eKqkQ7Ruv. source => "/path/to/", } backup => main, mode => “0644” } cron { logrotate: command => "/usr/sbin/logrotate", user => root, hour => 2, minute => 0 } exec { "tar -xf /Volumes/nfs02/important.tar": cwd => "/var/tmp", creates => "/var/tmp/myfile", path => ["/usr/bin", "/usr/sbin"] }Sunday, 3 February 13
  12. 12. Classes class mongodb($replicaset = ‘’, $disablenuma = ‘’) { $mongo_tgz = "mongodb-${arch}- ${version}.tgz"    $base_dir = "${base}"  group { "mongodb":         "ensure => present " } " user{ "mongodb":         "ensure => present,          gid => "mongodb",         "shell => "/sbin/nologin" " }    file { "$base_dir": " " ensure => "directory", " " owner => "mongodb", " " group => "mongodb", " " alias => "mongo-base" " } }Sunday, 3 February 13
  13. 13. File Server file { "/etc/sudoers": mode => 440, owner => root, group => root, source => "puppet:///modules/name/sudoers" } file { "$installdir/conf/mongo.conf": mode => 0744, ensure => present, content => template(mongodb/mongodb.conf.erb), }Sunday, 3 February 13
  14. 14. ERB Templates servers = [‘server1.domain’,‘server2.domain’,‘server3.domain’] file {/etc/foo.conf: ensure => file, content => template(foo/foo.conf.erb), } # foo.conf.erb # foo.conf.erb <% servers.each do |server| -%> server1.domain <%= server %> server2.domain <% end -%> server3.domainSunday, 3 February 13
  15. 15. Modules Static Files Module manifest ERB TemplatesSunday, 3 February 13
  16. 16. Site site.pp node "box1.domain" { include java include hadoop_master }Sunday, 3 February 13
  17. 17. Manifest • Puppet entry point • Nodes node "box1.domain" { include java include hadoop_master class { myapp: version => "1.5-SNAPSHOT", maven_repo => "http://devserver:8081/nexus/content/repositories/snapshots/", } } node “box2.domain” { include java include hadoop_slave }Sunday, 3 February 13
  18. 18. Facter Inventory • Companion ruby utility to puppet • Collect facts about one environment and expose as a map • Puppet uses it to decide what to deliver to a nodeSunday, 3 February 13
  19. 19. Facter - Example $ facter architecture => x86_64 domain => domain fqdn => devserver.domain hardwaremodel => x86_64 hostname => devserver ipaddress_eth1 => 192.168.95.15 ipaddress_lo => 127.0.0.1 is_virtual => true kernel => Linux memorytotal => 491.11 MB network_eth0 => 10.0.2.0 network_eth1 => 192.168.95.0 operatingsystem => OpenSuSE operatingsysrelease => 12.2 physicalprocessorcount => 1 processor0 => Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz processorcount => 1 ... ...Sunday, 3 February 13
  20. 20. Using facter values mapred-site.xml.erb ... <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value><%= $::processorcount %></value> </property> ...Sunday, 3 February 13
  21. 21. Puppet Extensions • The language itself • New types • New functions • New resourcesSunday, 3 February 13
  22. 22. The maven resource maven { "download-artifact": groupid => "com.gustavonalle", artifactid => "$artifact", version => "1.0-SNAPSHOT", repos => "http://devserver:8081/repo/snapshots/", directory => "/opt/myapp", classifier => “jar” require => File["basedir"], before => Exec["unzip"] }Sunday, 3 February 13
  23. 23. Maven resource • Support for SNAPSHOTS • Support for Releases • No need to install Maven in the server • Support Authentication • Support for http and https • Support for puppet:// protocolSunday, 3 February 13
  24. 24. Installing HadoopSunday, 3 February 13
  25. 25. Hadoop - Processes • Name Node running on at least one node • Secondary Name Name node running elsewhere • Data Node process running on nodes who are part of HDFS • Job Tracker running in the cluster • Task Tracker running in each node that can execute map or reduce taskSunday, 3 February 13
  26. 26. Hadoop HDFS • Name node knows all the slaves from <HADOOP_HOME>/conf/slaves slave1.domain.com slave2.domain.com ... • Slaves need to point at namenode in the file <HADOOP_HOME>/conf/core-site.xml <property> <name>fs.default.name</name> <value>hdfs://namenode:9000</value> </property>Sunday, 3 February 13
  27. 27. Hadoop Map Reduce • Job Tracker knows all the slaves from <HADOOP_HOME>/conf/slaves slave1.domain.com slave2.domain.com ... • Slaves need to point at jobtracker in the file <HADOOP_HOME>/conf/mapredsite.xml <property> <name>mapred.job.tracker</name> <value>master:9001</value> </property>Sunday, 3 February 13
  28. 28. Hadoop - SSH • SSH is required to do cluster-wide operations • ‘ssh localhost’ without asking password • su hadoop -c ‘ssh slave01’ without passwordSunday, 3 February 13
  29. 29. How puppet can help • Facter calculates optimal values for memory, cpu, number of maps • Generate ssh keys on the fly • Obtain hostnames automatically • Hide most of the complexity and expose only the bare minimal • Install in parallel high number of slavesSunday, 3 February 13
  30. 30. MongoDB - Replicaset Arbiter Primary (no data) SecondariesSunday, 3 February 13
  31. 31. MongoDB - Creating cluster • Replicaset must be done with all servers running • Using cmd tools and a bit of javascriptSunday, 3 February 13
  32. 32. How puppet can help • Ensure all servers are running and configured • Generate .js files and configuration files using templatesSunday, 3 February 13
  33. 33. Demo github.com/gustavonalle/ puppet Puppet Master Jenkins NexusSunday, 3 February 13
  34. 34. Demo ** Name Node Data Node Data Node Task Tracker * Job Tracker JVM Task Tracker M&R Job Mongo Secondary JVM box2.domain Mongo box1.domain Primary Mongo Arbiter * devserver.domain ** 2 Cpus * 1 CpuSunday, 3 February 13
  35. 35. site.pp class mongo_replicaset { class { mongodb: replicaset => fosdem, primary => box1.domain, secondaries => [box2.domain], arbiter => devserver.domain } } class customApp { class { myapp: version => "1.0-SNAPSHOT", maven_repo => "http://devserver.domain:8081/repo/snapshots/", } }Sunday, 3 February 13
  36. 36. site.pp (cont.) node "box1.domain" { include java class { hadoop: master => box1, slaves => [box1,box2] } include mongo_replicaset include customApp } node "box2.domain" { include java include mongo_replicaset class { hadoop: master => box1, } }Sunday, 3 February 13
  37. 37. Wrapping up • Server side software is not getting any simpler • But infrastructure is now “software” • Devops is here to staySunday, 3 February 13
  38. 38. Thank you github.com/gustavonalle/puppet gustavonalle@gmail.comSunday, 3 February 13

×