Your SlideShare is downloading. ×
NoSQL and Big Data for DevOps
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

NoSQL and Big Data for DevOps

554
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
554
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. NoSQL and Big Data for DevOps Gustavo FernandesSunday, 3 February 13
  • 2. Agenda • DevOps? • Toolset • BigData • NoSQL • Demo • Q&ASunday, 3 February 13
  • 3. DevOps - Motivations • Silos development/ops • Slow release cycles • Lack of awareness from either sideSunday, 3 February 13
  • 4. Developers • Payed to add new features constantly • “Works in my laptop” • Usually IGNORE non-functional requirementsSunday, 3 February 13
  • 5. Operations • Keep it stable • Reliable • Monitoring • Distance from codeSunday, 3 February 13
  • 6. Devops - Enablers • Agile • Infrastructure as softwareSunday, 3 February 13
  • 7. What is Devops? • Development + Operations • Discipline/Philosophy/Methodology • Role - coder of non-functional requirements • Faster, more reliable, continuous releases to productionSunday, 3 February 13
  • 8. Devops - Principles • Automate everything: release, deployment, provision • Infrastructure as code - TDD, tags, branches, ... • Agile to opsSunday, 3 February 13
  • 9. Open Source Toolset • Configuration management tool • Puppet, chef • Application Lifecycle management tools • Build tools: Maven, Gradle, Buildr, SBT, Rake • Maven repository: Nexus, Artifactory • Provisioning tools • Vagrant, Boxgrinder • CI servers: jenkinsSunday, 3 February 13
  • 10. Puppet • Custom Declarative Language • Describe resources and states • Applies states to servers • Standalone/client-server/pub-sub • TestableSunday, 3 February 13
  • 11. Puppet Resources file { "/my/file": user { opuser: ensure => present, password => $1$9VC1vFFa$GHKWgtdODti8eKqkQ7Ruv. source => "/path/to/", } backup => main, mode => “0644” } cron { logrotate: command => "/usr/sbin/logrotate", user => root, hour => 2, minute => 0 } exec { "tar -xf /Volumes/nfs02/important.tar": cwd => "/var/tmp", creates => "/var/tmp/myfile", path => ["/usr/bin", "/usr/sbin"] }Sunday, 3 February 13
  • 12. Classes class mongodb($replicaset = ‘’, $disablenuma = ‘’) { $mongo_tgz = "mongodb-${arch}- ${version}.tgz"    $base_dir = "${base}"  group { "mongodb":         "ensure => present " } " user{ "mongodb":         "ensure => present,          gid => "mongodb",         "shell => "/sbin/nologin" " }    file { "$base_dir": " " ensure => "directory", " " owner => "mongodb", " " group => "mongodb", " " alias => "mongo-base" " } }Sunday, 3 February 13
  • 13. File Server file { "/etc/sudoers": mode => 440, owner => root, group => root, source => "puppet:///modules/name/sudoers" } file { "$installdir/conf/mongo.conf": mode => 0744, ensure => present, content => template(mongodb/mongodb.conf.erb), }Sunday, 3 February 13
  • 14. ERB Templates servers = [‘server1.domain’,‘server2.domain’,‘server3.domain’] file {/etc/foo.conf: ensure => file, content => template(foo/foo.conf.erb), } # foo.conf.erb # foo.conf.erb <% servers.each do |server| -%> server1.domain <%= server %> server2.domain <% end -%> server3.domainSunday, 3 February 13
  • 15. Modules Static Files Module manifest ERB TemplatesSunday, 3 February 13
  • 16. Site site.pp node "box1.domain" { include java include hadoop_master }Sunday, 3 February 13
  • 17. Manifest • Puppet entry point • Nodes node "box1.domain" { include java include hadoop_master class { myapp: version => "1.5-SNAPSHOT", maven_repo => "http://devserver:8081/nexus/content/repositories/snapshots/", } } node “box2.domain” { include java include hadoop_slave }Sunday, 3 February 13
  • 18. Facter Inventory • Companion ruby utility to puppet • Collect facts about one environment and expose as a map • Puppet uses it to decide what to deliver to a nodeSunday, 3 February 13
  • 19. Facter - Example $ facter architecture => x86_64 domain => domain fqdn => devserver.domain hardwaremodel => x86_64 hostname => devserver ipaddress_eth1 => 192.168.95.15 ipaddress_lo => 127.0.0.1 is_virtual => true kernel => Linux memorytotal => 491.11 MB network_eth0 => 10.0.2.0 network_eth1 => 192.168.95.0 operatingsystem => OpenSuSE operatingsysrelease => 12.2 physicalprocessorcount => 1 processor0 => Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz processorcount => 1 ... ...Sunday, 3 February 13
  • 20. Using facter values mapred-site.xml.erb ... <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value><%= $::processorcount %></value> </property> ...Sunday, 3 February 13
  • 21. Puppet Extensions • The language itself • New types • New functions • New resourcesSunday, 3 February 13
  • 22. The maven resource maven { "download-artifact": groupid => "com.gustavonalle", artifactid => "$artifact", version => "1.0-SNAPSHOT", repos => "http://devserver:8081/repo/snapshots/", directory => "/opt/myapp", classifier => “jar” require => File["basedir"], before => Exec["unzip"] }Sunday, 3 February 13
  • 23. Maven resource • Support for SNAPSHOTS • Support for Releases • No need to install Maven in the server • Support Authentication • Support for http and https • Support for puppet:// protocolSunday, 3 February 13
  • 24. Installing HadoopSunday, 3 February 13
  • 25. Hadoop - Processes • Name Node running on at least one node • Secondary Name Name node running elsewhere • Data Node process running on nodes who are part of HDFS • Job Tracker running in the cluster • Task Tracker running in each node that can execute map or reduce taskSunday, 3 February 13
  • 26. Hadoop HDFS • Name node knows all the slaves from <HADOOP_HOME>/conf/slaves slave1.domain.com slave2.domain.com ... • Slaves need to point at namenode in the file <HADOOP_HOME>/conf/core-site.xml <property> <name>fs.default.name</name> <value>hdfs://namenode:9000</value> </property>Sunday, 3 February 13
  • 27. Hadoop Map Reduce • Job Tracker knows all the slaves from <HADOOP_HOME>/conf/slaves slave1.domain.com slave2.domain.com ... • Slaves need to point at jobtracker in the file <HADOOP_HOME>/conf/mapredsite.xml <property> <name>mapred.job.tracker</name> <value>master:9001</value> </property>Sunday, 3 February 13
  • 28. Hadoop - SSH • SSH is required to do cluster-wide operations • ‘ssh localhost’ without asking password • su hadoop -c ‘ssh slave01’ without passwordSunday, 3 February 13
  • 29. How puppet can help • Facter calculates optimal values for memory, cpu, number of maps • Generate ssh keys on the fly • Obtain hostnames automatically • Hide most of the complexity and expose only the bare minimal • Install in parallel high number of slavesSunday, 3 February 13
  • 30. MongoDB - Replicaset Arbiter Primary (no data) SecondariesSunday, 3 February 13
  • 31. MongoDB - Creating cluster • Replicaset must be done with all servers running • Using cmd tools and a bit of javascriptSunday, 3 February 13
  • 32. How puppet can help • Ensure all servers are running and configured • Generate .js files and configuration files using templatesSunday, 3 February 13
  • 33. Demo github.com/gustavonalle/ puppet Puppet Master Jenkins NexusSunday, 3 February 13
  • 34. Demo ** Name Node Data Node Data Node Task Tracker * Job Tracker JVM Task Tracker M&R Job Mongo Secondary JVM box2.domain Mongo box1.domain Primary Mongo Arbiter * devserver.domain ** 2 Cpus * 1 CpuSunday, 3 February 13
  • 35. site.pp class mongo_replicaset { class { mongodb: replicaset => fosdem, primary => box1.domain, secondaries => [box2.domain], arbiter => devserver.domain } } class customApp { class { myapp: version => "1.0-SNAPSHOT", maven_repo => "http://devserver.domain:8081/repo/snapshots/", } }Sunday, 3 February 13
  • 36. site.pp (cont.) node "box1.domain" { include java class { hadoop: master => box1, slaves => [box1,box2] } include mongo_replicaset include customApp } node "box2.domain" { include java include mongo_replicaset class { hadoop: master => box1, } }Sunday, 3 February 13
  • 37. Wrapping up • Server side software is not getting any simpler • But infrastructure is now “software” • Devops is here to staySunday, 3 February 13
  • 38. Thank you github.com/gustavonalle/puppet gustavonalle@gmail.comSunday, 3 February 13

×