Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

agri inventory - nouka data collector / yaoya data convertor

1,512 views

Published on

overview of agri inventory. composed by nouka data collector, yaoya data convertor

Published in: Technology
  • Be the first to comment

agri inventory - nouka data collector / yaoya data convertor

  1. 1. agri inventory
  2. 2. agri inventorynouka data collectornaya data store nayayaoya data convertor push crawl output yaoya nouka
  3. 3. yaoya output example(1) $ bin/yaoya --group system_A --name command_uname{ureturn_code: u0, ucommand_line: uuname -a, uexecute_at:u1339942622, ugroup_name: usystem_A, uvisible: uTrue, uhost_name:ubb-mba.local, utime: datetime.datetime(2012, 6, 17, 14, 17, 3), uoutput:uDarwin bb-mba.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 716:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386n, u_id:ObjectId(4fdde6e0c286630ca0000009), ucommand_name:ucommand_uname}{ureturn_code: u0, ucommand_line: uuname -a, uexecute_at:u1339943547, ugroup_name: usystem_A, uvisible: uTrue, uhost_name:ubbtest-c6, utime: datetime.datetime(2012, 6, 17, 14, 32, 28), uoutput:uLinux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST2012 x86_64 x86_64 x86_64 GNU/Linuxn, u_id:ObjectId(4fddea7dc286630ca000003c), ucommand_name:ucommand_uname}
  4. 4. yaoya output example(2) $ bin/csv --group system_A --name command_uname"bb-mba.local","Darwin bb-mba.local 10.8.0 Darwin KernelVersion 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386""bbtest-c6","Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64GNU/Linux"
  5. 5. default commands command_ifconfig = ifconfig -a  command_dmidecode = /usr/sbin/dmidecode command_hostname = hostname  command_resolv = cat /etc/resolv.conf command_uname = uname -a  command_passwd = cat /etc/passwd command_df = df -akl  command_group = cat /etc/group command_iptables = /sbin/iptables-save  command_shadow = cat /etc/shadow command_rpm = rpm -qa  command_proc_cpuinfo = cat /proc/cpuinfo command_chkconfig = /sbin/chkconfig --list  command_proc_meminfo = cat /proc/meminfo command_ip_addr = /sbin/ip addr show  command_proc_mounts = cat /proc/mounts command_ip_route = /sbin/ip route show To add command, just only define in nouka.conf!
  6. 6. components
  7. 7. nouka data collector pure python 2.4 run with python standard library POST result to naya with HTTP
  8. 8. naya data store fluentd / td-agent fluent-plugin-http-enhanced  https://github.com/parolkar/fluent-plugin-http-enhanced fluent-plugin-mongo mongodb
  9. 9. data format{ *** are same value in same execution ‘group_name’ : ‘<group_name’s value in config file>‘, *** host_name : <env HOSTNAME>, *** command_name : <command name in config file. named command_*>, command_line : <command line(value) in config file>, output : <output of command line>, return_code : <return code of command line>, ‘visible’ : ‘True’, *** execute_at : <execute date as unixtime>’, ***}
  10. 10. getting started
  11. 11. project repository nouka  https://bitbucket.org/netmarkjp/nouka yaoya  https://bitbucket.org/netmarkjp/yaoya
  12. 12. setup(1/2) td-agent  cat <<EOT >/etc/yum.repos.d/td.repo [treasuredata] name=TreasureData baseurl=http://packages.treasure- data.com/redhat/$basearch gpgcheck=0 EOT  yum -y install td-agent  /usr/lib64/fluent/ruby/bin/gem install bson_ext  /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-http- enhanced
  13. 13. setup(2/2) mongod  cat <<EOT >/etc/yum.repos.d/10gen.repo [10gen] name=10gen Repository baseurl=http://downloads- distro.mongodb.org/repo/redhat/os/x86_64 gpgcheck=0 EOT  yum -y install mongo-10gen.x86_64 mongo-10gen-server.x86_64 mercurial  yum -y install python-devel python-setuptools gcc  pip install mercurial
  14. 14. enjoy development!

×