agri inventory - nouka data collector / yaoya data convertor

  • 929 views
Uploaded on

overview of agri inventory. composed by nouka data collector, yaoya data convertor

overview of agri inventory. composed by nouka data collector, yaoya data convertor

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
929
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
0
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. agri inventory
  • 2. agri inventorynouka data collectornaya data store nayayaoya data convertor push crawl output yaoya nouka
  • 3. yaoya output example(1) $ bin/yaoya --group system_A --name command_uname{ureturn_code: u0, ucommand_line: uuname -a, uexecute_at:u1339942622, ugroup_name: usystem_A, uvisible: uTrue, uhost_name:ubb-mba.local, utime: datetime.datetime(2012, 6, 17, 14, 17, 3), uoutput:uDarwin bb-mba.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 716:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386n, u_id:ObjectId(4fdde6e0c286630ca0000009), ucommand_name:ucommand_uname}{ureturn_code: u0, ucommand_line: uuname -a, uexecute_at:u1339943547, ugroup_name: usystem_A, uvisible: uTrue, uhost_name:ubbtest-c6, utime: datetime.datetime(2012, 6, 17, 14, 32, 28), uoutput:uLinux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST2012 x86_64 x86_64 x86_64 GNU/Linuxn, u_id:ObjectId(4fddea7dc286630ca000003c), ucommand_name:ucommand_uname}
  • 4. yaoya output example(2) $ bin/csv --group system_A --name command_uname"bb-mba.local","Darwin bb-mba.local 10.8.0 Darwin KernelVersion 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386""bbtest-c6","Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64GNU/Linux"
  • 5. default commands command_ifconfig = ifconfig -a  command_dmidecode = /usr/sbin/dmidecode command_hostname = hostname  command_resolv = cat /etc/resolv.conf command_uname = uname -a  command_passwd = cat /etc/passwd command_df = df -akl  command_group = cat /etc/group command_iptables = /sbin/iptables-save  command_shadow = cat /etc/shadow command_rpm = rpm -qa  command_proc_cpuinfo = cat /proc/cpuinfo command_chkconfig = /sbin/chkconfig --list  command_proc_meminfo = cat /proc/meminfo command_ip_addr = /sbin/ip addr show  command_proc_mounts = cat /proc/mounts command_ip_route = /sbin/ip route show To add command, just only define in nouka.conf!
  • 6. components
  • 7. nouka data collector pure python 2.4 run with python standard library POST result to naya with HTTP
  • 8. naya data store fluentd / td-agent fluent-plugin-http-enhanced  https://github.com/parolkar/fluent-plugin-http-enhanced fluent-plugin-mongo mongodb
  • 9. data format{ *** are same value in same execution ‘group_name’ : ‘<group_name’s value in config file>‘, *** host_name : <env HOSTNAME>, *** command_name : <command name in config file. named command_*>, command_line : <command line(value) in config file>, output : <output of command line>, return_code : <return code of command line>, ‘visible’ : ‘True’, *** execute_at : <execute date as unixtime>’, ***}
  • 10. getting started
  • 11. project repository nouka  https://bitbucket.org/netmarkjp/nouka yaoya  https://bitbucket.org/netmarkjp/yaoya
  • 12. setup(1/2) td-agent  cat <<EOT >/etc/yum.repos.d/td.repo [treasuredata] name=TreasureData baseurl=http://packages.treasure- data.com/redhat/$basearch gpgcheck=0 EOT  yum -y install td-agent  /usr/lib64/fluent/ruby/bin/gem install bson_ext  /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-http- enhanced
  • 13. setup(2/2) mongod  cat <<EOT >/etc/yum.repos.d/10gen.repo [10gen] name=10gen Repository baseurl=http://downloads- distro.mongodb.org/repo/redhat/os/x86_64 gpgcheck=0 EOT  yum -y install mongo-10gen.x86_64 mongo-10gen-server.x86_64 mercurial  yum -y install python-devel python-setuptools gcc  pip install mercurial
  • 14. enjoy development!