• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
agri inventory - nouka data collector / yaoya data convertor
 

agri inventory - nouka data collector / yaoya data convertor

on

  • 1,125 views

overview of agri inventory. composed by nouka data collector, yaoya data convertor

overview of agri inventory. composed by nouka data collector, yaoya data convertor

Statistics

Views

Total Views
1,125
Views on SlideShare
830
Embed Views
295

Actions

Likes
1
Downloads
0
Comments
0

2 Embeds 295

http://okochang.hatenablog.jp 293
http://webcache.googleusercontent.com 2

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    agri inventory - nouka data collector / yaoya data convertor agri inventory - nouka data collector / yaoya data convertor Presentation Transcript

    • agri inventory
    • agri inventorynouka data collectornaya data store nayayaoya data convertor push crawl output yaoya nouka
    • yaoya output example(1) $ bin/yaoya --group system_A --name command_uname{ureturn_code: u0, ucommand_line: uuname -a, uexecute_at:u1339942622, ugroup_name: usystem_A, uvisible: uTrue, uhost_name:ubb-mba.local, utime: datetime.datetime(2012, 6, 17, 14, 17, 3), uoutput:uDarwin bb-mba.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 716:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386n, u_id:ObjectId(4fdde6e0c286630ca0000009), ucommand_name:ucommand_uname}{ureturn_code: u0, ucommand_line: uuname -a, uexecute_at:u1339943547, ugroup_name: usystem_A, uvisible: uTrue, uhost_name:ubbtest-c6, utime: datetime.datetime(2012, 6, 17, 14, 32, 28), uoutput:uLinux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST2012 x86_64 x86_64 x86_64 GNU/Linuxn, u_id:ObjectId(4fddea7dc286630ca000003c), ucommand_name:ucommand_uname}
    • yaoya output example(2) $ bin/csv --group system_A --name command_uname"bb-mba.local","Darwin bb-mba.local 10.8.0 Darwin KernelVersion 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386""bbtest-c6","Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64GNU/Linux"
    • default commands command_ifconfig = ifconfig -a  command_dmidecode = /usr/sbin/dmidecode command_hostname = hostname  command_resolv = cat /etc/resolv.conf command_uname = uname -a  command_passwd = cat /etc/passwd command_df = df -akl  command_group = cat /etc/group command_iptables = /sbin/iptables-save  command_shadow = cat /etc/shadow command_rpm = rpm -qa  command_proc_cpuinfo = cat /proc/cpuinfo command_chkconfig = /sbin/chkconfig --list  command_proc_meminfo = cat /proc/meminfo command_ip_addr = /sbin/ip addr show  command_proc_mounts = cat /proc/mounts command_ip_route = /sbin/ip route show To add command, just only define in nouka.conf!
    • components
    • nouka data collector pure python 2.4 run with python standard library POST result to naya with HTTP
    • naya data store fluentd / td-agent fluent-plugin-http-enhanced  https://github.com/parolkar/fluent-plugin-http-enhanced fluent-plugin-mongo mongodb
    • data format{ *** are same value in same execution ‘group_name’ : ‘<group_name’s value in config file>‘, *** host_name : <env HOSTNAME>, *** command_name : <command name in config file. named command_*>, command_line : <command line(value) in config file>, output : <output of command line>, return_code : <return code of command line>, ‘visible’ : ‘True’, *** execute_at : <execute date as unixtime>’, ***}
    • getting started
    • project repository nouka  https://bitbucket.org/netmarkjp/nouka yaoya  https://bitbucket.org/netmarkjp/yaoya
    • setup(1/2) td-agent  cat <<EOT >/etc/yum.repos.d/td.repo [treasuredata] name=TreasureData baseurl=http://packages.treasure- data.com/redhat/$basearch gpgcheck=0 EOT  yum -y install td-agent  /usr/lib64/fluent/ruby/bin/gem install bson_ext  /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-http- enhanced
    • setup(2/2) mongod  cat <<EOT >/etc/yum.repos.d/10gen.repo [10gen] name=10gen Repository baseurl=http://downloads- distro.mongodb.org/repo/redhat/os/x86_64 gpgcheck=0 EOT  yum -y install mongo-10gen.x86_64 mongo-10gen-server.x86_64 mercurial  yum -y install python-devel python-setuptools gcc  pip install mercurial
    • enjoy development!