Wissbi osdc pdf


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Wissbi osdc pdf

  1. 1. WissbiA toolset for distributed event processinghttp://lunastorm.github.io/wissbi/
  2. 2. Batch Processinghttp://www.yilan-local-life-news.org.tw/joinus_6/eq-detail.php?idNo=20
  3. 3. Streaming Processing
  4. 4. Successful Stories• "Wissbi is an easy-to-use tool for distributedevent processing"-- Scott Wang, 奇群科技• "Wissbi provides great utilities for logging,debugging, and monitoring your distributedsystem"
-- 前趨勢科技,王姓工程師• "Wissbi lets you easily manage your dataworkflow in a pipe and filter style with intuitivecommands"
-- lunastorm, Open Source Developer
  5. 5. Wissbi 的前世今生• Wissbi is a highly available andscalable distributed messagerouting framework like ZeroMQthat took a different path fromtraditional MQ middlewares.• The toolkits supports an elegantand intuitive integration model,allowing writing event drivenapplications as easy as "$ tail -flog | grep error > error.log".• Scott Wang a.k.a. lunastorm, Sr.Engineer at Zillians• Trend Message Exchange (TME) isa highly available and scalabledistributed message routingframework that took a different pathfrom traditional MQ middlewares.• TMEs client toolkit, MIST, supportsan elegant and intuitive integrationmodel, allowing writing event drivenapplications as easy as "$ tail -f log |grep error > error.log".• Scott Wang a.k.a. lunastorm, Sr.Engineer at Trend Micro
  6. 6. 源起Origin
  7. 7. I need amessaging system
  8. 8. 虧損超過四千萬不排除解散
  9. 9. Deployment GuideSystem Requirement• A working ZooKeeper deployment, version newer than 3.3.2 is required• You have set your hostname and hostname resolution correctly on each machine. Running hostname -i should returnan IP other than• Create an user account TME on each machine, for example, execute useradd -m TMEStandalone DeploymentBy standalone it means that all of the TME components (including ZooKeeper) will be deployed on the same host. Mostly thisis for development purposes. Because all default configurations will work in standalone mode, you only have to install thepackages and bring up the daemons to test and develop.CentOSYou will need the following 3rd party packages which are not in the official repository:1. jdk (get RPM from http://www.oracle.com/technetwork/java/javase/downloads/index.html)2. monit (http://pkgs.org/search/?keyword=monit)3. nodejs (for portal, http://pkgs.org/search/?keyword=nodejs)4. ruby (for portal, https://github.com/lunastorm/ruby19_centos/downloads, have to be at least 1.9.2, you can build fromhttps://github.com/imeyer/ruby-1.9.2-rpm)5. ruby-bundler (for portal, https://github.com/lunastorm/ruby19_centos/downloads)Then you can follow the following steps to install:1. install Suns JDK first2. download the dependency RPMs mentioned above3. download TME RPM binaries and place them in the same folder with the dependencies4. yum --nogpgcheck install *.rpmUbuntu1. Grab all the deb files you would like to install2. sudo dpkg -i tme-*.deb3. sudo apt-get update4. sudo apt-get -f installTME web portal only supports Ruby 1.9.2+, and Ubuntu 10.04 only ships Ruby 1.9.1You have to follow this step to use RVM to install Ruby 1.9.2:1. aptitude install build-essential libssl-dev libreadline5 libreadline5-dev zlib1g zlib1g-dev6. bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)2. source /etc/profile.d/rvm.sh3. rvm install 1.9.2 ; rvm default 1.9.24. Edit /opt/trend/tme/conf/portal-web/portal-web-conf.sh , add "source /etc/profile.d/rvm.sh"The web portal requires a JavaScript runtime installed. For example, you can install Node.js on the machine.On All DistributionFinally, execute /opt/trend/tme/bin/create_zookeeper_nodes.sh /tme2 to initialize the essential information onZooKeeper before you start the components. • Distributed DeploymentYou can choose to deploy different components on different machines of different hardware specs. Typically, the brokers arerequired to be deployed on more powerful machines than others. They will handle a large amount of client connections anddeliver messages so they consume more memory and use more CPU. On the other hand, the clients need not use too muchcomputing power, but it depends on the applications needs. The administration packages can be deployed on multiplemachines for redundancy.1. First, you will probably have a distributed ZooKeeper set up, for example, zk1.mydomain:2181,zk2.mydomain:2181,zk3.mydomain:21812. Follow the same way described in the standalone guide above to install the packages on the machines of yourchoice.3. Initialize the essential information on ZooKeeper: Execute /opt/trend/tme/bin/create_zookeeper_nodes.shzk1.mydomain:2181,zk2.mydomain:2181,zk3.mydomain:2181 /tme_root_prefix The first argument is the ZooKeeperquorums, and the second argument is a prefix path on ZooKeeper chosen by you. By separating the prefix path onService Start / StopYou can choose one of the following ways to start or stop the components.service and chkconfig1. sudo service tme-broker {start / stop / restart}2. sudo service tme-mistd {start / stop / restart}3. sudo service tme-graph-editor {start / stop / restart}4. sudo service tme-portal-collector {start / stop / restart}5. sudo service tme-portal-web {start / stop / restart}If you wish to start the services upon boot, the you can use chkconfig to turn on the services.monitIf you have installed and enabled monit, then you can use it to ensure the services are running.Under the configuration folders of the components, there are monit watchdog scripts that can be modified to fulfill your need,for example, send a notification when a daemon stops working.You can use the helper scripts to start / stop the components:1. sudo /opt/trend/tme/bin/{install|remove}_tme-broker.sh2. sudo /opt/trend/tme/bin/{install|remove}_tme-mistd.sh3. sudo /opt/trend/tme/bin/{install|remove}_tme-graph-editor.sh4. sudo /opt/trend/tme/bin/{install|remove}_tme-portal-collector.sh5. sudo /opt/trend/tme/bin/{install|remove}_tme-portal-web.shVerificationMessagingAfter MIST daemon is started and configured correctly, execute mist-session --list to show session information:$ mist-session -l0 sessions0 connectionsYou should get the response like above. If MIST daemon is not started correctly, you may receive the following error response:$ mist-session -lError connecting to MIST daemon!If MIST daemon is running correctly, then you can send your first Hello World message. Execute the script to send a messageto a queue named test:$ session_id=`mist-session` && echo Hello World! | mist-encode --wrap test --line | mist-sink $session_id --attach ; mist-session --destroy $session_iddestroyed 1453444792Then execute the script to receive one message from the queue named test:$ session_id=`mist-session` && mist-source $session_id --mount test && mist-source $session_id --attach --limit 1 | mist-decode --line ; mist-session --destroy $session_idexchange queue:test mountedHello World!destroyed 1453444796Congratulations! You can now transmit the messages.PortalOpen a browser to access http://**portal.host**:**portal.port** to see if it shows correctly.Graph EditorI don’t always read
the deployment guideWhen I screw upsomething
I call the developers
  10. 10. Rethink Your Problem
  11. 11. XXDo I reallyneed a broker?Do I really needsynchronization?
  12. 12. What Do I Need?• A message publisher• A message subscriber• A directory service for publishers to knowwhere the subscribers are
  13. 13. Name Foo BarAddressDirectoryServiceMessage
  14. 14. Name Foo BarAddress192.168.0.1:5566DirectoryServiceMessage
PublisherGet all subscribersSend Message
  15. 15. Directory Services• DNS• First attempt• LDAP• $&@?!&@!?• ...
  16. 16. What ifI use directory fordirectory service?
  17. 17. Using Filesystem forDirectory Service• Store metadata on the filesystem• Follows the philosophy "Everything is a file"• Use standard Unix commands to manage it• mkdir, touch, rm, ln, ...• Just like /proc and /sys
  18. 18. Using Filesystem forDirectory Service• You get authorization for free• chown, chmod, ...• Good for testing• Launch any number of cluster just withdifferent metadata directories
  19. 19. 分散式是吧?
  20. 20. Name FooAddress192.168.0.1:5566DirectoryServiceMessage
  21. 21. 連你阿嬤都會用的管理方式如果你阿嬤會用Unix...
  22. 22. Related Work
  23. 23. Plumber (Program)• The plumber, in the Plan 9 from Bell Labs and Inferno operating systems, is amechanism for reliable uni- or multicast inter-process communication offormatted textual messages. It uses the Plan 9 network file protocol, 9p, ratherthan a special-purpose IPC mechanism.• Any number of clients may listen on a named port (a file) for messages. Portsand port routing are defined by plumbing rules. These rules are dynamic. Eachlistening program receives a copy of matching messages. For example, if thedata /sys/lib/plumb/basic is plumbed with the standard rules, it is sent to theedit port. The port will write a copy of the message to each listener. In this case,all running editors will interpret this message as a file name, and open the file.• The plumber is the 9P file server that provides this service. Clients may uselibplumb to format messages. Since the messages are 9P, they are networktransparent.http://en.wikipedia.org/wiki/Plumber_(program)
  24. 24. Design Philosophy
  25. 25. Minimum Dependency$ ldd wissbi-publinux-vdso.so.1
/lib64/ld-linux-x86-64.so.2The only dependency is a compilerwhich supports C++11!
  26. 26. Easy to Test偷懶QQ
  27. 27. Basic Commands• wissbi-sub [message source]• wissbi-pub [message destination]
  28. 28. Live Demohttp://ascii.io/a/2913
  29. 29. wissbi-subWrite your filter with any language you likewissbi-pubPipePipe
  30. 30. wissbi-subFilterwissbi-pubPipePipewissbi-subFilterwissbi-pubPipePipewissbi-subFilterwissbi-pubPipePipe
  31. 31. You Will Have MultipleData PipelinessubFilterpubsubFilterpubsubFilterpubsubFilterpubsubFilterpubsubFilterpub
  32. 32. Daemonize Your Filters!• 用 config 寫程式• Daemon start / stop / restart / status• start / stop upon boot / shutdown• Watchdogs
  33. 33. <?xml version="1.0"?><!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd"><!-- =============================================================== --><!-- Configure the Jetty Server --><!-- --><!-- Documentation of this file format can be found at: --><!-- http://docs.codehaus.org/display/JETTY/jetty.xml --><!-- --><!-- =============================================================== --><Configure id="Server" class="org.mortbay.jetty.Server"><!-- =========================================================== --><!-- Server Thread Pool --><!-- =========================================================== --><Set name="ThreadPool"><!-- Default bounded blocking threadpool --><New class="org.mortbay.thread.BoundedThreadPool"><Set name="minThreads">10<Set name="maxThreads">50<Set name="lowThreads">25</New><!-- New queued blocking threadpool : better scalability<New class="org.mortbay.thread.QueuedThreadPool"><Set name="minThreads">10<Set name="maxThreads">25<Set name="lowThreads">5<Set name="SpawnOrShrinkAt">2</New>用 Config寫程式之Java 篇
  34. 34. # How many filter instances will be run in parallelWISSBI_FILTER_COUNT="1"# How to run the filterWISSBI_FILTER_CMD="sed --unbuffered -e "s/^/[ / ; s/$/ ]/""# If you run multiple instances in parallel, you can use the instance id $iin the command# WISSBI_FILTER_CMD="sed --unbuffered -e "s/^/$i: [ / ; s/$/ ]/""# The message sources name, leave it empty if the filter is a messagegeneratorWISSBI_FILTER_SOURCE="test.in"# The message sinks name, leave it empty if the filter is a messageterminalWISSBI_FILTER_SINK="test.out"WISSBI_FILTER_LOG_PREFIX="/tmp/filter-example"WISSBI_FILTER_PID_PREFIX="/tmp/filter-example"# If WISSBI_DEBUG_DUMP is set, message recording will be enabled, and 50messages# before the filter is terminated is dumped to the specified file.# If WISSBI_DEBUG_DUMP is set to empty, then a random dump filename will beused#WISSBI_DEBUG_DUMP=""wissbi filterexample
  35. 35. The easiest way to builda distributed pipeline
  36. 36. Live DemoA Calculator Service http://ascii.io/a/2914
  37. 37. The Wissbi Ecosystem
  38. 38. Log Collector#!/bin/shWISSBI_FILTER_COUNT="1"if [ -e "./wissbi_log_collector.sh" ]thenWISSBI_FILTER_CMD="./wissbi_log_collector.sh"elseWISSBI_FILTER_CMD="/usr/bin/wissbi_log_collector.sh"fiWISSBI_FILTER_SOURCE=""WISSBI_FILTER_SINK="wissbi.log"WISSBI_FILTER_LOG_PREFIX="/tmp/wsblogcollectord"WISSBI_FILTER_PID_PREFIX="/tmp/wsblogcollectord". /usr/bin/wissbi_filter_template.sh
  39. 39. wissbi.log/var/log/syslog/var/log/postfixtail -F | collector |wissbi-pub/var/log/apache/*tail -F | collector |wissbi-pubsolr /splunk /hadoop /...
  40. 40. Metric Collector• Ganglia Integration• Write message count into rrd files• Also uses wissbi as the transporting tool• Demo if we have time...
  41. 41. http://aws.amazon.comVery useful metricto scale yourserver farm
  42. 42. Current Limitation• Message size < 4k• One listening port is occupied per consumer
  43. 43. Platform Support• Ubuntu 12.04 with gcc 4.6.3• BSD branch with clang++ on the way...• OSX (等上面那個)• Windows• Wat?
  44. 44. Conclusion• 窮人的鮑魚 -- 九孔• 窮人的燕窩 -- 白木耳• 窮人的原子彈 -- 生化武器• 窮人的 Message Bus -- Wissbi• Minimize operating effort and cost• Maximize workflow flexibility
  45. 45. TME Wissbi代言 無發哥、伍佰勞動英雄
語言 丁ava 節能減碳C++操作 複雜 簡單版號 2.5 謙虛的0.1勝蘋狗日報彙整勝勝勝
  46. 46. Thank You