Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析

1,303 views

Published on

This is a short training for introduce Pi to use fluentd to collect data and use Google Cloud Logging and BigQuery as backend and then use Apps Script and Google Sheet as presentation layer.

Published in: Technology
  • Did you try ⇒ www.HelpWriting.net ⇐?. They know how to do an amazing essay, research papers or dissertations.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析

  1. 1. Linker Networks Corp. http://www.linkernetworks.com Use Raspberry Pi + Fluentd + GCP Cloud Logging, BigQuery as IoT Data Collect & Analysis Simon Su @ LinkerNetworks
  2. 2. var simon = {/** I am at GCPUG.TW **/}; simon.GDE = 'https://developers.google.com/experts/people/peihsin-su'; simon.nodejs = ‘http://opennodes.arecord.us'; simon.googleshare = 'http://gappsnews.blogspot.tw' simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw'; simon.blog = ‘http://peihsinsu.blogspot.com'; simon.slideshare = ‘http://slideshare.net/peihsinsu/'; simon.email = ‘simonsu.mail@gmail.com’; simon.say(‘Let’s go Cloud!');
  3. 3. Welcome to join us!
  4. 4. What we use... ● Docker ● Fluentd ● Google Cloud Logging ● Google BigQuery ● SpreadSheet
  5. 5. Docker ● Build, Ship and Run any App, Anywhere ● DevOps spirit ● Fast to scale
  6. 6. Fluentd ● Easy and fast ● Message delivery for at most once, at least once, exactly once
  7. 7. BigQuery ● TB Level data analysis ● Fast mining tool ● SQL like interface ● Multi-dataset join support ● Cheap & Pay by Use Google BigQuery - Big data with SQL like query feature, but fast...
  8. 8. Cloud Logging
  9. 9. Architecture ● BigQuery: Data Warehouse and support query and analysis ● Cloud Logging: Log gateway and easy to transfer to one or multi destination with guarantee ● Fluentd: Device layer gateway and support multi source for user to extend http(s) input tcp/udp input forward / secure forward other inputs ● Customize schema ● Co-relation ● Higher quota and performance ● Web console ● Export available for BQ, GCS, Pub/Sub ● Configure input source
  10. 10. Prepare Log Gateway using Cloud Logging
  11. 11. Still no GCP project?? (https://cloud.google.com/free-trial/)
  12. 12. curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh sudo bash install-logging-agent.sh service google-fluentd start Install Cloud Log Agent From: https://cloud.google.com/logging/docs/agent/installation
  13. 13. Configure # /etc/google-fluentd/google-fluentd.conf <source> type http port 9880 </source> <source> @type forward port 24224 </source> <filter *.**> @type stdout </filter> # Do not collect fluentd's own logs to avoid infinite loops. <match fluent.**> type null </match> # Configure all sources to output to Google Cloud Logging <match *.**> type google_cloud # Set the chunk limit conservatively to avoid exceeding the limit # of 10MB per write request. buffer_chunk_limit 2M flush_interval 5s # Never wait longer than 5 minutes between retries. max_retry_wait 300 # Disable the limit on the number of retries (retry forever). disable_retry_limit # Use multiple threads for processing. num_threads 8 </match> Set config to open fluentd forward and also screen output for debug..
  14. 14. Test run & watch log... # service google-fluentd start # tail -f /var/log/google-fluentd/google-fluentd.log 2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA: {"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"} 2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA: {"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"} 2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA: {"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"} ….(skip)
  15. 15. Prepare a Docker enabled Pi
  16. 16. Install Docker into Fluentd http://blog.hypriot.com/downloads/
  17. 17. Build the Fluentd Docker for Pi
  18. 18. Getting start with fluentd… (https://peihsinsu.gitbooks.io/fluentd-in-action/content/)
  19. 19. Build your Raspberry Pi Fluentd - Dockerfile # Dockerfile FROM hypriot/rpi-alpine-scratch MAINTAINER linkeriot RUN adduser -D -g '' -u 1000 docker ENV FLUENTD_VERSION 0.14 RUN apk --update add build-base ca-certificates ruby-dev && rm -rf /var/cache/apk/* && echo 'gem: --no-document' >> /etc/gemrc && gem install fluentd -v $FLUENTD_VERSION && apk del build-base WORKDIR /fluentd RUN mkdir log etc plugins config.d COPY fluent.conf /fluentd/etc/ ONBUILD COPY fluent.conf /fluentd/etc/ ONBUILD COPY plugins/* /fluentd/plugins/ ENV FLUENTD_OPT="" ENV FLUENTD_CONF="fluent.conf" EXPOSE 24224 EXPOSE 8888 VOLUME ["/fluentd/log"] CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
  20. 20. Build your Raspberry Pi Fluentd - fluentd.conf # fluentd.conf @include /fluentd/config.d/*.conf <label @mainstream> <filter **> @type record_transformer <record> linker_devid "#{ENV['TAG']}" </record> </filter> <match **> @type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 16 hard_timeout 60s <server> name loggateway host "#{ENV['FDN']}" port "#{ENV['FPORT']}" weight 60 </server> <secondary> @type file path /var/log/fluent/forward-failed </secondary> </match> </label>
  21. 21. Build and publish ● docker build -t your-image-name . ● docker push your-image-name
  22. 22. Run the Docker...
  23. 23. Run from your images docker run -d -p 24224:24224 --restart=always -e FDN=[remote-server-ip] -e FPORT=[remote-server-port] -e TAG=[your-tag] -v /var/log:/fluentd/log -v /data/cfg:/fluentd/config.d yout-image-name
  24. 24. Run using linker build images docker run -d -p 24224:24224 --restart=always -e FDN=[remote-server-ip] -e FPORT=[remote-server-port] -e TAG=[your-tag] -v /var/log:/fluentd/log -v /data/cfg:/fluentd/config.d linkeriot/iot-log-agent
  25. 25. Set log forward to BigQuery
  26. 26. Check log insert status...
  27. 27. Enable log forward to BigQuery...
  28. 28. BigQuery from SpreadSheet
  29. 29. Try your query….
  30. 30. Create Apps Script from Google SpreadSheet
  31. 31. Using my sample... Get source from gist: http://goo.gl/J90QD6
  32. 32. Test run the script
  33. 33. Setup the schedule...
  34. 34. Create the chart...
  35. 35. The End Q&A

×