This document discusses big data challenges and trends in data collection, storage, analysis, and stream processing. It introduces tools like Fluentd for data collection and filtering, Hadoop and OpenTSDB for storage, Hadoop Streaming and Hive for analysis, and custom dashboards or services like Tableau for visualization. It also covers stream computing with STORM, describing streams, spouts, bolts, and topologies, and provides an example of visualizing bandwidth usage with RedStorm on JRuby. The document is presented by an engineering team lead who founded BigData.SG and contributes to fluentd, pfeed, and other projects.

1. Ruby for the soul of
BigData Nerds

2. Who Am I?
●
Engineering Team Lead
Analytics & Data Platforms @ Viki.com
●
Founder of http://BigData.SG
●
Contributor to fluentd, pfeed, cartographer, watir

3. BigData & Its Challenges
"big data" is when the size of the data itself becomes part of the problem
- Mike Loukides
●
Twitter produces over 230 million tweets per day
●
Wal-Mart is logging one million transactions per hour
●
Facebook creates over 30 billion pieces of content
ranging from web links, news, blogs, photo

4. Everyone has a big data problem

5. Evolving Trends
Batch Processing
Hadoop , HPCC, Google BigQuery
Stream Processing
STORM (Twitter) & S4 (Yahoo)

6. Common Engineering Challenges
●
Data Collection
●
Filtering / Segmentation
●
Data Storage
●
Analysis
●
Visualization
●
Prediction / Extrapolation

7. Data Collection + Filtering /
Segmentation
http://fluentd.org/

8. Data Collection + Filtering /
Segmentation
You send events as:
Http://domain:8080/namespace?key1=value1&key2=value2
Fluent forwards the data as:
<timestamp> <namespace> {key1:value1,key2:value2}
http://fluentd.org/

9. Screencast:
http://www.bigdata.sg/videos/fluentd/

10. Storage
Hadoop HDFS
OpenTSDB
(http://opentsdb.net)
SciDB (DMAS)

11. Analysis
Hadoop Streaming (Ruby)
Hadoop Hive (Using rbhive)

12. Visualization
Custom Dashboard
(Rails + Google Charts / d3.js)
Some Hosted Services: tableaupublic.com, geckoboard.com, splunkstorm.com

13. Stream Computing

14. What is STORM?

15. STORM terminology
●
Streams
●
Spouts
●
Bolts
●
Topologies

16. RedStorm
(https://github.com/colinsurprenant/redstorm)
$ rvm use jruby-1.6.3
$ bundle install redstorm
$ bundle exec redstorm install

17. Visualizing average bandwidth
experienced by users while
watching videos on viki.com across
the globe.

24. Thank you!
Let's stay in touch :)
●
Signup for my newsletter at http://parolkar.com
●
Visit BigData.SG Meetup in Singapore.