Presented at Rubyconf India 2016

1. Streaming API,
Spark & Ruby

2. AGENDA

3. BIG DATA

4. Ruby in BIG DATA

6. Why we should know?

7. Insights from SAS

11. Data

12. Information

13. Knowledge

15. Data Science Field

16. Computer
Science
Maths &
Statitics
Subject
Matter
Expertise

17. Where it all began?

18. PROBLEM

19. How to store ?
How to process ?

20. Solution

21. HADOOP

23. HADOOP Ecosystem

25. STORAGE & PROCESS

26. • Distributed Storage: HDFS A distributed file
system where commodity hardware can be
used to form clusters and store the huge data
in distributed fashion.
• Distributed Processing: MapReduce Paradigm
It can easily scale to multiple nodes(1,500–2,000
nodes in a cluster), with just configuration
change.

27. Case Study

28. Daywise Analysis of
Rubyconfindia Tweets

30. Twitter API

31. Sample Data

32. --- !ruby/object:Twitter::Tweet
attrs:
:created_at: Tue Mar 08 11:00:57 +0000 2016
:id: 707159160945811457
:id_str: '707159160945811457'
:text: 'Once in a life time to meet Matz at the
awesome #kochi https://t.co/6oCIagsHCg
#ruby #india https://t.co/YRlpABApkP'

33. Tweet Count

35. CLOUDERA QUICKSTART

36. STORAGE LAYER

37. HDFS

38. • It is a distributed file system
• Streaming Data Access: Write once, read many
times
• Able to run on commodity Hardware
• Fault tolerance
• Replication: 3 nodes by default, configurable
• Block based: 64-256MB, configurable

39. Replication Factor 2

40. Name Node: Stores Meta Data
Meta Data:
/data/pristine/catalina.log.> 1, 2, 4
/data/pristine/myfile. >3,5
Data Node 1 Data Node 2 Data Node 3
1 2 45 5 2 3 4 1 3

41. HDFS Command Line

43. Copying File to HDFS

45. HDFS NameNode UI

47. PROCESS LAYER

48. YARN / MapReduce 2.0

49. • YARN: A framework for job
scheduling and cluster resource
management.
• MapReduce: Distributed
processing paradigm

50. Map Function
Input:
(input_key, value)
Output:
bunch of
(intermediate_key, value)
System applies the map
function in parallel to all
inputs
Reduce Function
Input:
(intermediate_key, value)
Output:
bunch of (values)
System will group all pairs
with the same
intermediate key and
apply the reduce function
OUTPUT RESULT
SHUFFLE
STAGE
FILE CHUNKS

51. Leveraging Ruby

52. Mapper.rb

53. #!/usr/bin/env ruby
STDIN.read.split("---
!ruby/object:Twitter::Tweet").each do |t|
date = t.match(/:created_at: .{30}/).to_s.split
puts "#{date[1]}" if date[1]
end

54. Reducer.rb

55. #!/usr/bin/env ruby
STDIN.readlines.group_by{|i| i.strip}
.map{|i,j| "#{i} #{j.count}" }
.each{|i| puts i}

56. HADOOP STREAMING API

57. hadoop jar /usr/lib/hadoop-mapreduce/hadoop-
streaming.jar –input /user/rubyconf/tweets.txt –output
/user/rubyconf/daywise –mapper mapper.rb –reducer
reducer.rb –file mapper.rb –file reducer.rb

59. YARN Resource Manager

61. Job Details

63. YARN Resource Manager

65. DAYWISE TWEETS

67. TREND

69. SPARK

70. SPARK Benefits
• SPEED
• Ease of Use
• Runs Everywhere
• Source: http://spark.apache.org/

71. • Source: http://spark.apache.org/

72. gem ruby-spark
• gem install ruby-spark
• ruby-spark build
• ruby-spark shell

73. Ruby with SPARK

74. require 'ruby-spark'
# Configuration
Spark.config do
set_app_name "RubySpark"
set 'spark.ruby.serializer', 'oj'
set 'spark.ruby.serializer.batch_size', 100
end
# Start Apache Spark
Spark.start

75. # Context reference
sc = Spark.sc
rdd = sc.text_file("hdfs://user/rubyconf/tweets.txt”)
# Collect all created days from dates
days = rdd.map(lambda {|t| date = t.match(/:created_at:
.{30}/).to_s.split; date[1] if date[1]})
# Creating key value pair
pairrdd = days.map(lambda { |x| [x,1] })
# Final output by using reducer
daywise = pairrdd.reduce_by_key( lambda{|x,y|
x+y}).collect_as_hash

76. Expertise & Learnings

78. Remember

79. We can use Ruby with
HADOOP Streaming API
& SPARK

80. SPARK is more
generalized distributed
computing model

81. Thank You
@_manoharaa