A Multiple Pairs Shortest Path Algorithm 解説Osamu Masutani
National Cheng Kung UniversityのWang氏の多点間最短経路探索(MPSP)に関する論文の解説。
Wang, I-Lin, Ellis L. Johnson, and Joel S. Sokol. "A multiple pairs shortest path algorithm." Transportation science 39.4 (2005): 465-476.
A Multiple Pairs Shortest Path Algorithm 解説Osamu Masutani
National Cheng Kung UniversityのWang氏の多点間最短経路探索(MPSP)に関する論文の解説。
Wang, I-Lin, Ellis L. Johnson, and Joel S. Sokol. "A multiple pairs shortest path algorithm." Transportation science 39.4 (2005): 465-476.
Short introduction of GDB/Python, with actual examples for debugging and (sort of) running Ruby-as-a-debugger-of-oneself on top of it.
This was prepared for YamiRubyKaigi 2011 (Darkness of RubyKaigi2011), but unfortunately gave up doing actual presentation due to too many pages.
4. Hadoop Streaming
Hadoop Streamingの仕組みを使うと、標準入出力に対応する
任意の言語でMap/Reduceを書くことができる
• WordCountの例
#!/usr/bin/env ruby
$stdin.each do |line|
line.strip.split(" ").each do |w|
puts "#{w}¥t1"
end
end
line = $stdin.gets
w, f = line.strip.split("¥t")
w_prev = w
f_sum = f.to_i
$stdin.each do |line|
w, f = line.strip.split("¥t")
if w == w_prev
f_sum += f.to_i
else
puts "#{w_prev} #{f_sum}"
w_prev = w
f_sum = f.to_i
end
end
puts "#{w_prev} #{f_sum}"
A B C D
A C B E
A E
Mapper
Reducer
A 1
A 1
A 1
B 1
B 1
C 1
C 1
D 1
E 1
E 1
A 3
B 2
C 2
D 1
E 2
5. Hadoop Streaming
Hadoop Streamingの仕組みを使うと、標準入出力に対応する
任意の言語でMap/Reduceを書くことができる
• WordCountの例
#!/usr/bin/env ruby
$stdin.each do |line|
line.strip.split(" ").each do |w|
puts "#{w}¥t1"
end
end
line = $stdin.gets
w, f = line.strip.split("¥t")
w_prev = w
f_sum = f.to_i
$stdin.each do |line|
w, f = line.strip.split("¥t")
if w == w_prev
f_sum += f.to_i
else
puts "#{w_prev} #{f_sum}"
w_prev = w
f_sum = f.to_i
end
end
puts "#{w_prev} #{f_sum}"
A B C D
A C B E
A E
Mapper
Reducer
A 1
A 1
A 1
B 1
B 1
C 1
C 1
D 1
E 1
E 1
A 3
B 2
C 2
D 1
E 2
Amazon Web Serviceの
ElasticMapReduceのサービスを利用して
簡単にHadoop Streamingが使える !
→Rubyで大規模データ処理!
6. 当然こんなちょっとした処理も可能
• 大規模データのIDのマッピング
商品1 ID1
商品2 ID2
...
商品100万 ID100万
def load_idmap(idmapfile)
idmap = {}
puts idmapfile
open(idmapfile).each do |line|
rkey, rval = line.strip.split(" ")
idmap[rkey] = rval
end
idmap
end
idmap = load_idmap("idmap.dat")
$stdin.each do |line|
line.strip.split(" ").each do |w|
puts "#{idmap[w]}¥t1"
end
end
Mapper
A B C D
A C B E
A E
1 2 3 4
1 3 2 5
1 5