Concurrency model for mysql data processing@rubyconf.tw 2012

Concurrency Model for
huge MySQL data
Mu-Fan Teng(@ryudoawaru)
@ RubyConf Taiwan 2012

12年12月8⽇日星期六

緣起

Background


Legacy environment
• A Mysql Database with 2.3gb data with Big5
charset and ISO-8859-1 encoding.
• The biggest table in DB is 1.5gb.


The purpose


Transcoding to UTF8


Try


Work Flow
1. mysqldump with -default-character-
set=latin1 parameter to generate SQL file.
2. Transcoding SQL file with tool like iconv/
bsdconv.
3. Edit transcoded SQL file to avoid 「slash」
problem.
4. Restore SQL file to new DB.


Failed!


Cause
• Too big size for most text editor.
• Many mis-encoding text.


Let’s reinvent the
wheel!


The new work ﬂow
• Connect DB
• Transcode
• Output db rows to SQL insert statement.
• Write SQL ﬁle


CORES_COUNT = 4
LIMIT = ARGV[0].to_i || 10000
sqls = CORES_COUNT.times.map do |x|
sprintf("SELECT * FROM cdb_posts ORDER BY pid LIMIT %d OFFSET %d;", LIMIT,
(x * LIMIT))
end

class String
def to_my_val
"'#{Mysql2::Client.escape self.force_encoding(‘Big5-
UAO’).encode('UTF-8', :invalid => :replace, :undef => :replace, :replace =>
'??')}'"
end
end

procs = sqls.map do |sql|
Proc.new do |out|
Mysql2::Client.new(database: DBNAME, reconnect: true, encoding:
'latin1').query(sql).each(as: :array) do |row|
out.print "INSERT INTO `cdb_posts` VALUES
(#{row.map(&:to_my_val).join(',')});n"
end
end
end
procs.each{|p| p.call(OUT)}


Thanks for Ruby 1.9’s
Awesome Encoding
class which supports
Big5-UAO.


Reduces almost 80% of
encoding problem.


But the ﬁle size is too
big to wait for
transcoding!


So I have to ﬁnd the
concurrency model to
make it faster.


Experiment Target
• Test the difference of performance
between thread and fork model.


H&W Platform
• 4 Cores Core2Quad CPU@2.5G
• 8GB RAM
• 1*SSD
• MacOS 10.8
• MRI 1.9.3p194


DBNAME = 'wwwfsc'
CORES_COUNT = 4
ForceEncoding = 'Big5-UAO'
LIMIT = ARGV[0].to_i || 10000
OUT = '/dev/null'

sprintf("SELECT * FROM cdb_posts ORDER BY pid LIMIT %d OFFSET %d;", LIMIT, (x
* LIMIT))
end

class String
def to_my_val
"'#{Mysql2::Client.escape
self.force_encoding(ForceEncoding).encode('UTF-8', :invalid => :replace, :undef
=> :replace, :replace => '??')}'"
end
end

Proc.new do |out|
open(out,'w') do |io|
io.print "INSERT INTO `cdb_posts` VALUES
end
end
end
end


Benchmark.bm(15) do |x|
x.report("Thread"){procs.map{|p| Thread.new{p.call(OUT)} }.each(&:join)}
x.report("Fork"){procs.each{|p| fork{p.call(OUT)} }; Process.waitall}
x.report("Normal"){procs.each{|p| p.call(OUT)}}
end


Result of 100k*4 rows


Thread


Fork


Circumstance
• Thread
‣ CPU utilization rate between 105 and
125 percent.
• Fork
‣ The rate changes frequently between
processes.


GVL still effects


Try again


Decompose the steps
to ﬁnd how to skip
GVL.


Experiment No.2


Minify the process to
query DB only.


DBNAME = 'wwwfsc'
CORES_COUNT = 4
limit = ARGV[0].to_i || 10000

sprintf("SELECT * FROM cdb_posts ORDER BY pid LIMIT %d OFFSET %d;", limit, (x *
limit))
end
procs = CORES_COUNT.times.map do |x|
Proc.new do
client = Mysql2::Client.new(database: DBNAME, reconnect: true)
result = client.query(sqls[x])
end
end

Benchmark.bmbm(15) do |x|
x.report("Thread"){procs.map{|p| Thread.new{p.call} }.each(&:join)}
x.report("Fork"){procs.each{|p| fork{p.call} }; Process.waitall}
x.report("Normal"){procs.each(&:call)}
end


It seems the Mysql2
Gem can skip GVL.


Experiment NO.3


Limit the experiment
to I/O operation only.


client = Mysql2::Client.new(database: DBNAME, reconnect: true, encoding:
'latin1')
sql_raws = sqls.map do |sql|
arr = []
client.query(sql).each(as: :array) do |row|
arr << "(#{row.map(&:to_my_val).join(',')})"
end
arr
end

procs = sql_raws.map do |arr|
proc do
io = open('/dev/null','w')
io.write "INSERT INTO `cdb_posts` VALUES "
io.write arr.join(',')
io.write "n"
io.close
end
end

Benchmark.bm(15) do |x|
x.report("Thread"){procs.map{|p| Thread.new{p.call} }.each(&:join)}
x.report("Fork"){procs.each{|p| fork{p.call} }; Process.waitall}
x.report("Normal"){procs.each{|p| p.call}}
end


Change I/O to different
ﬁles.


procs = sql_raws.map do |arr|
proc do
io = Tempfile.new(SecureRandom.uuid)#open('/dev/null','w')
puts io.path
io.write "INSERT INTO `cdb_posts` VALUES "
io.write arr.join(',')
io.write "n"
io.close
end
end


Result reversed


Implement the same
change to the ﬁrst
experiment.


Proc.new do
io = Tempfile.new(SecureRandom.uuid)
io.write "INSERT INTO `cdb_posts` VALUES
end
io.close
end
end


Dose not effect any


Conclusion
Thread fork normal

MySQL2-read Fast Fast x

Transcoding &
Slow Very fast x
iteration
Write to the
Very slow Slow Fast
same I/O
Write to the
Fast Slow Fast
different I/O


There is no effective
and 「all-around」
concurrency model.


The small I/O can’t
release GVL.


Kosaki-san’s slide


Matz is not a threading guy


End


Concurrency model for mysql data processing@rubyconf.tw 2012

More Related Content

What's hot

Viewers also liked

More from Mu-Fan Teng

Concurrency model for mysql data processing@rubyconf.tw 2012