What is mt-chamber?

What is mt-chamber?
Koichi Akabe
AHC Lab
2015-04-25
2015-04-25 Koichi Akabe (AHC-Lab) 1 / 20

Building mathine translation system is complicated
File Reader
I don't like it.
What's your...
Yes, we can!
...
...
File Reader
僕は嫌いです。
..は何ですか？
我々にはできる！
...
...
Tokenizer EN
I/do/n't/like/it/.
What/'s/your/...
Yes/,/we/can/!
...
...
Tokenizer JA
僕/は/嫌い/で/す/。
../は/何/で/す/か/？
我々/に/は/でき/る/！
...
...
Cleaner
I/do/n't/like/it/.
Yes/,/we/can/!
...
...
Parser EN
I do n't like it .
...
...
Parser JA
...
...
...
...Remove pairs
if len(words) > 80
▶ Tokenization (splits into words)
▶ Cleaning (removes long lines, personal info, and garbage)
▶ Syntax parsing
▶ Alignment (makes pairs of same meaning words)
▶ Training (extracts translation rules)
▶ Tuning (adjusts parameters)

We want:
▶ to write this process easier.
▶ to run this process faster.

“UNIX Pipeline” can make some processes faster
File Reader
I don't like it.
What's your...
Yes, we can!
...
...
File Reader
...
...
Tokenizer EN
I/do/n't/like/it/.
What/'s/your/...
Yes/,/we/can/!
...
...
Tokenizer JA
../は/何/で/す/か/？
...
...
Cleaner
I/do/n't/like/it/.
Yes/,/we/can/!
...
...
Parser EN
I do n't like it .
...
...
Parser JA
...
...
...
...

“UNIX Pipeline” can make some processes faster
File Reader
I don't like it.
What's your...
Yes, we can!
...
...
File Reader
...
...
Tokenizer EN
I/do/n't/like/it/.
What/'s/your/...
Yes/,/we/can/!
...
...
Tokenizer JA
../は/何/で/す/か/？
...
...
Cleaner
I/do/n't/like/it/.
Yes/,/we/can/!
...
...
Parser EN
I do n't like it .
...
...
Parser JA
...
...
...
...
▶ ○ Tokenization (splits into words)
▶ ○ Cleaning (removes long lines, personal info, and garbage)
▶ ○ Syntax parsing
▶ △ Alignment (makes pairs of same meaning words)
▶ Training (extracts translation rules)
▶ Tuning (adjusts parameters)

How to run it under multi-thread environment
File Reader
I don't like it.
What's your...
Yes, we can!
...
...
File Reader
...
...
Tokenizer EN
I/do/n't/like/it/.
What/'s/your/...
Yes/,/we/can/!
...
...
Tokenizer JA
../は/何/で/す/か/？
...
...
Cleaner
I/do/n't/like/it/.
Yes/,/we/can/!
...
...
Parser EN
I do n't like it .
...
...
Parser JA
...
...
...
...
Single-thread Multi-thread

More complicated situation
File Reader
File Reader
Tokenizer
Tokenizer
Cleaner
Parser
Parser
File WriterLower Caser
EN
JA
EN
JA
EN
JA
EN
EN
EN EN
JAJA
JA
EN
JA
JA
JAJA

Named pipe in UNIX supports multi-thread pipeline!
“Named pipe” is an implementation of multi-producer/-consumer
queue.
$ mkfifo ./ tok_queue # create named pipe
$ mkfifo ./ cln_queue
$ cleaner < ./ cln_queue > ... &
$ cleaner < ./ cln_queue > ... &
....
$ tokenizer < ./ tok_queue > ./ cln_queue &
$ tokenizer < ./ tok_queue > ./ cln_queue &
....
$ cat ./ corpus_data > ./ tok_queue
tok_queue
tokenizer
cln_queue
cleaner
cat

Solution?

Simple multi-producer/-consumer queue will break order
Single-thread (ST) programs will require sorted data, but it is not
guaranteed in multi-thread (MT) programs.
1
2
3
4
5
...
1
2
3
4
2
4
1
3
4
3
1
2
MT
Process
ST
Process
MT
Process
ST
Process
Queue
Queue
Items of faster threads are pushed into the beginning of a queue.

Also multi-thread programs have to care order if they use
multiple input
Pairs will be broken if each thread puts items into queues
independently.
1
2
3
4
4,1
3,3
1,2
2,4
MT
Process
ST
Process
MT
Process
1
2
3
4
MT
Process
ST
Process
1
2
3
4
5
...
Queue
1
2
3
4
5
...
Queue

mt-chamber
A framework for multi-thread pipeline process

mt-chamber guarantees line order and correct pairs
1
2
3
4
5
...
1
2
3
4
1
2
3
4
4
3
1
2
MT
Process
ST
Process
MT
Process
ST
Process
Queue
Queue
Blocks higher numbered items

mt-chamber guarantees line order and correct pairs
1
2
3
4
4,4
3,3
1,1
2,2
MT
Process
ST
Process
MT
Process
1
2
3
4
MT
Process
ST
Process
1
2
3
4
5
...
Queue
1
2
3
4
5
...
Queue
Makes correct pairs
before putting items

Short example
# ./ example_code
Alias KyTeaTokenizer System:command =" kytea -notags"
Read:file ="./ input" > inputdata
KyTeaTokenizer < inputdata > tokenized
Write:file ="./ output" < tokenized
$ ./mt -chamber.py --threads 20 < ./ example_code
Read System
./input kytea
Write
./output

Multiple input
Read:file ="./ en" > EN_raw
Read:file ="./ ja" > JA_raw
WesternTokenizer < EN_raw > EN_tok
KyTeaTokenizer < JA_raw > JA_tok
LengthCleaner < EN tok JA tok > EN clean JA clean
Write:file ="./ en.clean" < EN_clean
Write:file ="./ ja.clean" < JA_clean
Read KyTea Tok
./ja
Read Western Tok
./en
Length
Cleaner
Write
./ja.clean
Write
./en.clean

To deﬁne your own commands, write plugins
# plugins/LowerCaser.py
class Command:
InputSize = 1
OutputSize = 1
MultiThreadable = True
ShareResources = True
def routine(self , instream):
return (instream [0]. lower () ,)
All commands are implemented like this.

Experiments

Experiment setup
Measure performance of tokenizing and syntax parsing in each
environment.
Read line −→ Tokenize −→ Clean −→ Syntax parse −→ Write line
Thread size comparison:
▶ Unix pipe (each process is single thread)
▶ 1 thread (./mt-chamber.py --threads 1)
▶ 5 threads
▶ 10 threads
unsrt-limit comparison:
(“unsrt-limit” is the place of higher numbered items to wait lower
numbered ones for sorting)
▶ 100 items
▶ 500 items
▶ 1000 items

Experiment setup
Corpus:
▶ English-Japanese translation corpus
▶ 10000 sentences before cleaning (after: 9920 sentences)
▶ Japanese tokenizer: KyTea
▶ Syntax parser: Ckylark (both languages)

Experiment results
0
2000
4000
6000
Time[sec]
Unix Pipe 1 thread 5 threads 10 threads
unsrt-limit = 100
500
1000
▶ 1 thread mt-chamber is not slower than “Unix Pipe”
▶ unsrt-limit is important to run pipelines faster in multi-thread

See also
https://github.com/vbkaisetsu/mt-chamber

What is mt-chamber?

Recommended

Recommended

More Related Content

Similar to What is mt-chamber?

Similar to What is mt-chamber? (20)

Recently uploaded

Recently uploaded (20)

What is mt-chamber?