ZFS test
Upcoming SlideShare
Loading in...5
×
 

ZFS test

on

  • 644 views

 

Statistics

Views

Total Views
644
Views on SlideShare
484
Embed Views
160

Actions

Likes
0
Downloads
2
Comments
0

1 Embed 160

http://qbrc.swmed.edu 160

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

ZFS test ZFS test Presentation Transcript

  • ZFS test with VM on PC
  • Alignment test • Commands – cp ~/r100609_1_thy__coreRag1__Rag1_E1.fastq . – ~/src/bowtie2-2.1.0/bowtie2 -x ~/indexes/mm9 - U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.sam -- threads 2 • Check the process status (execution time ,CPU, memory) with “ps” command for each 10 seconds
  • Alignment test Uncompressed Compressed resident set size, the non-swapped physical memory that a task has used (in kiloBytes) % CPU Memory
  • I/O performance • Copy 1GB divided by 1 – 10 blocks • Command – for count in $(seq 1 10); do for directory in uncompressed compressed; do dd if=/dev/zero of=/mnt/$directory/foovar count=$count bs=`perl -e "print int(1000000000 / $count - 0.5) + 1"` 2>&1 | tail -n1 | sed 's/^.*, //'; done; done | paste - - > dd.txt &
  • I/O performance Blocks Uncompressed Compressed 1 50.1 MB/s 59.1 MB/s 2 56.0 MB/s 54.5 MB/s 3 56.0 MB/s 57.8 MB/s 4 57.6 MB/s 52.7 MB/s 5 53.9 MB/s 58.9 MB/s 6 57.2 MB/s 61.4 MB/s 7 52.3 MB/s 60.0 MB/s 8 43.9 MB/s 60.7 MB/s 9 58.2 MB/s 57.3 MB/s 10 48.1 MB/s 61.3 MB/s
  • File system disk space usage (df) Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/fedora-root 64247652 25341508 35635900 42% / devtmpfs 1017952 0 1017952 0% /dev tmpfs 1025732 80 1025652 1% /dev/shm tmpfs 1025732 708 1025024 1% /run tmpfs 1025732 0 1025732 0% /sys/fs/cgroup tmpfs 1025732 12 1025720 1% /tmp /dev/sda1 487652 67503 394549 15% /boot 129.112.138.227:/mnt/zfs1 203341312 8697920 194643392 5% /mnt/uncompressed 129.112.138.227:/mnt/zfs2 203341376 3823808 199517568 2% /mnt/compressed
  • File system disk space usage (df) Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/fedora-root 64247652 25341508 35635900 42% / devtmpfs 1017952 0 1017952 0% /dev tmpfs 1025732 80 1025652 1% /dev/shm tmpfs 1025732 708 1025024 1% /run tmpfs 1025732 0 1025732 0% /sys/fs/cgroup tmpfs 1025732 12 1025720 1% /tmp /dev/sda1 487652 67503 394549 15% /boot 129.112.138.227:/mnt/zfs1 203341312 2181504 201159808 2% /mnt/uncompressed 129.112.138.227:/mnt/zfs2 203341376 2181376 201160000 2% /mnt/compressed After gzipping of data files
  • ZFS test with Amazon EC2
  • Alignment test 1 • Commands (4 processes using 4 CPU cores) – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.1.sam --threads 1 & – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.2.sam --threads 1 & – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.3.sam --threads 1 & – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.4.sam --threads 1 &
  • Alignment test 1 • The test processes were terminated with memory error. Out of memory allocating the offs[] array for the Bowtie index. Please try again on a computer with more memory. Error: Encountered internal Bowtie 2 exception (#1) Command: /home/ec2-user/src/bowtie2-2.1.0/bowtie2-align -- wrapper basic-0 -x indexes/mm9 -S r100609_1_thy__coreRag1__Rag1_E1.1.sam --threads 1 -U r100609_1_thy__coreRag1__Rag1_E1.fastq bowtie2-align exited with value 1
  • Alignment test 2 • Commands (2 processes using 4 CPU cores) – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.1.sam -- threads 2 & – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.2.fastq -S r100609_1_thy__coreRag1__Rag1_E1.2.sam -- threads 2 &
  • Alignment test 2 Uncompressed resident set size, the non-swapped physical memory that a task has used (in kiloBytes) % CPU Memory
  • Alignment test 2 Compressed resident set size, the non-swapped physical memory that a task has used (in kiloBytes) % CPU Memory
  • Alignment test 3 • Checking the total memory usage • Commands (1 process using 2 CPU cores) – free -t -s 1 – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.sam -- threads 4 &
  • Alignment test 3 Uncompressed Compressed
  • I/O performance (Read rate) Blocks Uncompressed Compressed 1 1.1 GB/s 1.1 GB/s 2 1.6 GB/s 1.6 GB/s 3 1.9 GB/s 2.0 GB/s 4 2.2 GB/s 2.2 GB/s 5 2.3 GB/s 2.4 GB/s 6 2.4 GB/s 2.4 GB/s 7 2.5 GB/s 2.5 GB/s 8 2.6 GB/s 2.6 GB/s 9 2.6 GB/s 2.6 GB/s 10 2.6 GB/s 2.7 GB/s
  • I/O performance (Write rate 1) Blocks Uncompressed Compressed 1 43.6 MB/s 324 MB/s 2 1.1 GB/s 1.1 GB/s 3 1.2 GB/s 43.9 MB/s 4 42.8 MB/s 1.3 GB/s 5 46.0 MB/s 165 MB/s 6 57.1 MB/s 46.3 MB/s 7 374 MB/s 1.4 GB/s 8 1.1 GB/s 45.4 MB/s 9 42.5 MB/s 44.8 MB/s 10 510 MB/s 1.4 GB/s
  • I/O performance (Write rate 2) Blocks Uncompressed Compressed 1 814 MB/s 812 MB/s 2 1.1 GB/s 45.5 MB/s 3 43.3 MB/s 50.4 MB/s 4 233 MB/s 1.3 GB/s 5 1.3 GB/s 45.5 MB/s 6 41.8 MB/s 1.3 GB/s 7 98.6 MB/s 176 MB/s 8 60.4 MB/s 599 MB/s 9 47.3 MB/s 38.0 MB/s 10 1.5 GB/s 1.4 GB/s
  • File system disk space usage (df) Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvde1 6193088 2393636 3736840 40% / none 7619740 0 7619740 0% /dev/shm tank/compressed 79563136 12784384 66778752 17% /mnt/compressed tank/uncompressed 89918080 23139328 66778752 26% /mnt/uncompressed tank 66778752 0 66778752 0% /tank 100GB disk pooling (64G + 13G + 23G)
  • Conclusion • We adopted a short-read alignment step with bowtie2 to the tests of ZFS, as it would mainly utilize the computational resource including storage disk. • The tests show that there is no considerable difference of resource usage between on uncompressed and compressed disk partitions of ZFS. – Execution time, CPU, memory, I/O performance – But it may need more tests in more computational burden to recognize the stability of ZFS. • The adoption of compressed system makes more efficient usage of our storage, especially with unzipped data files. (refer slide 6-7)