ZFS test

  • 473 views
Uploaded on

 

More in: Technology , Sports
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
473
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. ZFS test with VM on PC
  • 2. Alignment test • Commands – cp ~/r100609_1_thy__coreRag1__Rag1_E1.fastq . – ~/src/bowtie2-2.1.0/bowtie2 -x ~/indexes/mm9 - U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.sam -- threads 2 • Check the process status (execution time ,CPU, memory) with “ps” command for each 10 seconds
  • 3. Alignment test Uncompressed Compressed resident set size, the non-swapped physical memory that a task has used (in kiloBytes) % CPU Memory
  • 4. I/O performance • Copy 1GB divided by 1 – 10 blocks • Command – for count in $(seq 1 10); do for directory in uncompressed compressed; do dd if=/dev/zero of=/mnt/$directory/foovar count=$count bs=`perl -e "print int(1000000000 / $count - 0.5) + 1"` 2>&1 | tail -n1 | sed 's/^.*, //'; done; done | paste - - > dd.txt &
  • 5. I/O performance Blocks Uncompressed Compressed 1 50.1 MB/s 59.1 MB/s 2 56.0 MB/s 54.5 MB/s 3 56.0 MB/s 57.8 MB/s 4 57.6 MB/s 52.7 MB/s 5 53.9 MB/s 58.9 MB/s 6 57.2 MB/s 61.4 MB/s 7 52.3 MB/s 60.0 MB/s 8 43.9 MB/s 60.7 MB/s 9 58.2 MB/s 57.3 MB/s 10 48.1 MB/s 61.3 MB/s
  • 6. File system disk space usage (df) Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/fedora-root 64247652 25341508 35635900 42% / devtmpfs 1017952 0 1017952 0% /dev tmpfs 1025732 80 1025652 1% /dev/shm tmpfs 1025732 708 1025024 1% /run tmpfs 1025732 0 1025732 0% /sys/fs/cgroup tmpfs 1025732 12 1025720 1% /tmp /dev/sda1 487652 67503 394549 15% /boot 129.112.138.227:/mnt/zfs1 203341312 8697920 194643392 5% /mnt/uncompressed 129.112.138.227:/mnt/zfs2 203341376 3823808 199517568 2% /mnt/compressed
  • 7. File system disk space usage (df) Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/fedora-root 64247652 25341508 35635900 42% / devtmpfs 1017952 0 1017952 0% /dev tmpfs 1025732 80 1025652 1% /dev/shm tmpfs 1025732 708 1025024 1% /run tmpfs 1025732 0 1025732 0% /sys/fs/cgroup tmpfs 1025732 12 1025720 1% /tmp /dev/sda1 487652 67503 394549 15% /boot 129.112.138.227:/mnt/zfs1 203341312 2181504 201159808 2% /mnt/uncompressed 129.112.138.227:/mnt/zfs2 203341376 2181376 201160000 2% /mnt/compressed After gzipping of data files
  • 8. ZFS test with Amazon EC2
  • 9. Alignment test 1 • Commands (4 processes using 4 CPU cores) – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.1.sam --threads 1 & – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.2.sam --threads 1 & – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.3.sam --threads 1 & – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.4.sam --threads 1 &
  • 10. Alignment test 1 • The test processes were terminated with memory error. Out of memory allocating the offs[] array for the Bowtie index. Please try again on a computer with more memory. Error: Encountered internal Bowtie 2 exception (#1) Command: /home/ec2-user/src/bowtie2-2.1.0/bowtie2-align -- wrapper basic-0 -x indexes/mm9 -S r100609_1_thy__coreRag1__Rag1_E1.1.sam --threads 1 -U r100609_1_thy__coreRag1__Rag1_E1.fastq bowtie2-align exited with value 1
  • 11. Alignment test 2 • Commands (2 processes using 4 CPU cores) – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.1.sam -- threads 2 & – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.2.fastq -S r100609_1_thy__coreRag1__Rag1_E1.2.sam -- threads 2 &
  • 12. Alignment test 2 Uncompressed resident set size, the non-swapped physical memory that a task has used (in kiloBytes) % CPU Memory
  • 13. Alignment test 2 Compressed resident set size, the non-swapped physical memory that a task has used (in kiloBytes) % CPU Memory
  • 14. Alignment test 3 • Checking the total memory usage • Commands (1 process using 2 CPU cores) – free -t -s 1 – ~/src/bowtie2-2.1.0/bowtie2 -x indexes/mm9 -U r100609_1_thy__coreRag1__Rag1_E1.fastq -S r100609_1_thy__coreRag1__Rag1_E1.sam -- threads 4 &
  • 15. Alignment test 3 Uncompressed Compressed
  • 16. I/O performance (Read rate) Blocks Uncompressed Compressed 1 1.1 GB/s 1.1 GB/s 2 1.6 GB/s 1.6 GB/s 3 1.9 GB/s 2.0 GB/s 4 2.2 GB/s 2.2 GB/s 5 2.3 GB/s 2.4 GB/s 6 2.4 GB/s 2.4 GB/s 7 2.5 GB/s 2.5 GB/s 8 2.6 GB/s 2.6 GB/s 9 2.6 GB/s 2.6 GB/s 10 2.6 GB/s 2.7 GB/s
  • 17. I/O performance (Write rate 1) Blocks Uncompressed Compressed 1 43.6 MB/s 324 MB/s 2 1.1 GB/s 1.1 GB/s 3 1.2 GB/s 43.9 MB/s 4 42.8 MB/s 1.3 GB/s 5 46.0 MB/s 165 MB/s 6 57.1 MB/s 46.3 MB/s 7 374 MB/s 1.4 GB/s 8 1.1 GB/s 45.4 MB/s 9 42.5 MB/s 44.8 MB/s 10 510 MB/s 1.4 GB/s
  • 18. I/O performance (Write rate 2) Blocks Uncompressed Compressed 1 814 MB/s 812 MB/s 2 1.1 GB/s 45.5 MB/s 3 43.3 MB/s 50.4 MB/s 4 233 MB/s 1.3 GB/s 5 1.3 GB/s 45.5 MB/s 6 41.8 MB/s 1.3 GB/s 7 98.6 MB/s 176 MB/s 8 60.4 MB/s 599 MB/s 9 47.3 MB/s 38.0 MB/s 10 1.5 GB/s 1.4 GB/s
  • 19. File system disk space usage (df) Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvde1 6193088 2393636 3736840 40% / none 7619740 0 7619740 0% /dev/shm tank/compressed 79563136 12784384 66778752 17% /mnt/compressed tank/uncompressed 89918080 23139328 66778752 26% /mnt/uncompressed tank 66778752 0 66778752 0% /tank 100GB disk pooling (64G + 13G + 23G)
  • 20. Conclusion • We adopted a short-read alignment step with bowtie2 to the tests of ZFS, as it would mainly utilize the computational resource including storage disk. • The tests show that there is no considerable difference of resource usage between on uncompressed and compressed disk partitions of ZFS. – Execution time, CPU, memory, I/O performance – But it may need more tests in more computational burden to recognize the stability of ZFS. • The adoption of compressed system makes more efficient usage of our storage, especially with unzipped data files. (refer slide 6-7)