Bioawk is a tool that extends GNU awk to facilitate working with biological file formats like FASTA, FASTQ, SAM, BED, GFF, and VCF. It directly reads gzipped files and treats spanning sequences as single records. Some key functions added in Bioawk include calculating GC content, reversing/reverse complementing sequences, and working with quality values. Bioawk allows for convenient parsing, manipulation and statistical analysis of genomic data.

1. Awk primer
and
Bioawk
Coby Viner
Lab meeting: tech – Wednesday, Dec. 2, 2020

2. (GNU) awk
• A complete programming language
• Operates on a per-line (row) basis
• Designed to operate upon columnar data
• By default, whitespace-delimited columns
• Outputs on a per-row basis

3. General syntax
BEGIN { print "START"; }
{ print }
END { print "END"; }'
<file 1> … <file N>
Adapted from Bruce Barnett’s “Intro. To AWK”: https://www.grymoire.com/Unix/Awk.html.
$0;
Columns: $1, $2, …, $NF
awk '

4. Key special variables
• NF – Number of fields (columns)
• NR – Number of records (rows; all files)
• FNR – Number of records (rows; per file)
• FS – (input) field separator (default: " ")
• OFS – Output field separator (default: " ")

5. Examples
BEGIN { FS=OFS="t"; }
{ print $1,$2,$3; }
FNR > 1 { }
grep -v 'chr[mM]' <f> | awk '{print
$1,$2,$3}' | sed 's/chr//;'
awk '$1 !~ /chr[mM]/ {sub(/chr/, "");
print $1,$2,$3}' <f>

6. Examples: two files
awk 'FNR==NR{a[$1]=$2; next}
{print $1,$2,a[$2]; }'
<file 1> <file 2>

7. Bioawk, by Heng Li
• Behaves like GNU awk, on non-bio. data
• Install from GitHub repo. or others' Dockers
• Supports: BED, GFF, FASTA, FASTQ, SAM, VCF
 List formats with: bioawk -c help
• Directly reads gzipped files (usually)
• -t short for bioawk -F't' -v OFS="t"
• Treats spanning seqs as a single record

8. Bioawk - generic/BED files
Parse column names:
bioawk -c header '{ print $chr }'
<file.gz>
chr1
chr3
chrX

9. Bioawk - examples - GFF*/GTFs
Find all exons less than 100 bp, which are
annotated as the main functional isoform
(i.e., APPRIS principal 1):
bioawk -c gff '$feature == "exon" &&
($end - $start) < 100 &&
$attribute ~ /appris_principal_1/'
gencode.vXX.annotation.gtf.gz
Example adapted from https://hpc.nih.gov/apps/bioawk.html

10. Bioawk - examples - FASTAs
Reverse complement:
bioawk -c fastx
'{print ">"$name;
print revcomp($seq)}'
seq.fa.gz
Example taken from the README.

11. Bioawk - examples - FASTAs
List of sequence names and lengths:
bioawk -c fastx
'{print $name,
length($seq)}'
seq.fa.gz
Adapted from a DNA.today blog post, by Jean-Yves Sgro (January 25, 2020).

12. Bioawk - examples - FASTQs
%GC and mean Phred quality score:
awk -c fastx
'{ print ">"$name;
print gc($seq);
print meanqual($qual);
}' seq.fq.gz
Adapted from Istvan Albert's Bioawk tutorial (on GitHub).

13. Bioawk - examples - SAM files
Extract mapped reads:
sambamba view x.bam |
bioawk -c sam '!and($flag,4)'
Adapted from Istvan Albert's Bioawk tutorial (on GitHub).

14. Bioawk - examples - VCF files
bioawk -c vcf '{
freq[$filter]++
total++
}
END {
for(val in freq)
printf "%st%dt%dn",
val, freq[val], freq[val]*100/total
}'
From sahilseth's flowr Bioawk tips, itself adapted from Stephen Turner's "Bioinformatics one-liners".
Assess pipeline—
sequence filter statistics:
• filter (e.g. LowQual)
• number of filter
occurrences
• percentage of total filters

15. Bioawk - examples - VCF files
VCF data: Erik Garrison's vcflib, sample.vcf.
PASS 5 55
q10 1 11
. 3 33

16. Bioawk: list of added functions
• gc($seq)
• meanqual($seq)
• reverse($seq) / revcomp($seq)
• qualcount($qual, threshold)
• Number of quality values above the threshold parameter.
• trimq(qual, beg, end, param=0.05)
• Trims using Richard Mott's algorithm (used in Phred).
• Bitwise AND/OR/XOR