The document outlines Ben Busby's presentation on using NCBI tools and databases for metagenomics. It discusses searching databases like PubMed and SRA using tools like EDirect, BLAST, and taxonomy. It also covers computational resources at NCBI including sequence analysis, contig generation, and the EUtilities API. Finally, it advertises upcoming NCBI hackathons for hands-on training in using NCBI bioinformatics resources and tools.

1. Ben Busby, Ph.D.
NCBI Genomics Outreach Coordinator
Chair, Department of Bioinformatics and Data Science, FAES
ben.busby@nih.gov

2.  Review of basic NCBI search systems, tools, databases for
metagenomics
 NCBI Hackathons
 Hands-on Session
 Finding literature and molecular data for metagenomics (EDirect)
 Basic Computational resources at NCBI
 Taxonomy
 MoleBLAST
 Many other flavors of BLAST
 SRA-BLAST
 MagicBLAST
 Contig generation (with Spades)
2

5. 5
EUtils (Search API) Command Line
EDirect

6. 6
PubMed and PMC (Open) FTP

7. 7
Instead of PubMed FTP…
An automated tool (alpha)

13. 13

14. 14

35. Combined score is
the average of SVs,
mappability, GC..
NCBI region list
Encode blacklist

39. phenvar.colorado.edu

40. 40
EDirect (Search API) Cookbook

42. My View of Data Transfer Principles
• Metadata Search
• Rapid NoSQL (for now)
• Integration
• Non-ambiguous identifiers
• Transferring Small amounts of Data
• Data still gets transferred in the cloud
• Underlying structure
• Finding specific data from validated formats
• Democratization of Data
• Rapid comparison by domain experts
• Reporting
• Metrics to report data upload and [unique IP] download of datasets
• Post-publication User Review

43. 43

44. • August 14-16, NIH Campus
• September 25-27, Pittsburgh, PA
• January 8-10, NIH Campus
• February 19-21, Baylor College of Medicine
• March 11-14, NIH Campus
• April 2-4, 2018, UCSC
• May 15-17, 2018, Ann Arbor, MI
• June 20-22, Boulder CO
ncbi-hackathons.github.io
Come work at NCBI for 4-6 weeks!
Email bioinformatics-training@ncbi.nlm.nih.gov
for more information!

46. Thanks again to Peter Cooper!
Materials: http://bit.ly/2chunww

47. Learn the basics of using the EDirect programs to extract
custom reports from the NLM/NCBI Literature and
molecular databases

48. YouTube:
https://youtu.be/BLnYW33Mtb0
YouTube:
https://youtu.be/aK7-KX5q30k
Past webinar recordings on
YouTube (youtube.com/ncbinlm)

49.  Command line version of E-Utilities (API for Entrez system)
 Each program outputs XML that can be piped directly into another
EDirect program or Linux utility
 Requirements
 UNIX, Linux, Mac OSX, CygWin (Windows)
 Perl with LWP::Simple
 Package contents
 einfo
 esearch
 esummary
 efetch
 elink
 epost
 efilter (performs an esearch after an elink or esearch)
 xtract (powerful XML parser)
Documentation
<ncbi>/books/NBK179288/#chapter6

50.  Allows construction of custom pipelines for processing
data
 Generates highly flexible custom output reports
 Built in batch access
09/12/2016 50

51. esearch efetch
xtract
Search
Retrieval
Parsing and
reporting
epost
elink
History
Searching
Linking
Storing and combining

52. Provides information about the available databases
 Available indexed fields
 Available links
 Produces XML (or text output with –fields, -links)
09/12/2016
einfo –dbs
einfo –db dbname
einfo –db dbname –fields
einfo –db dbname -links

53.  Uses standard web Entrez queries
 try searches on web interface first
 Results stored in web environment
 Pipe output to efetch, elink
esearch -db pubmed -query 'antibiotic resistance[MeSH] AND metagenomic'
<ENTREZ_DIRECT>
<Db>pubmed</Db>
<WebEnv>NCID_1_55733393_130.14.22.215_9001_1473536392_1519353658_0MetA0_S_MegaS
tore_F_1</WebEnv>
<QueryKey>1</QueryKey>
<Count>165</Count>
<Step>1</Step>
</ENTREZ_DIRECT>

54.  Returns related records in the same (-related) or different
(-target) database
 Use link name from einfo to get the most precise results (-
name linkname)
 Pipe into efetch
 Use with –cmd neighbor to get a table of linked identifiers
(elink XML)

55.  Produces full XML records and Summaries (Docsums) for
many databases
 Also specialized output for PubMed, sequence databases,
Gene and others
 In many cases Docsums contain enough information
(efetch –format docsum == esummary)
 Parsing values from full XML can be more challenging

56. -format -mode Report Type
acc Accession Number
est EST Report
fasta FASTA
fasta xml TinySeq XML
fasta_cds_aa FASTA of CDS Products
fasta_cds_na FASTA of Coding Regions
ft Feature Table
gb GenBank Flatfile
gb xml GBSet XML
gbc xml INSDSet XML
gbwithparts GenBank with Contig Sequences
gene_fasta FASTA of Gene
gp GenPept Flatfile
gp xml GBSet XML
gpc xml INSDSet XML
gss GSS Report
ipg Identical Protein Report
ipg xml IPGReportSet XML
native text Seq-entry ASN.1
native xml Bioseq-set XML
seqid Seq-id ASN.1

57.  General Full-featured XML parser
 Produces tab delimited output
 Loops over XML structure using exploration options
 Prints out selected items (elements) from XML
 Conditional execution
 Flexible output formats
57

58.  -pattern places the data from individual records into
separate rows.
 -element extracts values from specified fields into
separate columns.
 -group, -block, and -subset limit element exploration to
selected XML subregions.
58

59. 59
<Result Set>
<DocumentSummary>
<Tag>value</Tag>
<Tag>
<Tag>
<Tag>valu
<Tag>valu
<Tag>valu
</Tag>
</Tag>
<Tag>
<Tag>
<Tag>valu
<Tag>valu
<Tag>valu
</Tag>
</Tag>
</DocumentSummary
<DocumentSummary>
etc.
</Result Set>
Pattern (Record)
Group
Block
Group Block
-pattern
-group
-block
-subset

60. EDirect Example 1:
Explore articles and authors for antibiotic resistance
genes in metagenomic samples
09/12/2016 60

61. 09/12/2016 61
• Loop over DocumentSummaries
Table of Articles
• Print out PMID, Source, Title
• Only match <Source>PLoS One</Source>
-match “Source:PLoS One”
Author counts
• Report and count Author names
-element Name
• Use built in sort-uniq-count-rank to process final output

62.  -sep
 specifies the character separating multiple fields in an ‘-
element’ argument
 -tab
 specifies the character separating multiple values of an ‘-
element’ argument
 Place –sep and –tab before -element
09/12/2016 62

63. 63
ESearch set of PMIDStext query
ESummary set of PMIDSPubMed Docsums
pubmed
PubMed
xtract
match Source:PLoS One
ID,Source,Title
Name
09/12/2016
sort-
uniq-
rank-
count

64. EDirect Example 2:
Retrieve the correct sequence in FASTA format from
the nuccore database
09/12/2016 64

65. 09/12/2016 65
• efetch in nuccore can be used with –chr_start and –
chr_stop to adjust from 0-based to 1-based
coordinates and strand.
• You can use xargs to pass the last three columns of
the test file to efetch to get all genomic regions

77.  Originally published in 2009
(doi:10.1128/AEM.01541-09)
 Most cited tool for analyzing 16S
rRNA gene sequences
 3,410 citations (WoS: 1/8/2016)
 Working on 37th release
 Overview
 100% open source, GPL v3
 OS independent
 Command line interface
 Written in C/C++
http://www.mothur.org

78.  Deposition of 16S rRNA gene sequences to the SRA has
been a major problem
 Worked with SRA staff to make a customized portal to
simplify submission of PCR-generated 16S rRNA gene
sequences
 Command enforces co-submission of sample and
processing metadata
 Originally released in March 2015. So far 86 submissions,
61 studies, 6367 runs, 116 Gbp total
http://www.mothur.org/wiki/make.sra

79. 1. Provide the necessary MIMARKS* metadata data about
samples with get.mimarkspackage
2. Create a project file describing user and their project
using supplied template file
3. Parse MIMARKS, project file, and sff or fastq files to
generate an xml file for submission using make.sra
4. Email the SRA to let them know about submission using
mothur created files and await further instructions
* http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3367316/
http://www.mothur.org/wiki/Creating_a_new_submission

80.  MIMARKS: minimum information about a marker gene
sequence (doi:10.1038/nbt.1823)
 Command supplies user with a blank text file with sample
names and necessary metadata for their environment.
User fills in details.
 For each environmental package and environment there
is a wiki page with required and optional parameters and
allowed values. Can also be extended to include
additional metadata
http://www.mothur.org/wiki/Get.mimarkspackage
http://www.mothur.org/wiki/Human_gut

81. USERNAME [UserName]
Last [LastName]
First [FirstName]
EMAIL [Email@mail.com]
CENTER [University or Center Name]
TYPE institute
WEBSITE [www.Website.org]
ProjectName [ProjectName]
ProjectTitle [Project Title]
Description [Project Description]
Grant id=[GrantID], agency=[GrantAgency],
title=[GrantTitle]
User completes information in brackets
http://www.mothur.org/wiki/Project_File

83. 84

84. 85

85. 86

86. 87

87. 88