White Paper: Next-Generation Genome Sequencing Using EMC Isilon Scale-Out NAS: Sizing and Performance Guidelines


Published on

This EMC Isilon sizing and performance guideline White Paper reviews the Key Performance Indicators (KPIs) that most strongly impact the production processes for the storage of data from Next-Generation Sequencing (NGS) workflows.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

White Paper: Next-Generation Genome Sequencing Using EMC Isilon Scale-Out NAS: Sizing and Performance Guidelines

  1. 1. White PaperNEXT-GENERATION GENOME SEQUENCINGUSING EMC ISILON SCALE-OUT NAS: SIZINGAND PERFORMANCE GUIDELINES Abstract This EMC Isilon Sizing and Performance Guideline white paper reviews the Key Performance Indicators (KPIs) that most strongly impact the production processes for Next-Generation Sequencing (NGS) workflows. August 2012
  2. 2. Copyright © 2012 EMC Corporation. All Rights Reserved.EMC believes the information in this publication is accurate asof its publication date. The information is subject to changewithout notice.The information in this publication is provided ―as is.‖ EMCCorporation makes no representations or warranties of any kindwith respect to the information in this publication, and specificallydisclaims implied warranties of merchantability or fitness for aparticular purpose.Use, copying, and distribution of any EMC software described inthis publication requires an applicable software license.For the most up-to-date listing of EMC product names, seeEMC Corporation Trademarks on EMC.com.All other trademarks used herein are the property of theirrespective owners.Part Number H19061Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 2
  3. 3. Table of ContentsExecutive summary............................................................................... 4Introduction ........................................................................................ 4NGS workflow – sequencing instruments and file types ............................. 5NGS workflow – HPC ............................................................................. 7NGS workflow – Isilon scale-out NAS .....................................................10EMC Isilon scale-out NAS overview ........................................................12 Simple .............................................................................................. 12 Scalable............................................................................................ 13 Predictable ........................................................................................ 13 Efficient ............................................................................................ 13 Available ........................................................................................... 14 Enterprise-ready ................................................................................. 14NGS: key performance indicators ...........................................................17 HPC server parameters ......................................................................... 17 Network infrastructure parameters........................................................... 18 Isilon storage configuration parameters .................................................... 18 Summary .......................................................................................... 19Conclusion ..........................................................................................20 Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 3
  4. 4. Executive summaryNext-generation sequencing (NGS) workflows are comprised of genome sequencerinstrumentation, high-performance computing (HPC) infrastructure, a network-attached storage (NAS) platform, and the network infrastructure connecting thesecomponents together.Raw NGS data is the largest component of an NGS process, making data storagecapacity and scalability important factors in NGS performance. The raw TIFF imagefrom the sequencer can be up to 70 percent of the total dataset. These files may becompressed and stored for later use. Most organizations do not save the TIFF images,but retain either the BCL or FASTQ files as the raw files. Each sequencing run can alsogenerate analysis data in the range of 50-200 GB. With faster sequencers and largerread lengths, this can add up to between approximately 1 PB and 2 PB per year for afacility with three NGS sequencers.Beyond capacity scalability, I/O performance is also a critical file storage attribute foroverall NGS performance and efficiency. NGS is I/O bound rather than processor bound,and therefore storage I/O performance has a high impact on overall NGS performancein relation to other NGS workflow parameters.Internal EMC testing has determined that the Key Performance Indicators (KPIs) thatmost affect the performance of NGS applications are: Total random access memory (RAM) size on HPC cluster nodes (recommended at 3 GB/core) RAM and SSD allocation on the EMC® Isilon® storage cluster – place maximum allowable RAM on the performance layer and minimum recommended on the archival layer with about 1 percent to 2 percent of the raw storage capacity as SSD Storage configuration parameters: NFS version V4, NFS async enabled, TCP MTU (jumbo frames), LACP (2x 1 Gb/s or 4x 1 Gb/s), and tuning the Grid Engine packageIntroductionOver the past five years, the precision and effectiveness of sequencing technologyhas considerably increased the pace of biological research and discovery. The resourcesfocused on molecular biology, cellular biology, and bioinformatics continue to accelerateat a significant pace. Projections indicate that before the end of the 21st century, wecould gain a full understanding of the workings of our DNA. Such knowledge couldallow us to improve our collective quality of life through a better understanding of howa specific genetic variation impacts a drug’s efficacy or toxicity, or, by possibly providingthe knowledge to eradicate a range of genetically based disorders.DNA exome sequencing is an approach to selectively sequence the coding regions ofthe genome as an easier yet still effective alternative to whole genome sequencing.The exome of the human genome is formed by exons. Exons are short, functionally Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 4
  5. 5. important coding sequences of DNA within the gene’s mature messenger RNA that constitute about 1.5 percent of the human genome.1 Many large-scale exome sequencing projects are underway to analyze human diseases. This technology is often the choice as it is more affordable than whole genome sequencing (WGS) and therefore allows analyzing more patients. In addition, it has the advantage that resulting data volumes are much smaller and therefore easier to handle. However, recent studies2 focused on this question found that both technologies complement each other. As neither the whole genome nor the large- scale exome sequencing technologies cover all sequencing variants, it is optimal to conduct both experiments in parallel. A single human genome—composed of a total of about 3.2 billion base pairs—requires about 1.2 GB of unassembled storage. Industry analysts predict that the estimated number of human whole genomes sequenced will explode from 25,000 genomes in 2012, to between 50,000 and 100,000 in 2013, and up to about 1 million by 2015. The key enabling technology for NGS are the many commercial sequencers available from various companies, including Illumina, Life Technologies, Roche/454 Life Sciences , and others. These sequencers interface to a computer network, which correlates and concatenates the billions of overlapping segments of DNA sequence short reads that have been streamed to or stored on a NAS system. Accommodating the output rate of the sequencers requires a precisely designed and balanced system. The peak rate of data (base pairs) produced by an Illumina sequencer, for example, is already approaching 600 GigaBases per week, equivalent to about 100 whole human genomes. The range of data per year for an Illumina sequencer is from 350 TB to 1 PB. The components of the NGS workflow are comprised of:  Genome Sequencer instruments,  HPC infrastructure,  NAS platform, and the  Network infrastructure stitching these components together. These four components make up the hierarchy of the NGS gene-sequencing architecture. Each component depends on the other and must have the ability to adapt and scale to meet current and future sequencing needs. If one component creates a bottleneck, then the performance of the entire NGS system suffers. The focus of this document endeavors are optimum performance and sizing guidelines for the core components of NGS: the HPC infrastructure and network-attached storage. NGS workflow – sequencing instruments and file types The applications at the heart of NGS data creation come from important established and emerging organizations involved in bringing NGS to market. The list includes software from Illumina ®, Life Technologies (Applied Biosystems), Roche/454, Ion Torrent,1 See Gilbe rt W (February 1978). ―Why genes in pie ce s?‖. Nature 271 (5645): 501.2 See Performance comparison of exome DNA sequencing te chnologie s. Clark MJ, C hen R , Lam HY, Karcze wsk i KJ, Che n R , Eusk irchen G, Butte AJ, Snyde r M, Department of Ge ne tics, Stanford Unive rsity School of Medicine , Stanford, C A, USA. Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 5
  6. 6. Pacific Biosciences, and a myriad of open source offerings such as Galaxy. Runningthese applications in a research and analysis environment places complex and specialrequirements on the IT systems, and in particular, the storage infrastructure. Thisdocument will focus on the Illumina technologies—specifically CASAVA™ for thesequence analysis software. Other Genome Assembly and Analysis Platforms likeGalaxy would be summarized in subsequent documents.An NGS environment typically consists of Scientific, Lab, and Analysis users: The Scientific User initiates the method of genome sequencing and instrumentation. This may also be the Analysis User. The Lab User runs the experiment (chemistry workflow) using a multiplexed sampling scheme (or lanes) supported by the NGS instrument. The Analysis User works on the results from the genome sequencing study with bioinformatics tools and algorithms.Most commercial NGS data centers also have a trained storage administrator on theirstaff. With the growing use of NGS technologies, a new user has emerged for thesestorage systems. The scientist or researcher running the experiments frequentlyhandles the data directly. Data management has to be intuitive to allow this new userto run experiments and administer the data with minimal difficulty. In addition, thestorage administrator needs access to the more advanced management features toset sophisticated management policies. These help with optimization of performanceand use of storage system. It is important that the storage system deployed providemanagement capabilities tuned to both types of users.A graphical representation of the typical NGS data flow is shown below in Figure 1:Figure 1. NGS architecture, data flow, and file typesThe results stage of the NGS workflow as shown in Figure 1 consists of a number ofsuccessive steps each involving file conversions and each resulting in approximately5x smaller file sizes. These steps include conversion of the raw image file into base-call data, then of base call data into FASTQ text-based file format for storing both Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 6
  7. 7. biological sequence and its corresponding quality scores, for example, using LQUAL or QUAL formats. This is followed by conversion into BAM (Binary Alignment Map) file data followed by conversion into Variant Calls (VCF) file data, which is converted next into results data in SRA format. This tertiary file data is typically kept forever, needs to be kept safe, as well as available, and accumulates over time. Today’s instruments produce higher level information and may avoid some of the intermediate steps, thus reducing output data compared to previous NGS systems. Therefore, data flows generated by the latest NGS instruments have typically decreased in size per run. This dec rease has been offset by a larger number of experiments, secondary data, and increased consumption by users working downstream on many different efforts and workflows. The size and characteristics of data produced from these efforts place unpredictable demands on capacity as well as on throughput of the storage systems. NGS storage environments need to be able to adapt to demands for more capacity from post-processing work done by researchers downstream from the first data capture. NGS workflow – HPC NGS applications have both common and unique analysis tools. All applications generate large files that must be managed through multiple rounds of processing. Although many tools were written specifically for easy implementation on a high-end desktop computer (e.g., 64-bit dual- or quad-core, 16 GB RAM), routine analysis is typically conducted on high-performance compute clusters. Using a high-performance compute cluster, secondary analysis processing can generally be done at a rate equal to or faster than primary data generation. Due to the open- ended nature of tertiary analysis, a similar rate estimate cannot be precisely stated. It is important that the parallelization of the NGS analysis platform be well understood before planning on optimum server CPU core sizing. Most of the NGS tools are at least multi-processor aware or are highly parallelized by simply dividing the sequence data, the assembly algorithm, variant calling, or all, and starting separate analysis on these data subsets. For NGS applications, the current parallelization per process is typically between 75 percent and 90 percent. As genomics has very large, semi-structured, file-based data and is modeled on post- process streaming data access and I/O patterns that can be parallelized, it is ideally suited for the Hadoop software framework3 which consists of two main components: a file system and a compute system—the Hadoop Distributed File System (HDFS) and the MapReduce framework, respectively.3 See Hadoop in the life scie nces: an Isilon Systems white pape r. Joshi S. Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 7
  8. 8. Figure 2. Amdahl’s Law and parallelization One of the basic tenets in HPC, Amdahl’s Law4, postulates that adding more microprocessor cores to a process does not speed it up linearly. A 64 core HPC platform is estimated to be the performance threshold for 75 percent parallelization per NGS process, which delivers a speedup of 4x (see Figure 2). Even more than 100 cores per active NGS process do not speed up the process substantially when the algorithm(s) are between 75 percent and 90 percent parallelization. During actual testing of the NGS processes in the range of 75-90 percent parallelization, the speedup from 12 cores to 72 cores was found to be only about 1.25x. Horizontal platforms like Hadoop that combine compute and data in a parallel context would benefit genome assembly considerably.4 See ―Validity of the single proce ssor approach to achie ving large -scale computing capabilities‖, Amdahl G, AFIPS Conference Proceedings (30): 483–485, 1967 Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 8
  9. 9. Figure 3. Performance curves for NGS using Illumina CASAVAAs shown in Figure 3 above, the NGS process is storage I/O and memory bound. Theperformance curves show a direct relationship between NGS performance and saturationof read/write I/O and memory functionality. In contrast, there is an inverse relationshipbetween the CPU core utilization and storage I/O and memory functionality. This numbermay be due to mutual dependencies or portions of the process that can only be performedsequentially; NGS algorithms requiring movement of large amounts of data in and outof the CPU; startup overhead including base calling and other large numbers of smallfile writes; and degree of serialization involved in communication.In view of the above discussion, it is recommended that the HPC server hardwareplatform be configured with: Best I/O chipset, for example, using the latest generation Intel I/O controllers Highest DRAM speed (with a minimum of 3 GB per core of RAM) Multi-core CPU set with > 2 GHz processors Simplified BIOS and driver upgrades with a single management console for all driver upgrades Linux driver compatibility (over 90 percent of all HPC systems are Linux-based) Disk drives between 200 GB and 600 GB with RAID 10 Cluster management tools such as GangliaIncreasing the network bandwidth up to 4 Gbps would alleviate the read I/O andmemory saturation. Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 9
  10. 10. NGS workflow – Isilon scale-out NASFigure 4. Data flow using Illumina NGS processNGS production processes generate potentially millions of files with terabytes ofaggregate storage impacting the capacity and manageability limits of existing fileserver structures.Figure 4 shows the data flow including a file number and capacity summary of an actualNGS process using Illumina sequencer and Isilon scale-out NAS storage. As can be seen,the process generates over 500,000 having aggregate size of greater than 5 TB overthe course of the 48-hour run.Raw NGS data is the largest component of an NGS process. The raw TIFF image can beup to 70 percent of the total dataset. These files may be compressed and stored forlater use. Most organizations do not save the TIFF images, but retain either the BCLor FASTQ files. If sequencing as a service is used, the input to the process is a BAMfile. Each sequencing run can also generate intermediate and final analysis data in the Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 10
  11. 11. range of 50-200 GB. With faster sequencers and larger read lengths, this can add up tobetween 1 PB and 2 PB per year for a facility with three NGS sequencers.Genomics is a data reduction process from the raw instrument information (images orvoltages) to the variants. This reduction process follows the ―Rule of One-Fifth‖ asshown in the sizing table below: File format Size, GB Illumina Size, GB IonTorrent Comments TIFF, 2500 750 TIFF Range: 2.5 to 4 TB, Ion WELLS Torrent™ is WELLS voltage format BCL / SFF 500 500 Ion Torrent uses SFF BA M 100 100 2x compression (~200 GB normal) VCF 20 20 Variant calls SRA, EMR 4 4 EMR (Electronic Medical Record) includes Radiology and Pathology imagesTable 1. Data reduction for the NGS process; human whole genome, all filesizes are approximateRaw instrument data typically consists of large image files (2-5 TB per run are the norm),usually in TIFF format or an electropherogram file format native to a sequencer (forexample, the SEQ format native to the Illumina sequencer). These files are only keptlong enough (7-10 days) to verify that the experiment worked. The image file for theexperiment is usually the largest file size in NGS.Intermediate or secondary data consists of raw data processed into information ofincreasing value, stored for medium- to long-term storage (1 year or more), requireshigh bandwidth access for fast analysis, and is expensive to re-create, so storage needsto be highly available. These include files in BCL format for base calling and conversionwith an aggregate ratio of approximately one-fifth compared to raw instrument data.Beyond capacity scalability, I/O performance is also a critical file storage attribute foroverall NGS performance and efficiency. As discussed earlier, NGS is I/O bound, ratherthan processor bound, and thus storage I/O performance has a high impact on overallNGS performance in relation to other NGS workflow parameters. As a result, NGSenvironments require a file storage infrastructure that is purpose-built to address thecapacity and performance scalability, efficiency, availability, and manageability challengesof next-generation NGS environments. Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 11
  12. 12. EMC Isilon scale-out NAS overviewNGS is an unstructured file-based process, not a block-based storage process. The EMCIsilon scale-out NAS manages unstructured file data through a single namespace throughits storage appliance nodes arranged in clusters which support massive scalability.A short description of the EMC Isilon storage solution and the EMC Isilon OneFS® fileoperating system with each of its features summarized below confirms its suitabilityfor next-generation genomic sequencing:SimpleOneFS combines the three layers of traditional storage architectures—the file system,volume manager, and RAID/data protection—into one unified software layer, creatinga single intelligent distributed file system that runs on an Isilon storage cluster.Figure 5: OneFS eliminates the need for complex file managementThis scale-out hardware provides the appliance on which OneFS distributed file systemresides. A single EMC Isilon cluster consists of multiple storage nodes, which are rack-mountable enterprise appliances containing memory, CPU, networking, NVRAM, storagemedia and the InfiniBand back-end network that connects the nodes together. Hardwarecomponents are best-of-breed and benefit from ever-improving cost and efficiencycurves. OneFS allows nodes to be added or removed from the cluster at will and at anytime, abstracting the data and applications away from the hardware. Adding nodes —instead of adding volumes and LUNS via physical disks—becomes an extremely simpletask at the petabyte (PB) scale, which is common in NGS. Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 12
  13. 13. ScalableFigure 6: Linear scalability with OneFSEMC Isilon provides a high-performance, fully symmetric cluster-based distributedstorage platform. It has linear scalability with increasing capacity—from 18 TB to15.5 PB in a single filesystem—as compared to traditional storage. The concept ofnode-based capacity growth with linear scaling is critical to NGS where scale needs tobe painless, since the process can generate upwards of 8 TB per week per instrument.The researchers and clinicians need to focus on managing scientific data and patients,not managing storage.PredictableAlong with raw scaling of capacity, balancing of the content across the new nodes needsto be predictable for an NGS workflow due to its sustained throughput requirement.Since the instrument end keeps changing with newer technologies faster than theHPC or Storage, this balancing and scale become invaluable. Dynamic contentbalancing is performed as nodes are added or data capacity changes. There is noadded management time for the administrator, or increased complexity within thestorage system. The storage reporting application, InsightIQ, can be used to plan thegrowth of a system from storage statistics bot h for infrastructure and for budgeting.EfficientOperational Expenditure (Opex) hinges upon efficiency, specifically in NGS, since thetotal storage can run into PBs. A recent survey conducted by Scripps Institute concludedthat more than 35 percent of institutions today are at petabyte scale in NGS with a10 percent year-over-year growth.Isilon scale-out NAS offers an 80 percent efficiency ratio and ―smart pooling‖ of thedata across multiple tiers, making dynamic, rule-based data transfer between storagepools an integral piece of the NGS process. This efficiency is at the application leveland tiered by the performance types: S-Series node for high performance (I/O per second) X-Series node for high throughput NL-Series node for archive Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 13
  14. 14. Figure 7: Storage tiering based on node typeThe tiers in the storage cluster as shown in Figure 7 above are identified as ―pools‖and managed by the EMC Isilon SmartPools™ application. A pool is a group of similarnodes which is defined by the user and is based on the functionality or workflow. A poolis governed by policies which can be changed based on needs; default policies arebuilt in. Policies defined by any standard file metadata: file type, size, name, location,owner, age, last accessed, etc. Data can be migrated from pool to pool. The timing forthis data movement is configurable: default is 1x/day @ 10 PM.AvailableData availability and redundancy are the core requirements of the scientific and clinicalstaff in NGS. As NGS moves into the clinical realm, availability becomes even moreimportant. Flexible data protection occurs during power loss, node or disk failures,loss of quorum, and storage rebuild. OneFS avoids the use of hot spare drives, andsimply borrows from the available free space in the system in order to recover fromfailures; this technique is called virtual hot spare.Since all data, metadata, and parity information is distributed across the nodes of thecluster, the Isilon cluster does not require a dedicated parity node or drive, or a dedicateddevice or set of devices to manage metadata. This helps to ensure that no one nodecan become a single point of failure and makes the cluster ―self-healing.‖Enterprise-readyThe NGS data system does not exist as an island; it usually coexists with other storageand IT systems. The standard protocols that OneFS supports build the standards-basedprotocol bridges to other information systems from NGS. Specifically, connectivity to theIsilon scale-out NAS cluster is via standard file protocols: CIFS, SMB, NFS, FTP/HTTP,iSCSI, and HDFS. The complete data lifecycle is accessible to the centralized IT group.Snapshots, Replication, and Quotas are supported via a simple web-based UI.Data is given infinite longevity and future-proofs the enterprise from evolving hardwaregenerations—eliminating the cost and pain of data migrations and hardware refreshes. Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 14
  15. 15. Active Directory (AD), LDAP, NIS and local users are standardized authentication andaccess control available at scale. Simultaneous or rolling upgrades to OneFS are possible,with little or no impact to the production environment.Figure 8: Standard protocols are critical to enterprisesThe software to manage OneFS is automated to eliminate complexity, as shown inFigure 9 below:Figure 9: OneFS software management suiteAll of the applications shown above are available as software licenses and are web-basedthrough the main administrative user interface. A comprehensive command-line basedadministration interface is also available. Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 15
  16. 16. OneFS Software Management Suite Making data management easier for NGS OneFS infrastructure software solutions meet critical data protection, access, management, and availability needs Application Category What it does SmartPools Resource Implements a highly efficient, automated management tie red storage strategy to optim ize storage pe rformance and costs SmartConnect Data access Enables client conne ction load balancing and dynamic NFS failove r and failback of client conne ctions across storage nodes to optimize use of cluste r resources SnapshotIQ Data prote ction Prote cts data efficiently and re liably with se cure, near-instantaneous snapshots while incurring little to no pe rformance ove rhead InsightIQ Pe rformance Max im izes pe rformance of your management Isilon scale-out storage system with innovative pe rformance -monitoring and re porting tools SmartQuotas Data Assigns and manages quotas that management partition storage into easily managed segments at the cluste r, dire ctory, sub-dire ctory, use r, and group le ve ls SyncIQ Data re plication Replicates and distributes large , m ission - critical data se ts to multiple shared storage systems in m ultiple sites for re liable disaste r re cove ry capabilityTable 2. Functional overview of the OneFS software suite Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 16
  17. 17. NGS: key performance indicators As discussed in the HPC section, performance of the NGS process is highly dependent on the I/O performance of the file storage system and memory resources available in the NGS architecture. In addition, there are a range of second-order factors that need to be considered in terms of optimizing performance for a specific NGS process, including5:  How much faster can a given problem be solved with multiple workers (or server cores) instead of one?  How much more work can be done with multiple workers (or server cores) instead of one?  What impact do the communication requirements of the parallel NGS application have on overall performance and scalability?  What fraction of the resources in an NGS configuration is actually used productively for solving the NGS problem? The KPI for NGS consists of factors that can be used to predict and optimize the performance of an NGS configuration and can be broken down into four categories:  HPC server attributes: RAID, number of processor cores per HPC node, total RAM size per HPC node  NGS network infrastructure: TCP MTU, Channel Bonding, DNS  Sun Grid Engine parameters: number of nodes, PAR_EXECD_INST_COUNT  Isilon file storage attributes:  SSD size and RAM  NFS protocol parameters: NFS server OS, async, number of threads, locks  Software RAID, maximum number of directories at a level, maximum number of files in a directory, number of files less than 8 KB HPC server parameters RAID: With modern multi-core CPUs, the performance of software RAID is very close to that of hardware RAID. RAID 10 (first mirroring, then striping the mirrors) is recommended for the HPC nodes with a minimum of two identical drives per node where both drives are bootable. The benefit of such a configuration is that the server continues to boot seamlessly even in the face of a failure of a single drive. Total processor cores: The empirical rule-of-thumb for total number of threads and processes running in parallel is determined by the equation: ∑ (threads + processes) = (2 x total cores) + 1 Please note that the total number of threads include functions such as NFS and HPC queuing as well as the processes that run NGS algorithms; it is important to document all the processes that are multi-threaded. Amdahl’s law vis-à-vis parallelization is also an important consideration.5 See Introduction to High Performance Computing for Scientists and Engineers , Hage r G, Welle in G, © 2010 Taylor & Francis Group, LLC Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 17
  18. 18. Total RAM size: NGS analysis requires large file processing, including functionsrelated to string processing, clustering of large files, and statistical quality measures,and thus easily becomes memory-bound. As a result, a large DDR3-based RAM poolis optimal.Network infrastructure parametersTCP MTU: The default Maximum Transmission Unit (MTU) (or frame size) of currentEthernet systems is 1500 B. However, higher bandwidth network infrastructures canhandle a much higher MTU of 9000 B (called ―jumbo frames‖) for efficient data transfer.Please note that the jumbo frame setting needs to be completed both on the HPC servernode(s) and the switch(es).Ethernet Bonding (LACP): Ethernet Bonding using the Link Aggregation ControlProtocol (LACP) is a method used to alleviate bandwidth limitations and port-cable-port failure issues. By combining several Ethernet interfaces to a virtual ―bond‖ interface,the network bandwidth can be increased since LACP splits the communications andsends frames among all the Ethernet links. Bonding 2x 1 GbE interfaces provides therequired bandwidth between HPC server nodes and NAS file storage.Isilon storage configuration parametersNFS Master OS: By default, EMC Isilon OneFS operating system is the NFS server. Itis recommended that this default be maintained since SmartConnect and other OneFSfeatures may be affected if the HPC master node OS is chosen as the NFS server.NFS V4: NFS V4 provides improved performance, security, and robustness vis-à-visNFS V3. These include support of multiple operations per RPC operation (vs. a singleoperation per RPC in NFS V3), use of Kerberos and access control lists (ACLs) forsecurity (vs. UNIX file permissions in NFS V3), use of TCP transport (vs. UDP in NFSV3), and integrated file locking (vs. use of the adjunct Network Lock Manager protocolfor NFS V3). As a result, it is recommended that sites utilize NFS V4 for the NGSenvironments. Please note that initial setting up of NFSv4 can be cumbersome.NFS async: The NFS async (asynchronous) mode allows the server to reply to clientrequests as soon as it has processed the request and handed it off to the local filesystem, without waiting for the data to be written to stable storage. However, writeperformance is better when synchronous mode is used (also called ‘noasync’), especiallyfor smaller file sizes. This is the recommended mode, especially since NFSv4 usesTCP connectivity.NFS number of threads: This is the number of NFS server daemon threads that arestarted when the system boots. T he OneFS NFS server usually has 16 threads as itsdefault setting; this value can be changed via the Command Line Interface (CLI):isi_sysctl_cluster sysctl vfs.nfsrv.rpc.[minthreads,maxthreads]Increasing the number of NFS daemon threads improves response minimally; themaximum number of NFS threads need to be limited to 64.NFS ACL: The NFS ACL (Access Control List) for NFSv4 is a list of permissions associatedwith a set of files or directories which contain one or more Access Control Entries (ACEs).There are four types of ACEs: Allow, Deny, Audit, and Alarm; with three kinds offlags: group, inheritance, and administrative. There are 13 file permissions and Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 18
  19. 19. 14 directory permissions. OneFS manages NFS ACLs which need to be mapped to the NFS client using the idmapd configuration. NFS locks: The mounting and locking processes have been enhanced in NFSv4 which supports mandatory as well as advisory locking. Caching and open delegation provide performance improvements in most situations. More information about state is stored on the servers in the HPC tier, enabling recovery of the files when they are in use. 6,7 Maximum number of directories at a level and files within a directory : While Isilon OneFS supports an upper bound of 100,000 files in a directory as well as number of directories at a level, in order to ensure highest performance while traversing a directory tree, the maximum number of directories at a level and the maximum number of files within a directory needs to be below 10,000. Number of small (<8 KB) files: Random-write operations on small files have low response times and can degrade overall application performance. In order to optimize performance, it is recommended that Base Call files that are typically <8 KB be aggregated into 128 KB or larger ZIP archive files. SGE number of nodes: The Sun Grid Engine (SGE) package is a popular distributed resource manager (DRM) and scheduler package for controlling access to and control of cluster resources. It is recommended that at least a minimum of three SGE nodes be used for NGS for performance and backup reasons. While a commercial version of SGE available from Oracle, SGE is also available as open source. Other popular open source DRM packages are Torque/Maui and Lava. Execution daemons: The SGE PAR_EXECD_INST_COUNT variable contained within the SGE configuration file defines the number of parallel execd (execution daemons) for the NGS HPC cluster. DNS location: If the HPC NGS system is run within a private network, it is recommended that Linux BIND be installed on the HPC master node with DNS forwarding to the organization’s DNS server. Summary Internal EMC testing determined that the KPIs that affect the performance the most are:  RAM on HPC cluster server nodes (recommended at 3 GB/core)  RAM and SSD on the Isilon storage cluster—maximum allowable RAM on the performance layer and minimum recommended on the archival layer with about 1 percent to 2 percent of the raw storage capacity as SSD  Storage configuration parameters: NFS version V4, NFS async enabled, TCP MTU (jumbo frames), LACP and the Grid Engine package6 See info on Isilon SmartLock : http://www.em c.com /collate ral/software /white-pape rs/h8325-wp-isd-smartlock.pdf7 See info on Isilon high-pe rformance computing: http://www.isilon.com/high-pe rformance-computing Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 19
  20. 20. ConclusionNGS production processes generate potentially millions of files with terabytes ofaggregate storage impacting the capacity and manageability limits of existing fileserver structures. Raw instrument data typically consists of large image files (2-5 TBper run are the norm), usually in TIFF format. The image file for the experiment isusually the largest file size in NGS.Genomics is a data reduction process from the raw instrument information (images orvoltages) to the variants which follows the ―Rule of One-Fifth.‖ Intermediate orsecondary data consists of raw data files including files in BCL format for base calling andconversion have an aggregate ratio of approximately one-fifth compared to rawinstrument data.Internal EMC testing has determined that the KPIs that affect the performance ofNGS applications the most are: total RAM size on HPC cluster nodes (recommendedat 3 Gb/core, RAM and SSD on the Isilon storage cluster [typically 1 percent of RAMstorage]), and storage configuration parameters with NFS version V4, NFS asyncenabled, TCP MTU (jumbo frames), LACP (2x 1 Gb/s or 4x 1 Gb/s) and a GridEngine package.NGS environments require a file storage infrastructure that is purpose-built to addressthe capacity and performance scalability, efficiency, availability, and manageabilitychallenges of next-generation NGS applications. Cumulative network bandwidth betweenHPC and NAS increases with the total number of Isilon nodes on the storage cluster.Isilon scale-out NAS presents a range of benefits optimal for NGS. The Isilon approachof enabling storage I/O and capacity growth through addition of cluster nodes is optimalsince NGS requires storage performance and capacity scalability to be implementedas seamlessly as possible. In addition, dynamic content balancing performed withinIsilon scale-out NAS as nodes are added or data capac ity changes is ideal for an NGSworkflow due to its sustained throughput requirement.Isilon scale-out NAS also offers an 80 percent efficiency ratio and ―smart pooling‖ ofthe data across multiple performance tiers, making dynamic, rule-based data transferbetween storage pools an integral piece of the NGS process. Flexible, multi-dimensionaldata protection which occurs within Isilon scale-out NAS during power loss, node or diskfailures, loss of quorum, and storage rebuild enables non-stop data availability for NGS. Next-Generation Genome Sequencing Using EMC Isilon Scale-out NAS 20