Extreme scriptingand other adventures in data-intensive computingIan FosterAllan Espinosa, Ioan Raicu, Mike Wilde, Zhao ZhangComputation InstituteArgonne National Lab & University of Chicago

How data analysis happens at data-intensive computing workshops

How data analysis really happensin scientific laboratories%foo file1 > file2% bar file2 > file3%foo file1 | bar > file3% foreachf (f1 f2 f3 f4 f5 f6 f7 … f100)foreach?foo $f.in | bar > $f.outforeach? end%% Now where on earth is f98.out, and how did I generate it again?Now: command not found.%

Extreme scriptingComplexscriptsSwiftMany activitiesNumerous filesComplex dataData dependenciesMany programsSimplescriptsBigcomputersSmallcomputersMany processorsStorage hierarchyFailureHeterogeneityPreservingfile system semantics,ability to call arbitrary executables

Functional magnetic resonance imaging (fMRI) data analysis

AIRSN program definition(Run or) reorientRun (Run ir, string direction) {foreachVolume iv, i in ir.v {or.v[i] = reorient(iv, direction);}}(Run snr) functional ( Run r, NormAnat a, Air shrink ) {Run yroRun = reorientRun( r , "y" );Run roRun = reorientRun( yroRun , "x" );Volume std = roRun[0];Run rndr = random_select( roRun, 0.1 );AirVectorrndAirVec = align_linearRun( rndr, std, 12, 1000, 1000, "81 3 3" );Run reslicedRndr = resliceRun( rndr, rndAirVec, "o", "k" );Volume meanRand = softmean( reslicedRndr, "y", "null" );Air mnQAAir = alignlinear( a.nHires, meanRand, 6, 1000, 4, "81 3 3" );Warp boldNormWarp = combinewarp( shrink, a.aWarp, mnQAAir );Run nr = reslice_warp_run( boldNormWarp, roRun );Volume meanAll = strictmean( nr, "y", "null" )Volume boldMask = binarize( meanAll, "y" );snr = gsmoothRun( nr, boldMask, "6 6 6" );}

Many many tasks:Identifying potential drug targets 2M+ ligands Protein xtarget(s) Benoit Roux et al.

6 GB2M structures(6 GB)~4M x 60s x 1 cpu~60K cpu-hrsFREDDOCK6Select best ~5KSelect best ~5K~10K x 20m x 1 cpu~3K cpu-hrsAmberSelect best ~500~500 x 10hr x 100 cpu~500K cpu-hrsGCMCZINC3-DstructuresManually prepDOCK6 rec fileManually prepFRED rec fileNAB scriptparameters(defines flexibleresidues, #MDsteps)NABScriptTemplateDOCK6Receptor(1 per protein:defines pocketto bind to)FREDReceptor(1 per protein:defines pocketto bind to)PDBproteindescriptions1 protein(1MB)BuildNABScriptAmber prep:2. AmberizeReceptor4. perl: gen nabscriptNABScriptstartAmber Score:1. AmberizeLigand3. AmberizeComplex5. RunNABScriptFor 1 target:4 million tasks500,000 cpu-hrs(50 cpu-years)endreportligandscomplexes

IBM BG/P570 Teraflop/s, 164,000 cores, 80 TB

DOCK on BG/P: ~1M tasks on 119,000 CPUs 118784 cores 934803 tasks Elapsed time: 7257 sec Compute time: 21.43 CPU years Average task: 667 secTime (sec)Relativeefficiency 99.7% (from 16 to 32 racks)Utilization: 99.6% sustained, 78.3% overallIoan Raicu et al.

Managing 160,000 coresFalkonHigh-speed local “disk”Slower shared storage

Scaling Posix to petascaleGlobal file systemChirp(multicast)Staging Torus and tree interconnects IntermediateCN-striped intermediate file systemLargedatasetMosaStore(striping)…IFScomputenodeIFScomputenodeLFSLFSIFSsegIFSsegZOID on I/O nodeComputenode(local datasets)Computenode(local datasets)ZOID IFSLocal. . .

Efficiency for 4 second tasks and varying data size(1KB to 1MB) for CIO and GPFS up to 32K processors

+++++++=Provisioning for data-intensive workloadsExample: on-demand “stacking” of arbitrary locations within ~10TB sky surveyChallengesRandom data accessMuch computingTime-varying loadSolutionDynamic acquisition of compute & storageData diffusionSloanDataSIoanRaicu

IoanRaicu“Sine” workload, 2M tasks, 10MB:10ms ratio, 100 nodes, GCC policy, 50GB caches/node

Same scenario, but with dynamic resource provisioning

Data diffusion sine-wave workload: SummaryGPFS  5.70 hrs, ~8Gb/s, 1138 CPU hrsDD+SRP  1.80 hrs, ~25Gb/s, 361 CPU hrsDD+DRP  1.86 hrs, ~24Gb/s, 253 CPU hrs

Data-intensive computing @ Computation Institute: Example applicationsAstrophysicsCognitive scienceEast Asian studiesEconomicsEnvironmental scienceEpidemiologyGenomic medicineNeurosciencePolitical scienceSociologySolid state physics

Sequencing outpaces Moore’s lawBLASTOn EC2,US$Next-gen Solexa454SolexaGigabasesFolker Meyer, Computation Institute

Data-intensive computing @ Computation Institute: Hardware1000 TBtape backupDynamic provisioning500 TB reliable storage (data & metadata)Parallel analysisDiversedatasourcesRemote accessP A D S180 TB, 180 GB/s 17 Top/s analysisDiverseusersDataingestOffload to remote data centersPADS: Petascale Active Data Store (NSF MRI)

Data-intensive computing @ Computation Institute: SoftwareHPC systems software (MPICH, PVFS, ZeptOS)Collaborative data tagging (GLOSS)Data integration (XDTM)HPC data analytics and visualizationLoosely coupled parallelism (Swift, Hadoop)Dynamic provisioning (Falkon)Service authoring (Introduce, caGrid, gRAVI)Provenance recording and query (Swift)Service composition and workflow (Taverna)Virtualization management (Workspace Service)Distributed data management (GridFTP, etc.)

Data-intensive computing is an end-to-end problemLowChaosZone ofcomplexityAgreementaboutoutcomesPlan and controlHighLowHighCertainty about outcomesRalph Stacey, Complexity and Creativity in Organizations, 1996

We need to function in the zone of complexityLowChaosAgreementaboutoutcomesPlan and controlHighLowHighCertainty about outcomesRalph Stacey, Complexity and Creativity in Organizations, 1996

The Grid paradigmPrinciples and mechanisms for dynamic virtual organizationsLeverage service oriented architectureLoose coupling of data and servicesOpen software,architectureEngineeringBiomedicineComputer sciencePhysicsHealthcareAstronomyBiology1995 2000 2005 2010

As of Oct19, 2008:122 participants105 services70 data35 analytical

Multi-center clinical cancer trials image capture and review(Center for Health Informatics)

SummaryExtreme scripting offers the potential for easy scaling of proven working practicesInteresting technical problems relating to programming and I/O modelsMany wonderful applicationsData-intensive computing is an end-to-end problemData generation, integration, analysis, etc., is a continuous, loosely coupled process

Thank you!Computation Institutewww.ci.uchicago.edu