Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Uncertainty and Sensitivity Analysis using HPC and HTC

1,183 views

Published on

Presentation made by Dr. André Barbosa @ University of Porto during the OpenSees Days Portugal 2014 workshop

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Uncertainty and Sensitivity Analysis using HPC and HTC

  1. 1. OPENSEES DAYS PORTUGAL 2014 UNCERTAINTY AND SENSITIVITY ANALYSIS USING HPC AND HTC André R. Barbosa (1) Andre.Barbosa@oregonstate.edu (1) Assistant Professor, School of Civil and Construction Engineering, Oregon State University Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  2. 2. Design Alternatives Hazard Analysis Introduction Structural Analysis Damage Analysis Loss Analysis Decision Making L,D P[IM| X,D] ν[IM] P[EDP | IM] ν[EDP] P[DM| EDP] ν[DM] P[DV| DM] Select ν[DV] L,D Intensity Measure L: Location D: Design Engineering Demand Par. Damage Measure Decision Variable q Parametric sensitivity studies / optimization / design (Luis Celorrio-­‐Barragué) q Probabilistic seismic demand analysis Ø Cloud Method Ø Incremental dynamic analysis (Filipe Ribeiro) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 2
  3. 3. Design Alternatives Hazard Analysis Introduction Structural Analysis Damage Analysis Loss Analysis Decision Making L,D P[IM| X,D] ν[IM] P[EDP | IM] ν[EDP] P[DM| EDP] ν[DM] P[DV| DM] Select ν[DV] L,D Intensity Measure L: Location D: Design Engineering Demand Par. Damage Measure Decision Variable q Parametric sensitivity studies q Probabilistic seismic demand analysis Ø Cloud Method Ø Incremental dynamic analysis Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 3
  4. 4. Probabilistic Seismic Hazard Analysis flt N =Σ ∫ ∫ ⎡⎣ > = = ⎤⎦ ( ) ( ) ( ) ν im ν P IM im M m R r f m f r dm dr IM i i i M R i = 1 R M Fault j Site AAenua8on rela8ons R i i fR(r) IM m0 M mu , i i Magnitude Source-­‐to-­‐site distance fR(r) IM m0 M mu fM(m) R Seismic hazard curve M-­‐R deaggrega8on IM= Sa (T1 ) Fault i fM(m) Fault k R R ( ) IM ν im Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  5. 5. Response estimation accounting for modeling uncertainty q PSDA equa9on accoun9ng for model parameter uncertainty: ν edp P EDP edp IM f d dν im Θ = ∫ > Θ Θ Θ⋅ q Response es9ma9on: XLB XM XUB { } 1, , | , ,..., k lk P⎡⎣EDP > edp IM = im Θ = θ θ ⎤⎦ Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 5 ( ) [ | , ] ( ) ( ) EDP IM IM EDPLB EDPM EDPUB INPUT NLTH ANALYSIS OUPUT μθ + aσθ
  6. 6. Parameter uncertainty progagation INPUT Probability Distribution of RV X XL B XM X UB 3D NL FE MODEL TIME HISTORY ANALYSIS Uncertainty in ground motion Intensity Measure (IM) Ground motion profile (GM) Uncertainty in structural properties Mass Viscous damping Strength Stiffness OUTPUT Probability Distribution of EDP j EDP(XL B ) EDP(XM ) EDP(XU B ) Global EDPs U : Max Roof Displacement A : Max Floor Acceleration. IDR : Max Interstory Drift Ratio Local EDPs Member: Curvature Strains: Reinforcing Steel Concrete Faggella , Barbosa, Conte, Spacone, Restrepo, 2013 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  7. 7. Parameter uncertainty progagation 3D NL FE MODEL TIME HISTORY ANALYSIS INPUT Probability Distribu9on of Variable X X LB X M X UB OUTPUT EDP(X LB ) EDP(X M ) EDP(X Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto UB ) Probability Distribu9on of EDP j TORNADO x10 , x50 , x90 FOSM (First Order Second Moments) xm-as , xm , xm+as TORNADO (swing) EDP(x10) – EDP( x90) FOSM mEDP , sEDP MEAN and STD
  8. 8. TORNADO x10 , x50 , x90 3D NL FE MODEL TIME HISTORY ANALYSIS Swing = EDP(x10) – EDP(x90) 11th value Tornado sensitivity analysis Median GM 0 0.5 1 1.5 2 2.5 3 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 EDP Empirical CDF XLB XM XUB Procedure 1. Perform Monte Carlo Simulation using all ground motions (GM), fixing all other variables at their best estimates (median values) (e.g. GM = 20) 2. For each EDP, determine Median GM, and perturbe all other variables one at a time about their median value Sa GM Damping Mass Fy Fc Es Ec Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  9. 9. First Order Second Moment (FOSM) sensitivity analysis q Mean values q Variance-covariance matrix [ ] Σθ = ⎡⎣ρijσ iσ j ⎤⎦; i, j =1, 2,K , n T 1 2 n = μ , μ ,K , μ θ μ q Taylor series expansion of the response EDP ( ) ( ) ( ) ( ) lin r r r rθ θ θ θ μ θ θ θ μ θ μ = ≈ = +∇ ⋅ − Ø Sensitivity r r r ∂ = + Δ − − Δ ∂ Δ Δ = ( θ ) ( μ θ ) ( μ θ ) i i i i θ 2 θ θ σ i i a i θ i XLB μθ + aσθ XM XUB Ø Covariance matrix of the response n ⎛ ⎞ Σ = Σ ∂ r ⎜ ⎟ ⋅ + ΣΣ ni − ⎛ ∂ r ⎞⎛ ∂ r ⎞ ∂ ⎜ ⎟⎜⎜ ⎟⎟ ⋅ ⋅ ⎝ ⎠ ⎝ ∂ ⎠⎝ ∂ ⎠ σθ ρθ θ σθ σθ 2 i ij i j θ θ θ EDPLB EDPM EDPUB 2 1 2 2 i = 1 i i = 1 j = 1 i j r Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 9
  10. 10. Number of FE runs for TORNADO or FOSM analyses Median GM 11th value 0 0.5 1 1.5 2 2.5 3 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Number of FE runs: nruns = GM+ 2⋅RV⋅EDP ( ) runs e.g., n = 20 + 2×7×10 =160 1 med MONTE CARLO 2 IMLB TORNADO 3 dLB TORNADO 4 mLB TORNADO 5 fyLB TORNADO 6 fcLB TORNADO 7 EsLB TORNADO 8 EcLB TORNADO 9 IMUB TORNADO 10 dUB TORNADO 11 mUB TORNADO 12 fyUB TORNADO 13 fcUB TORNADO 14 EsUB TORNADO 15 EcUB TORNADO 10 EZ_erzi KB_kobj LP_cor LP_gav LP_gilb LP_lex1 LP_lgpc LP_srtg TO_ttr007 TO_ttrh02 CL_clyd CL_gil6 LV_fgnr LV_mgnp MH_andd MH_clyd MH_hall PF_cs05 PF_cs08 PF_temb EDP 1 EDP 2 GM 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 EDP Empirical CDF Sa GM Damping Mass Fy Fc Es Ec TORNADO Swing = EDP(x10) – EDP(x90) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  11. 11. Parallelization of the analyses using XSEDE 0.4 0.2 0 -0.2 Parallel Computer -0.4 0 5 10 15 20 Time (sec ) Acceleration (g) GM 1, Par j 0.4 0.2 0 -0.2 -0.4 0 5 10 15 20 Time (sec ) Acceleration (g) GM 2, Par j … 0.4 Acceleration (g) SUPERCOMPUTERS 0.2 0 -0.2 -0.4 0 5 10 15 20 Time (sec ) GM N, Par j … OpenSees Mul9ple Parallel Interpreter (McKenna and Fenves 2007) hVp://opensees.berkeley.edu/OpenSees/parallel/TNParallelProcessing.pdf Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  12. 12. Case study: Bonefro 4 story building Example 1: Bonefro Italy Molise 2002 earthquake, Italy Faggella et al. 2008 Severe damage to first story infills and columns Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  13. 13. Model Variation of the res (pcolanssse) uunndceerrt adiinffteyr ent modeling assumptions Bare Frame Stairs Diaphragms (2x2) NL Infills NL Inf. Bare 1st story NL Shear columns Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  14. 14. Variation of the response under different modeling Model uncertainty 12 assumptions 2000 1500 Base Shear (KN) 1000 500 0 shell 2x2 infilled bare frame stairs part. infilled 0 50 100 150 200 Top floor displacement (mm) ADRS Demand Spectrum Capacity Spectra infilled 0.71 0.83 part. infilled 0.89 0.15 T C 0.4 2 shell 2x2 1.25 stairs 1.09 1 0.8 0.6 0.4 0.2 0 bare frame 0 0.05 0.1 0.15 0.2 Sde (m) Se/g , F*/gm* TH Average Bare Frame Diaph.2x2 Stairs NL Inf. Bare1 NL Infills NLshear col. 0 50 100 150 200 4 3 2 1 0 Displacements (mm) Floor TH Average 0 0.5 1 1.5 2 4 3 2 1 0 Floor Drift % Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  15. 15. Parameter uncertainty Uncertainty in structural properties • Mass • Viscous damping • Strength • Stiffness Ec (GPa) Uncertainty in ground motion • Intensity Measure (IM) • Ground motion profile (GM) Distrib. MCS Logn. Norm. Norm. Logn. Norm. Norm. Norm. XM On EDP 0.2931 0.03 0.87 451 25 210 28 COV % // 84 40 10 10 6.4 3.3 8 Probability Functions based on • Seismic hazard • Values adopted in the literature • Experimental samples (material testing) 5 Ground motion and structural random variables GM IM=Sa(T1) (g) Damping (%) Mass (ton/m2) Fy (MPa) Fc (MPa) Es (GPa) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  16. 16. 3D Response Engineering Demand Parameters (EDPs) 25 X Y Rz V G Outputs (EDPs) Μ, Χ LOCAL Member Sections Curvature Member Sections Moment σ , ε Steel GLOBAL U : Max Roof Displacement A : Max Floor Acceleration. IDR : Max Interstory Drift Ratio R Concrete core Concrete unconf. 4001 3001 2001 1001 4008 3008 2008 121 122 1008 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  17. 17. Median MGM (11° value) Outputs (EDPs) R Tornado for MGM, all other variables perturbed one at a time about the median 26 Results of MCS and TORNADO analysis Monte Carlo using 20 ground motions all other variables at medians X Y Rz V G 3D EDPs Floor DOFs Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  18. 18. A : Max Floor Acceleration. Member Sections Curvature Member Sections Moment Outputs (EDPs) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 25 ! X Rz G Μ, Χ σ , ε Steel Concrete core Concrete unconf. LOCAL IDR : Max Interstory Drift Ratio 1001 1008 2001 2008 3001 3008 4001 4008 121 122 R
  19. 19. Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 3D Response Engineering Demand Parameters Y X Rz V G Μ, Χ Member Sections Curvature Member Sections Moment σ , ε Steel GLOBAL U : Max Roof Displacement A : Max Floor Acceleration. Concrete core Concrete unconf. LOCAL IDR : Max Interstory Drift Ratio 1001 1008 2001 2008 3001 3008 4001 4008 121 122 R Outputs (EDPs)
  20. 20. PEER PBEE Methodology Design Alternatives Hazard Analysis Structural Analysis Damage Analysis Loss Analysis Decision Making L,D P[IM| X,D] ν[IM] P[EDP | IM] ν[EDP] P[DM| EDP] ν[DM] P[DV| DM] Select ν[DV] L,D Intensity Measure L: Location D: Design Engineering Demand Par. Damage Measure Decision Variable q Parametric sensitivity studies q Probabilistic seismic demand analysis Ø Cloud Method Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 20
  21. 21. Example 2: NEHRP Building Modeling Approach g u&& Ø Walls: Nonlinear truss modeling approach Ø Columns and beams: Force-based beam-column elements Ø Diaphragms: Flexible diaphragms allowing for plastic hinge elongation NL Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 21 q Rigid-end zone modeling for beam-column joints (ASCE41-06) REZ NL NL NL NL q Comprehensive/significant valida8on at system level ? … q Comprehensive/significant valida8on at component level
  22. 22. Observed computational building behavior EW: 0.44 % NS: 2.93 % N Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 22 (%)
  23. 23. “Cloud method”: Selection of earthquake records q NGA database (total 3551 records) Ø Mechanism: Strike-slip (1004 records) Ø Magnitude range: 5.5 to 8 (772 records) Ø Distance: 0 – 40 kms (203 records) Ø Vs30: C/D range (90 records) 40 35 30 25 20 15 10 5 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 23 0 5.5 6.0 6.5 7.0 7.5 8.0 Source-to-site distance Rrup Magnitude Mw Non-pulse Pulse q 90 ground mo8on records selected from 14 earthquakes 6.0 6.5 7.0 7.5 8.0 Magnitude Mw Non-pulse Pulse
  24. 24. q Motivation Ø Perform parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with OpenSees runs. q Application Example/Production campaign 1 (1) Probabilistic seismic demand hazard analysis using the “cloud method” q Some numbers for this application example Number of NLTH analyses 180 Average duration of NLTH analysis 12 hours Average size of output data (compressed) 1.4 GB Estimated clock time on a desktop computer (180x12) 2,160 hours 90 days Estimated size of output data (180x1.4) 250 GB 1. OpenSeesMP + Xsede? 2. Local Cluster? 3. Other options? Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 24 OpenSees and Large Number of Runs GM1 GM2 GM180 ...
  25. 25. Possible Parallelization Options q OpenSeesMP + MPICH2 – useful for Domain Decomposition + Parameter Studies (addressed by other talks in this meeting) q Condor + OpenSees Sequential – Parameter Studies Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  26. 26. HTCondor q HTCondor (http://research.cs.wisc.edu/htcondor/) is a specialized workload management system for computational-intensive jobs. Ø Project started in 1988, directed at users with large computing needs and environments with heterogeneous distributed resources. Ø HTCondor is composed of 3 parts: (1) Submit Node Submit job Schedd (2) Central Manager Collector Negotiator (3) Worker Node Startd Get results GM1 Worker Node Startd … GM180 Worker Node Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  27. 27. Oregon State University: HTCondor + OpenSees q “Opportunistic” computing resources: q Student computer labs (used by students mainly during the day, and during the term …) q Instruction computer labs (used during the term only during classes …) q College of Engineering at OSU: 16 computer labs (~1500 cores) http://monhost.engr.orst.edu/labs/ Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  28. 28. Implementation of HTCondor at Oregon State University (1) Submit Node (3) Worker Nodes 1 • 8 core Intel i7 • Windows Server • 16 GB RAM • SSD drive • 2 TB HDD 15K • 20 TB NAS (2) Central Manager … • Windows 7 Premium • 8 GB RAM • 2 x 1GB cards • 1 TB 7.2 K The good news: ~ 1500cores Communication w/ IT, Dealing w/ Job recovery, W/O speed, data transfers, …? Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  29. 29. Ø Perform parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with OpenSees runs. q Some numbers for this application example Number of NLTH analyses 180 Average duration of NLTH analysis 12 hours Average size of output data 1.4 GB Estimated clock time on a desktop computer (180x12) 2,160 hours 90 days Estimated size of output data (180x1.4) 250 GB Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 29 OpenSees and Large Number of Runs Clock time 36 hours !! q Motivation q Application Example/Production campaign 1 (1) Probabilistic seismic demand hazard analysis using the “cloud method”
  30. 30. (a) (b) (c) Individual Ekqe 2.5- and 97.5-perc Median Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 30 OpenSees and Parameters Studies PFD – peak floor displacement; PIDR – peak interstory drift ratio; PFA – peak floor absolute acceleration
  31. 31. HTCondor and Open Science Grid q HTCondor (hAp://research.cs.wisc.edu/htcondor/) is a specialized workload management system for computa9onal-­‐intensive jobs. Ø Project started in 1988, directed at users with large compu9ng needs and environments with heterogeneous distributed resources. q Open Science Grid is a national, distributed computing grid for data-intensive research. Ø Consortium of approx. 80 national laboratories and universities. Ø Version of Condor for the grid Ø Opportunistic resource usage: resources are sized for peak needs of large experiments (Atlas, CMS, etc.), OSG allows for non-paying organizations to use their resources. q NEES and Open Science Grid have been active partners in creating the tools and infrastructures for making use of opportunistic resources Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 31
  32. 32. Response estimation accounting for parameter uncertainty XLB XM GM Damping XUB μθ Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 32 EDPLB EDPM EDPUB INPUT NLTH ANALYSIS OUPUT Uncertainty in structural properties • Mass • Viscous damping • Strength • Stiffness Engineering demand parameters • Roof drift ratio • Peak floor accelerations • Shear demand in walls • Residual deformatios.. μθ + aσθ (%) Mass fy (ksi) *fc (ksi) Es (ksi) *Ec (ksi) XM MCS 0.02 68.7 6.84 29000 4714 COV % // 40 10 10 10 3.3 8
  33. 33. Using Open Science Grid: Production Campaign 2 q Production campaign (1) Probabilistic seismic demand hazard analysis using the cloud method (2) Sensitivity of probabilistic seismic demand hazard to FE model parameters q Some numbers for production campaign 2 (99% complete) Number of NLTH analyses per parameter set realization 180 Average duration of NLTH analysis 12 hours Average size of output data 1.4 GB Parameters considered 6 Perturbations considered 4 Estimated clock time on a desktop computer (180x12x[(6x4x2)+1]) 105,840 hours 12.1 years Estimated size of output (compressed) data (180x1.4x[(6x4x2)+1]) 12 TB Clock time 30 days !! Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 33
  34. 34. 30,000 OSG users: André R. Barbosa, Taylor Gugino (UCSD) OSG support: Gabriele Garzoglio, Marko Slyz (OSG) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 34 Wall clock time in HTCondor / OSG 12 clusters of 180 jobs “Desktop”: 26,000 hours OSG: 60,000 hours 25,000 20,000 15,000 10,000 5,000 0 Wall Time (hours) (job preemp9on)
  35. 35. 160,000 120,000 80,000 40,000 0 OSG users: André R. Barbosa, Taylor Gugino (UCSD) OSG support: Gabriele Garzoglio, Marko Slyz (OSG) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Wall Time (hours) Wall clock time in HTCondor / OSG
  36. 36. Comparison Between Parallelization Options OpenSeesMP HTCondor Straight forward implementation of Domain Decomposition through OpenSees framework with parallel solving algorithm like MUMPS No ready built solution for large problems, OpenSees sequential does not have parallel solvers for large problems MPICH2 networking setup is relatively easier Job management easier Condor pool setup requires some learning Condor requires maintenance and administration Very active user support through OpenSees user community, most attractive aspect of using OpenSeesMP There is no specific user community as such. Limited tests show 190 % Speed up from one processor to two processor Limited tests show 153 % Speed up from one processor to two processor Main complication is compilation of OpenSeesMP, really really tough!! But once over it OpenSeesMP is really powerfull!!! Global implementation, if want to connect to other grid systems. Steep learning curve , knowledge of networking (Computer science) Khaled Mashfiq, MS – La Sapienza, Rome Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
  37. 37. Conclusions 37 ü A workflow for running parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with large number of parameters and OpenSees runs has been developed for using NEEShub, Xsede, and Open Science Grid. ü HTCondor ü Pegassus (see Frank Mckenna’s presentation) ü OpenSees + Condor q User interfaces for submitting jobs, receiving results q Data visualization ü Management and Analysis of Large Research Data Sets q Where and what to store? q Post-processing? Data compression algorithms?
  38. 38. Andre.Barbosa@oregonstate.edu Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 38

×