Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.                                              Upcoming SlideShare
×

# 3D modelling and inversion in escript

108 views

Published on

Lutz Gross of the University of Queensland describes running geophysical inversion using e-script, an open source package based on PDEs and python. Other examples of what e-script can do are also shown, such as diffusion calculations, mantle convection, flow in porous media, seismo-electrics and much more!

Published in: Science
• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here • Be the first to comment

• Be the first to like this

### 3D modelling and inversion in escript

1. 1. 1 March’18 3D modelling and inversion in escript Dr. Lutz Gross School of Earth & Environmental Sciences The University of Queensland St Lucia, Australia
2. 2. 2 March’18 esys-escript from esys.escript import * import esys.escript.unitsSI as U Q=10*U.W/U.m**3; K=1.7*U.W/(U.m*U.K); rhocp=1.5*U.Mega*U.J/(U.m**3*U.K) # … time step size: dt=1.*U.year; n_end=100 # … 40 x 20 grid on 10km x 5km mydomain=Rectangle(40, 20,l0=10*U.km, l1=5*U.km) x=mydomain.getX() # … create PDE and set coefficients: mypde=LinearPDE(domain) mypde.setValue(A=K*kronecker(mydomain), D=rhocp/dt, q=whereZero(x- 5*U.km)) # … initial temperature is a vertical, linear profile: T=0*U.Celsius+30*U.K/U.km*(5*U.km-x) n=0 while n<n_end : mypde.setValue(Y=Q+rhocp/dt*T, r=T) T=myPDE.getSolution() n+=1 saveVTK(“u%s”%n, temperature=T, flux=-K*grad(T) ) print(“Time “,n*dt,”: min/max temperature =”,inf(T), sup(T)) −∇t ( A ∇ u+Bu)+C ∇ u+Du=−∇t X+Y Mathematical Model esys-escript: scripted Model Implementation Geometry Desktop Parallel Supercomputers
3. 3. 3 March’18 Esys-Escript Software ● Direct funding for 13+ years: ~\$3M ● Modeling with PDEs ● Rapid prototyping of new models ● Ease of Use ● “Supercomputing for the masses” ● Programming in python ρcp ∂T ∂t −∇ t K ∇ T=Q
4. 4. 4 March’18 Finite Element Method (FEM) ● approximative solution PDEs in variational form ● by continuous, piecewise linear approximation ● Solve discrete problem with sparse matrix – With algebraic multi-grid (AMG) (see later) ● Domain decomposition of mesh on parallel computers – Hidden from then user
5. 5. 5 March’18 PDEs in esys-escript general linear PDE for solution u : −∇ t ( A ∇ u+Bu)+C ∇ u+D u=−∇ t X+Y Identify PDE coefficients!? + boundary conditions
6. 6. 6 March’18 How to use esys-escript Temperature Diffusion: ρcp ∂T ∂t −∇ t K ∇ T=Q at topT=0 o C Q Domain 10km 5km Assume homogeneous rock Initial temperature T(t=0) = vertical linear profile
7. 7. 7 March’18 ∂T ∂t ≈ T (n) −T (n−1) dt −∇t K⏟ = A ∇ T(n) + ρcp dt⏟ =D T(n) =Q+ ρcp dt T(n−1) ⏟ =Y Apply Backward Euler Method How to deal with the time derivative?
8. 8. 8 March’18 esys-escript Implementation from esys.escript import * import esys.escript.unitsSI as U Q=10*U.W/U.m**3; K=1.7*U.W/(U.m*U.K); rhocp=1.5*U.Mega*U.J/(U.m**3*U.K) # … time step size: dt=1.*U.year; n_end=100 # … 40 x 20 grid on 10km x 5km mydomain=Rectangle(40, 20,l0=10*U.km, l1=5*U.km) x=mydomain.getX() # … create PDE and set coefficients: mypde=LinearPDE(domain) mypde.setValue(A=K*kronecker(mydomain), D=rhocp/dt, q=whereZero(x- 5*U.km)) # … initial temperature is a vertical, linear profile: T=0*U.Celsius+30*U.K/U.km*(5*U.km-x) n=0 while n<n_end : mypde.setValue(Y=Q+rhocp/dt*T, r=T) T=myPDE.getSolution() n+=1 saveVTK(“u%s”%n, temperature=T, flux=-K*grad(T) ) print(“Time “,n*dt,”: min/max temperature =”,inf(T), sup(T))
9. 9. 9 March’18 Visualization by visit
10. 10. 10 March’18 Some applications ● Mantel Convection ● Pores Media Flow ● Geomechanics ● Seismo-Electric ● Geophysical inversion ● Earthquakes ● Volcanoes ● Tsunamis ● ...
11. 11. 11 March’18 Example: Seismo-Electric ● With Simon Shaw (Msc)
12. 12. 12 March’18 Experiment Set-up Wet V
13. 13. 13 March’18 Electro-kinetic Coupling ➢ Ions in pores fluid are separated by surface charge on solid ➢ Movement of fluid creates a current ➢ Induces an electrical field
14. 14. 14 March’18 Mathematical Model ∂2 p ∂t 2 =∇ t vp 2 ∇ p+s(t) Propagation of pressure wave p in solid ppore=B⋅p with Skempton coefficient 0≤B<1 Assume saturated media under undrained conditions (Revil et al 2013): Darcy's law : flux q=− K ηf ∇ ppore
15. 15. 15 March’18 Mathematical Model (cont.) Electro-kinetic Coupling: flux induces current : j=QV⋅q −∇t σ ∇ u=∇t j Electric potential u from electric current: electrical field E=−∇ u Horizontal component of E is measured on surface electric conductivity σ
16. 16. 16 March’18 Qv vs permeability
17. 17. 17 March’18 Single Layer: Horizontal E Coupled Spectral Element (SEM) – Finite Element (FEM)
18. 18. 18 March’18 Trace Records: Horizontal E Dipole offset from source time Maximum signal offset = half of interface depth (for 2D model) (~10m spacing) Travel time in layer → vp
19. 19. 19 March’18 Inversion
20. 20. 20 March’18 Geophysical Inversion Construct 3D structure of the Earth's subsurface from 2D data collected above/near the surface. Measurements on/near surface physical property in the subsurface Physical model Vertical gravity → density
21. 21. 21 March’18 Electric Resistivity Tomography (ERT) Apply electrical charge Measure response in voltage Region of increased electrical conductivity σ by a pollutant Initial guess σInitial guess σ update σupdate σ Defect small ? Defect small ? Numerical prediction of responses from σ Numerical prediction of responses from σ Defect = difference of measurement and prediction Defect = difference of measurement and prediction Done!Done!
22. 22. 22 March’18 Inversion as Optimization Problem Find argmin ρ∈V J (ρ) data misfit : D(ϕ)= 1 2 ∫Ω (w⋅∇ ϕ−g) 2 dx regularization: R(m)= 1 2 ∫Ω ‖∇ ρ‖ 2 dx PDE constraint : −Δ ϕ=4 πG⋅(ρref +ρ') J(ρ)⏟ costfunction =D(ϕ)⏟ misfit +μ⋅ R(ρ)⏟ regularization
23. 23. 23 March’18 Regularization Western Queensland ~ 64 Million Cells More regularization, larger μ
24. 24. 24 March’18 Quasi-Newton Method get gradient g (ν) =∇ J (m (ν) ) find search direction s (ν) by solving ~ Ηs (ν) =g (ν) with ~ Η≈ Hessian ∇ ∇ J m(ν+1) =m(ν) +α⋅s(ν) with min α J (m(ν) +α⋅s(ν) ) via line search ν←ν+1 initial guess m(0) iteration step ν:
25. 25. 25 March’18 Inversion in escript ● PDEs: Forward problems → cost function J ● One for each experiment, frequency ● PDEs: Adjoint problems → gradient of J ● One for each experiment, frequency ● PDE: approx. inverse Hessian → preconditioner ● Use Hessian of regularization term ● Joint Inversion: System of coupled PDEs
26. 26. 26 March’18 Remarks ● Actually used in escript: ● Broyden–Fletcher–Goldfarb–Shanno (BFGS) – Iterative Quasi-Newton method – Self-preconditioning NLCG ● PDEs solved by FEM ● Same mesh for forward and adjoint → uniqueness ● see Lamichhane & Gross 2017
27. 27. 27 March’18 Australia gravity inversion (FEILDS) ● by A. Aitken (UWA), C. Altinay (now HPE) ● density corrections to the AusREM ρref ● Implemented in esys-escript ● 1/8o resolution @ 200Million unknowns ● Inversion run on >22000 cores on Magnus ● Hybrid mode – Shared memory: OpenMP on nodes – Distributed memory: MPI across nodes
28. 28. 28 March’18 Architecture (conceptional) Computational nodes with CPU- cores sharing memory on node (OpenMP, pthreads). Message passing between computational nodes via network (MPI)
29. 29. 29 March’18 Weak Scalability Constant execution time for doubled problem size on the double number of compute cores 3D: For ½ resolution and 8x # cores the compute time remains constant.
30. 30. 30 March’18 Weak Scalability: what to expect? Computational load ~ #cells/p=const. log(p) Execution Time Extra communication costs = const. Synchronization & global communications ~ log(p) # cores =p #cells per core =const.
31. 31. 31 March’18 Aspects of Scalability ● Memory Requirements ● Keep constant per core! ● Use domain decomposition ● Communication Costs ● Sparsity of connectivity in grid/mesh ● Algorithm ● Number of iteration steps independent from number of unknowns → difficult to achieve
32. 32. 32 March’18 Issues for Large Scale Inversion ● Iteration count matters now; expect ● Higher costs for more observation stations ● in the gradient of the cost function ● Use: adjoint states ● Dense matrices ● forward problem for Greens functions → FEM ● density updates → low rank approximations (BFGS) count =O( 3 √#ncells)
33. 33. 33 March’18 Inversion Benchmark ● Joint gravity-magnetic data ● 3D Rectangular Grid ● Same mesh for all PDEs ● Synthetic Data with noise ● Platform: NCI Canberra ● Intel Xeon Sandy Bridge, 8-core, dual socket, 2.6 GHz ● Infiniband FDR interconnect ● MPI + OpenMP ρ=f (k)
34. 34. 34 March’18 PDE calls vs. Regularization μ
35. 35. 35 March’18 Weak Scalability 30000 grid cells per core
36. 36. 36 3D ERT 64 electrodes per line
37. 37. 37 June 2016 ERT with Multi-Grid grid #nodes rel. time rel. size AMG speedup vs PCG 51 140608 1.00 1.00 5.79 99 1000000 6.57 7.11 13.39 151 3511808 23.57 24.98 20.44 199 8000000 52.29 56.90 219 10648000 71.43 75.73 3 level AMG on four core Time/size =const! Codd & Gross 2018
38. 38. 38 June 2016 Structural Joint Inversion ● Invert for density and susceptibility ● spatial change should go together somehow Gravity Anomalies QLD: Magnetic field intensity QLD
39. 39. 39 March’18 Cost Funtion J (m)=D1(ϕ1)+μ1 R(m1)⏟ grav +D2(ϕ2)+μ2 R(m2)⏟ mag +νc⋅∫C(m1 ,m2)⏟ cross gradient gravity −Δ ϕ1=4 πG⋅ρref m1 magnetic −Δϕ2=∇(kref e m2 B)
40. 40. 40 March’18 Cross Gradient function “Where density and susceptibility change they change in the same direction.” ∇ m1 density contour normal susceptibility contour normal ∇ m2 C(m1, m2)=∣∇ m1×∇ m2∣ 2 minimize angle between contour normals
41. 41. 41 March’18 Example Measure of correlation: M= ∇ m1⋅∇ m1 |∇ m1||∇ m2| ∈[−1,1]
42. 42. 42 March’18 Example Results (vertical) m1 , νc=0 m2 , νc=0 m1 , νc=10 8 μ m2 , νc=108 μ M 2 , νc=0 M2 , νc=108 μ
43. 43. 43 March’18 Can we do better? ∫C(m1 ,m2)∝R(m1)⋅R(m2) small vs. regularization C(2) (m1 ,m2)= ( 1 |∇ m1| 2 + 1 |∇ m2| 2 )|∇ m1×∇ m2|2
44. 44. 44 March’18 Example again m1 , νc=100μ m2 , νc=100μ M2 , νc=100μ allways works: νc=100μ
45. 45. 45 March’18 QLD Data (vertical) m1 , νc=100μ m1 , νc=100μ M 2 , νc=100μ 30 Million cells on ~200 cores
46. 46. 46 March’18 What is next ● Current focus on ERT-type methods ● Simulation-Geophysical Signal coupling ● CCSRDF Sequestration projects with Uni Melbourne and Uni Savoy Monte Blanc ● EM/MT problems ● More on seismic & FWI