Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1 March’18
3D modelling and inversion in escript
Dr. Lutz Gross
School of Earth & Environmental Sciences
The University of...
2 March’18
esys-escript
from esys.escript import *
import esys.escript.unitsSI as U
Q=10*U.W/U.m**3; K=1.7*U.W/(U.m*U.K); ...
3 March’18
Esys-Escript Software
●
Direct funding for 13+ years: ~$3M
●
Modeling with PDEs
●
Rapid prototyping of new mode...
4 March’18
Finite Element Method (FEM)
●
approximative solution PDEs in variational form
●
by continuous, piecewise linear...
5 March’18
PDEs in esys-escript
general linear PDE for solution u :
−∇
t
( A ∇ u+Bu)+C ∇ u+D u=−∇
t
X+Y
Identify PDE coeff...
6 March’18
How to use esys-escript
Temperature Diffusion:
ρcp
∂T
∂t
−∇
t
K ∇ T=Q
at topT=0
o
C
Q
Domain
10km
5km
Assume ho...
7 March’18
∂T
∂t
≈
T
(n)
−T
(n−1)
dt
−∇t
K⏟
= A
∇ T(n)
+
ρcp
dt⏟
=D
T(n)
=Q+
ρcp
dt
T(n−1)
⏟
=Y
Apply Backward Euler Metho...
8 March’18
esys-escript Implementation
from esys.escript import *
import esys.escript.unitsSI as U
Q=10*U.W/U.m**3; K=1.7*...
9 March’18
Visualization by visit
10 March’18
Some applications
●
Mantel Convection
●
Pores Media Flow
●
Geomechanics
●
Seismo-Electric
●
Geophysical invers...
11 March’18
Example: Seismo-Electric
●
With Simon Shaw (Msc)
12 March’18
Experiment Set-up
Wet
V
13 March’18
Electro-kinetic Coupling
➢
Ions in pores fluid are
separated by surface
charge on solid
➢
Movement of fluid
cr...
14 March’18
Mathematical Model
∂2
p
∂t
2
=∇
t
vp
2
∇ p+s(t)
Propagation of pressure wave p in solid
ppore=B⋅p with Skempto...
15 March’18
Mathematical Model (cont.)
Electro-kinetic Coupling: flux induces current : j=QV⋅q
−∇t
σ ∇ u=∇t
j
Electric pot...
16 March’18
Qv vs permeability
17 March’18
Single Layer: Horizontal E
Coupled Spectral Element (SEM) – Finite Element (FEM)
18 March’18
Trace Records: Horizontal E
Dipole offset from source
time
Maximum signal offset
= half of interface depth
(fo...
19 March’18
Inversion
20 March’18
Geophysical Inversion
Construct 3D structure of the Earth's subsurface
from 2D data collected above/near the s...
21 March’18
Electric Resistivity Tomography (ERT)
Apply electrical
charge
Measure response
in voltage
Region of increased
...
22 March’18
Inversion as Optimization Problem
Find argmin
ρ∈V
J (ρ)
data misfit : D(ϕ)=
1
2
∫Ω
(w⋅∇ ϕ−g)
2
dx
regularizati...
23 March’18
Regularization
Western Queensland
~ 64 Million Cells
More regularization, larger μ
24 March’18
Quasi-Newton Method
get gradient g
(ν)
=∇ J (m
(ν)
)
find search direction s
(ν)
by solving
~
Ηs
(ν)
=g
(ν)
wi...
25 March’18
Inversion in escript
●
PDEs: Forward problems → cost function J
●
One for each experiment, frequency
●
PDEs: A...
26 March’18
Remarks
●
Actually used in escript:
●
Broyden–Fletcher–Goldfarb–Shanno (BFGS)
– Iterative Quasi-Newton method
...
27 March’18
Australia gravity inversion (FEILDS)
●
by A. Aitken (UWA), C. Altinay (now HPE)
● density corrections to the A...
28 March’18
Architecture (conceptional)
Computational nodes with CPU-
cores sharing memory on node
(OpenMP, pthreads).
Mes...
29 March’18
Weak Scalability
Constant execution time for doubled problem
size on the double number of compute cores
3D: Fo...
30 March’18
Weak Scalability: what to expect?
Computational load ~
#cells/p=const.
log(p)
Execution
Time
Extra communicati...
31 March’18
Aspects of Scalability
●
Memory Requirements
●
Keep constant per core!
●
Use domain decomposition
●
Communicat...
32 March’18
Issues for Large Scale Inversion
●
Iteration count matters now; expect
●
Higher costs for more observation sta...
33 March’18
Inversion Benchmark
●
Joint gravity-magnetic data
●
3D Rectangular Grid
●
Same mesh for all PDEs
●
Synthetic D...
34 March’18
PDE calls vs. Regularization
μ
35 March’18
Weak Scalability
30000 grid cells per core
36
3D ERT
64 electrodes per line
37 June 2016
ERT with Multi-Grid
grid #nodes rel. time rel. size AMG speedup vs
PCG
51 140608 1.00 1.00 5.79
99 1000000 6....
38 June 2016
Structural Joint Inversion
●
Invert for density and
susceptibility
●
spatial change should go
together someho...
39 March’18
Cost Funtion
J (m)=D1(ϕ1)+μ1 R(m1)⏟
grav
+D2(ϕ2)+μ2 R(m2)⏟
mag
+νc⋅∫C(m1 ,m2)⏟
cross gradient
gravity −Δ ϕ1=4 ...
40 March’18
Cross Gradient function
“Where density and susceptibility change they
change in the same direction.”
∇ m1
dens...
41 March’18
Example
Measure of correlation: M=
∇ m1⋅∇ m1
|∇ m1||∇ m2|
∈[−1,1]
42 March’18
Example Results (vertical)
m1 , νc=0 m2 , νc=0
m1 , νc=10
8
μ m2 , νc=108
μ
M
2
, νc=0
M2
, νc=108
μ
43 March’18
Can we do better?
∫C(m1 ,m2)∝R(m1)⋅R(m2) small vs. regularization
C(2)
(m1 ,m2)=
( 1
|∇ m1|
2
+
1
|∇ m2|
2
)|∇...
44 March’18
Example again
m1 , νc=100μ m2 , νc=100μ
M2
, νc=100μ
allways works: νc=100μ
45 March’18
QLD Data (vertical)
m1 , νc=100μ
m1 , νc=100μ
M
2
, νc=100μ
30 Million cells
on ~200 cores
46 March’18
What is next
●
Current focus on ERT-type methods
●
Simulation-Geophysical Signal coupling
●
CCSRDF Sequestrati...
Upcoming SlideShare
Loading in …5
×

3D modelling and inversion in escript

108 views

Published on

Lutz Gross of the University of Queensland describes running geophysical inversion using e-script, an open source package based on PDEs and python. Other examples of what e-script can do are also shown, such as diffusion calculations, mantle convection, flow in porous media, seismo-electrics and much more!

Published in: Science
  • Be the first to comment

  • Be the first to like this

3D modelling and inversion in escript

  1. 1. 1 March’18 3D modelling and inversion in escript Dr. Lutz Gross School of Earth & Environmental Sciences The University of Queensland St Lucia, Australia
  2. 2. 2 March’18 esys-escript from esys.escript import * import esys.escript.unitsSI as U Q=10*U.W/U.m**3; K=1.7*U.W/(U.m*U.K); rhocp=1.5*U.Mega*U.J/(U.m**3*U.K) # … time step size: dt=1.*U.year; n_end=100 # … 40 x 20 grid on 10km x 5km mydomain=Rectangle(40, 20,l0=10*U.km, l1=5*U.km) x=mydomain.getX() # … create PDE and set coefficients: mypde=LinearPDE(domain) mypde.setValue(A=K*kronecker(mydomain), D=rhocp/dt, q=whereZero(x[1]- 5*U.km)) # … initial temperature is a vertical, linear profile: T=0*U.Celsius+30*U.K/U.km*(5*U.km-x[1]) n=0 while n<n_end : mypde.setValue(Y=Q+rhocp/dt*T, r=T) T=myPDE.getSolution() n+=1 saveVTK(“u%s”%n, temperature=T, flux=-K*grad(T) ) print(“Time “,n*dt,”: min/max temperature =”,inf(T), sup(T)) −∇t ( A ∇ u+Bu)+C ∇ u+Du=−∇t X+Y Mathematical Model esys-escript: scripted Model Implementation Geometry Desktop Parallel Supercomputers
  3. 3. 3 March’18 Esys-Escript Software ● Direct funding for 13+ years: ~$3M ● Modeling with PDEs ● Rapid prototyping of new models ● Ease of Use ● “Supercomputing for the masses” ● Programming in python ρcp ∂T ∂t −∇ t K ∇ T=Q
  4. 4. 4 March’18 Finite Element Method (FEM) ● approximative solution PDEs in variational form ● by continuous, piecewise linear approximation ● Solve discrete problem with sparse matrix – With algebraic multi-grid (AMG) (see later) ● Domain decomposition of mesh on parallel computers – Hidden from then user
  5. 5. 5 March’18 PDEs in esys-escript general linear PDE for solution u : −∇ t ( A ∇ u+Bu)+C ∇ u+D u=−∇ t X+Y Identify PDE coefficients!? + boundary conditions
  6. 6. 6 March’18 How to use esys-escript Temperature Diffusion: ρcp ∂T ∂t −∇ t K ∇ T=Q at topT=0 o C Q Domain 10km 5km Assume homogeneous rock Initial temperature T(t=0) = vertical linear profile
  7. 7. 7 March’18 ∂T ∂t ≈ T (n) −T (n−1) dt −∇t K⏟ = A ∇ T(n) + ρcp dt⏟ =D T(n) =Q+ ρcp dt T(n−1) ⏟ =Y Apply Backward Euler Method How to deal with the time derivative?
  8. 8. 8 March’18 esys-escript Implementation from esys.escript import * import esys.escript.unitsSI as U Q=10*U.W/U.m**3; K=1.7*U.W/(U.m*U.K); rhocp=1.5*U.Mega*U.J/(U.m**3*U.K) # … time step size: dt=1.*U.year; n_end=100 # … 40 x 20 grid on 10km x 5km mydomain=Rectangle(40, 20,l0=10*U.km, l1=5*U.km) x=mydomain.getX() # … create PDE and set coefficients: mypde=LinearPDE(domain) mypde.setValue(A=K*kronecker(mydomain), D=rhocp/dt, q=whereZero(x[1]- 5*U.km)) # … initial temperature is a vertical, linear profile: T=0*U.Celsius+30*U.K/U.km*(5*U.km-x[1]) n=0 while n<n_end : mypde.setValue(Y=Q+rhocp/dt*T, r=T) T=myPDE.getSolution() n+=1 saveVTK(“u%s”%n, temperature=T, flux=-K*grad(T) ) print(“Time “,n*dt,”: min/max temperature =”,inf(T), sup(T))
  9. 9. 9 March’18 Visualization by visit
  10. 10. 10 March’18 Some applications ● Mantel Convection ● Pores Media Flow ● Geomechanics ● Seismo-Electric ● Geophysical inversion ● Earthquakes ● Volcanoes ● Tsunamis ● ...
  11. 11. 11 March’18 Example: Seismo-Electric ● With Simon Shaw (Msc)
  12. 12. 12 March’18 Experiment Set-up Wet V
  13. 13. 13 March’18 Electro-kinetic Coupling ➢ Ions in pores fluid are separated by surface charge on solid ➢ Movement of fluid creates a current ➢ Induces an electrical field
  14. 14. 14 March’18 Mathematical Model ∂2 p ∂t 2 =∇ t vp 2 ∇ p+s(t) Propagation of pressure wave p in solid ppore=B⋅p with Skempton coefficient 0≤B<1 Assume saturated media under undrained conditions (Revil et al 2013): Darcy's law : flux q=− K ηf ∇ ppore
  15. 15. 15 March’18 Mathematical Model (cont.) Electro-kinetic Coupling: flux induces current : j=QV⋅q −∇t σ ∇ u=∇t j Electric potential u from electric current: electrical field E=−∇ u Horizontal component of E is measured on surface electric conductivity σ
  16. 16. 16 March’18 Qv vs permeability
  17. 17. 17 March’18 Single Layer: Horizontal E Coupled Spectral Element (SEM) – Finite Element (FEM)
  18. 18. 18 March’18 Trace Records: Horizontal E Dipole offset from source time Maximum signal offset = half of interface depth (for 2D model) (~10m spacing) Travel time in layer → vp
  19. 19. 19 March’18 Inversion
  20. 20. 20 March’18 Geophysical Inversion Construct 3D structure of the Earth's subsurface from 2D data collected above/near the surface. Measurements on/near surface physical property in the subsurface Physical model Vertical gravity → density
  21. 21. 21 March’18 Electric Resistivity Tomography (ERT) Apply electrical charge Measure response in voltage Region of increased electrical conductivity σ by a pollutant Initial guess σInitial guess σ update σupdate σ Defect small ? Defect small ? Numerical prediction of responses from σ Numerical prediction of responses from σ Defect = difference of measurement and prediction Defect = difference of measurement and prediction Done!Done!
  22. 22. 22 March’18 Inversion as Optimization Problem Find argmin ρ∈V J (ρ) data misfit : D(ϕ)= 1 2 ∫Ω (w⋅∇ ϕ−g) 2 dx regularization: R(m)= 1 2 ∫Ω ‖∇ ρ‖ 2 dx PDE constraint : −Δ ϕ=4 πG⋅(ρref +ρ') J(ρ)⏟ costfunction =D(ϕ)⏟ misfit +μ⋅ R(ρ)⏟ regularization
  23. 23. 23 March’18 Regularization Western Queensland ~ 64 Million Cells More regularization, larger μ
  24. 24. 24 March’18 Quasi-Newton Method get gradient g (ν) =∇ J (m (ν) ) find search direction s (ν) by solving ~ Ηs (ν) =g (ν) with ~ Η≈ Hessian ∇ ∇ J m(ν+1) =m(ν) +α⋅s(ν) with min α J (m(ν) +α⋅s(ν) ) via line search ν←ν+1 initial guess m(0) iteration step ν:
  25. 25. 25 March’18 Inversion in escript ● PDEs: Forward problems → cost function J ● One for each experiment, frequency ● PDEs: Adjoint problems → gradient of J ● One for each experiment, frequency ● PDE: approx. inverse Hessian → preconditioner ● Use Hessian of regularization term ● Joint Inversion: System of coupled PDEs
  26. 26. 26 March’18 Remarks ● Actually used in escript: ● Broyden–Fletcher–Goldfarb–Shanno (BFGS) – Iterative Quasi-Newton method – Self-preconditioning NLCG ● PDEs solved by FEM ● Same mesh for forward and adjoint → uniqueness ● see Lamichhane & Gross 2017
  27. 27. 27 March’18 Australia gravity inversion (FEILDS) ● by A. Aitken (UWA), C. Altinay (now HPE) ● density corrections to the AusREM ρref ● Implemented in esys-escript ● 1/8o resolution @ 200Million unknowns ● Inversion run on >22000 cores on Magnus ● Hybrid mode – Shared memory: OpenMP on nodes – Distributed memory: MPI across nodes
  28. 28. 28 March’18 Architecture (conceptional) Computational nodes with CPU- cores sharing memory on node (OpenMP, pthreads). Message passing between computational nodes via network (MPI)
  29. 29. 29 March’18 Weak Scalability Constant execution time for doubled problem size on the double number of compute cores 3D: For ½ resolution and 8x # cores the compute time remains constant.
  30. 30. 30 March’18 Weak Scalability: what to expect? Computational load ~ #cells/p=const. log(p) Execution Time Extra communication costs = const. Synchronization & global communications ~ log(p) # cores =p #cells per core =const.
  31. 31. 31 March’18 Aspects of Scalability ● Memory Requirements ● Keep constant per core! ● Use domain decomposition ● Communication Costs ● Sparsity of connectivity in grid/mesh ● Algorithm ● Number of iteration steps independent from number of unknowns → difficult to achieve
  32. 32. 32 March’18 Issues for Large Scale Inversion ● Iteration count matters now; expect ● Higher costs for more observation stations ● in the gradient of the cost function ● Use: adjoint states ● Dense matrices ● forward problem for Greens functions → FEM ● density updates → low rank approximations (BFGS) count =O( 3 √#ncells)
  33. 33. 33 March’18 Inversion Benchmark ● Joint gravity-magnetic data ● 3D Rectangular Grid ● Same mesh for all PDEs ● Synthetic Data with noise ● Platform: NCI Canberra ● Intel Xeon Sandy Bridge, 8-core, dual socket, 2.6 GHz ● Infiniband FDR interconnect ● MPI + OpenMP ρ=f (k)
  34. 34. 34 March’18 PDE calls vs. Regularization μ
  35. 35. 35 March’18 Weak Scalability 30000 grid cells per core
  36. 36. 36 3D ERT 64 electrodes per line
  37. 37. 37 June 2016 ERT with Multi-Grid grid #nodes rel. time rel. size AMG speedup vs PCG 51 140608 1.00 1.00 5.79 99 1000000 6.57 7.11 13.39 151 3511808 23.57 24.98 20.44 199 8000000 52.29 56.90 219 10648000 71.43 75.73 3 level AMG on four core Time/size =const! Codd & Gross 2018
  38. 38. 38 June 2016 Structural Joint Inversion ● Invert for density and susceptibility ● spatial change should go together somehow Gravity Anomalies QLD: Magnetic field intensity QLD
  39. 39. 39 March’18 Cost Funtion J (m)=D1(ϕ1)+μ1 R(m1)⏟ grav +D2(ϕ2)+μ2 R(m2)⏟ mag +νc⋅∫C(m1 ,m2)⏟ cross gradient gravity −Δ ϕ1=4 πG⋅ρref m1 magnetic −Δϕ2=∇(kref e m2 B)
  40. 40. 40 March’18 Cross Gradient function “Where density and susceptibility change they change in the same direction.” ∇ m1 density contour normal susceptibility contour normal ∇ m2 C(m1, m2)=∣∇ m1×∇ m2∣ 2 minimize angle between contour normals
  41. 41. 41 March’18 Example Measure of correlation: M= ∇ m1⋅∇ m1 |∇ m1||∇ m2| ∈[−1,1]
  42. 42. 42 March’18 Example Results (vertical) m1 , νc=0 m2 , νc=0 m1 , νc=10 8 μ m2 , νc=108 μ M 2 , νc=0 M2 , νc=108 μ
  43. 43. 43 March’18 Can we do better? ∫C(m1 ,m2)∝R(m1)⋅R(m2) small vs. regularization C(2) (m1 ,m2)= ( 1 |∇ m1| 2 + 1 |∇ m2| 2 )|∇ m1×∇ m2|2
  44. 44. 44 March’18 Example again m1 , νc=100μ m2 , νc=100μ M2 , νc=100μ allways works: νc=100μ
  45. 45. 45 March’18 QLD Data (vertical) m1 , νc=100μ m1 , νc=100μ M 2 , νc=100μ 30 Million cells on ~200 cores
  46. 46. 46 March’18 What is next ● Current focus on ERT-type methods ● Simulation-Geophysical Signal coupling ● CCSRDF Sequestration projects with Uni Melbourne and Uni Savoy Monte Blanc ● EM/MT problems ● More on seismic & FWI

×