1.
VISUALIZING LARGE CELL-BASED
SIMULATIONS: A BLOODY MESS?
PAUL MELIS
(& CASPER VAN LEEUWEN)
SURFSARA

2.
Dutch national supercomputing center
SURFsara supports (scienti c) research in the Netherlands
with advanced ICT services:
high-performance computing
data storage
visualisation
networking
e-Science
Users are Dutch universities, research institutes, but also companies
SURFsara has around 110 employees
"Our" 3astructure located in Amsterdam Data Tower
SURFSARA

3.
Science
Modeling, parallel computations, ...
And a little viz
Compute infrastructure & support
Visualization infra, support & tooling
Animation production
DIVISION OF LABOR

4.
CONTEXT
Computational biomedicine research
University of Amsterdam
Section Computational Science
Dr. Gábor Závodszky, Prof. Alfons Hoekstra, ...
CompBioMed project

5.

6.
HEMOCELL: MODELING & SIMULATION OF CELLULAR MODELS
HTTPS://WWW.HEMOCELL.EU/
RBC: 1,280 triangles

7.
A COLLECTION OF CELLS IN PLASMA
RBC: 1,280 triangles
PLT: 128 triangles
Deformation model based on
physics and data from
literature
Distributed computation of
thousands of cells
0:00 / 0:13 0:00 / 0:03

8.
QUITE A FEW CELLS IN A BIT OF BLOOD
(Placement of cells without overlap took one week compute time @ 32 cores)

9.
Not just geometry, but also
per-vertex values (like force and velocity) and
per-cell values (like pressure and volume).
6,000 RBCs + 500 PLTs
×
288 compute processes
×
500 simulation timesteps
↓
± 830,000 les
± 500 GB data
All data saved to HDF5 le format
QUITE A BIT OF DATA FROM A SIMULATION

10.
ANALYSIS USING (SCIENTIFIC) VISUALISATION

11.
INFOVIZ IN PARAVIEW?
OR EVEN VISUAL ANALYTICS?

12.
JUPYTER NOTEBOOK
0:00 / 0:42

13.
WRITE CUSTOM VISUALISATION TOOL
(D3.JS, SQLITE)
0:00 / 0:44

14.
Di erent tools for di erent uses
Each has their own strengths and weaknesses
Might have to convert data...

15.
THE VIRTUAL HUMAN
MOVIE SHOWCASING COMPBIOMED RESEARCH ACTIVITIES
Total animation length: about 5 minutes
Simulations from UCL, UvA, ITMO, BSC, ...
Script, art references, lighting, camera movements, ...
BSC (Barcelona supercomputing center) in the lead:
Overall direction, production, art direction, etc.
SURFsara:
Production of UvA cell-based blood simulation sequence
DISPLAY AT SCIENCE MUSEUM LONDON ON AN IMAX SCREEN...

16.
MATERIALS AND SHADING
Subsurface scattering, bump mapping, fresnel shading, ...
HDR light probe as single light source
X-ray shader for making certain elements semi-transparent

17.
SIMULATION DATA?
Several large simulation datasets (couple of TBs in
total)
Wanted to use original simulation output as much
as possible in work ow
Didn't want to convert it all to a di erent format
just for visualization
→ Read directly from HDF5 les
BLENDER

18.
blender_import.py
Loading geometry from HDF5 les
(using h5py module)
Geometry de-duplication
(remove overlap, using numpy and some
C++)
Merging per-processor parts
Filter geometry (by cell position or cell
ID)
Place cells side-by-side
...
BLENDER PYTHON SCRIPTING TO THE RESCUE!
$ blender -P blender_import.py -- [options] <hdf5-files>
Options:
-p One Blender object per part file
-c One Blender object per cell
-s All geometry in a single Blender object
-w Put cells side-by-side ("wall")
-r Assign random colored materials to parts
$ blender -P blender_import.py -- -r RBC.01000000.p.*.h5

19.
RENDERING ON CARTESIUS
(THE DUTCH NATIONAL SUPERCOMPUTER)
47,776 CPU cores
130 TB memory
7.7 TB storage
1.843 P op/s

20.
All jobs run through Batch system (SLURM)
Can only allocate full nodes
Usually there is room for short (< 1 hour) jobs
Nice if there are lots of idle nodes
Can render all frames in an animation
in parallel if lucky :)
... but not always room
Might have to wait, just like our other users
render.py
SLURM batch le (render.job)
Submit job for rendering
RENDERING ON A SUPERCOMPUTER (CARTESIUS)
$ blender -P render.py -- <frame>
-c <camera> Camera to use
-f <format> Output file format, png or exr
-g Use grey material override
-p Use preview sampling quality
-r Render in 50% reduced resolution
#!/bin/sh
#SBATCH -p short
#SBATCH -t 59:00
#SBATCH -C ivy
#SBATCH --job-name stentflow3
module load eb
module load Blender/2.78c-intel-2016b-Python-3.6.1-pm
module load h5py/2.7.0-intel-2016b-Python-3.6.1
frame=$SLURM_ARRAY_TASK_ID
blender -b -P ./render.py -- "$@" $frame
# Frames 1-150, full quality, save to OpenEXR
$ sbatch --array=1-150 render.job
# Different camera
$ sbatch --array=1-150 render.job -c overview
# Low resolution, lower quality, grey material override, PNG
$ sbatch --array=1-150 render.job -r -p -g -f png

21.
1,620 (thin) CPU nodes
24 CPU cores → 24 SBU/hour
64 GPU nodes, each with 2x NVIDIA Tesla K40m
16 CPU cores × 3 → 48 SBU/hour
→ GPU rendering cheaper if at least 2× faster
stent- ow 3 scene
frame @ 2048x1080 pixels
Blender 2.79
CPU (haswell) GPU
23:35.82 13:47.81 ≈ 1.71 × faster
9.44 SBU 11.04 SBU ≈ 17% more expensive
CPU OR GPU RENDERING?

22.
SHOWTIME @ LONDON SCIENCE MUSEUM LATES (27/9)

23.
"... WORTHY OF AN IMAX THEATER."
0:00 / 1:08

24.
RENDERING 8M CELLS @ 1,280 TRIANGLES PER CELL
All cells present (even ones on the "inside")
All cells of same type (RBC, PLT) have same geometry
→ Instancing will do the trick!

25.
Hacked XML parser to add new cells element
C++ (cells implementation)
Render!
10,025,244,672 triangles
HACKING STANDALONE CYCLES TO CREATE THE SCENE (INSTANCED GEOMETRY)
<state interpolation="smooth" shader="rbc">
<cells scene="cells.bin" type="rbc" n="-1" geometry="rbc_normal_translated.bin" />
</state>
<state interpolation="smooth" shader="plt">
<cells scene="cells.bin" type="plt" n="-1" geometry="plt_normal_translated.bin" />
</state>
Mesh *mesh = read_cell_mesh(...geometry...); /* Read cell geometry once */
for (int i = 0; i < n; i++)
{
/* ... read next cell's position and orientation from cells.bin ... */
/* Create object (= instance of the mesh) */
Object *object = new Object();
object->mesh = mesh;
object->tfm =
state.tfm
* transform_translate(make_float3(tx, ty, tz))
* transform_rotate(DEG2RADF(rz), make_float3(0,0,1))
* transform_rotate(DEG2RADF(ry), make_float3(0,1,0))
* transform_rotate(DEG2RADF(rx), make_float3(1,0,0));
scene->objects.push_back(object);
}
$ cycles --device CPU --output render.png --samples 64 cycles_scene.xml

26.
0:00 / 1:53

27.
TO SUMMARIZE

28.
THANKS FOR THE ATTENTION!
Questions, remarks, ...?
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement
No 675451 (CompBioMed project). The disseminated results re ect the authors views only. The European Commission is not responsible for any use that may be made of the information this presentation contains.