Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Paulm visualisation-superd

31 views

Published on

Paul Melis, group leader Visualization SURFsara:
Visualizing large cell-based simulations: a bloody mess?
A case study of different visualization approaches applied to data from cell-based simulations of blood by the University of Amsterdam. From classical scientific visualization, through information visualization using a custom tool to interactively visualizing millions of cells. Plus what it takes to turn the simulation of data into beautiful animations worthy of an IMAX theater.

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Paulm visualisation-superd

  1. 1. VISUALIZING LARGE CELL-BASED SIMULATIONS: A BLOODY MESS? PAUL MELIS (& CASPER VAN LEEUWEN) SURFSARA
  2. 2. Dutch national supercomputing center SURFsara supports (scienti c) research in the Netherlands with advanced ICT services: high-performance computing data storage visualisation networking e-Science Users are Dutch universities, research institutes, but also companies SURFsara has around 110 employees "Our" 3astructure located in Amsterdam Data Tower SURFSARA
  3. 3. Science Modeling, parallel computations, ... And a little viz   Compute infrastructure & support Visualization infra, support & tooling Animation production DIVISION OF LABOR
  4. 4. CONTEXT Computational biomedicine research University of Amsterdam Section Computational Science Dr. Gábor Závodszky, Prof. Alfons Hoekstra, ... CompBioMed project
  5. 5.        
  6. 6. HEMOCELL: MODELING & SIMULATION OF CELLULAR MODELS HTTPS://WWW.HEMOCELL.EU/ RBC: 1,280 triangles    
  7. 7. A COLLECTION OF CELLS IN PLASMA RBC: 1,280 triangles PLT: 128 triangles Deformation model based on physics and data from literature Distributed computation of thousands of cells 0:00 / 0:13 0:00 / 0:03
  8. 8. QUITE A FEW CELLS IN A BIT OF BLOOD (Placement of cells without overlap took one week compute time @ 32 cores)
  9. 9. Not just geometry, but also per-vertex values (like force and velocity) and per-cell values (like pressure and volume). 6,000 RBCs + 500 PLTs × 288 compute processes × 500 simulation timesteps ↓ ± 830,000 les ± 500 GB data All data saved to HDF5 le format QUITE A BIT OF DATA FROM A SIMULATION
  10. 10. ANALYSIS USING (SCIENTIFIC) VISUALISATION
  11. 11. INFOVIZ IN PARAVIEW? OR EVEN VISUAL ANALYTICS?
  12. 12. JUPYTER NOTEBOOK 0:00 / 0:42
  13. 13. WRITE CUSTOM VISUALISATION TOOL (D3.JS, SQLITE) 0:00 / 0:44
  14. 14. Di erent tools for di erent uses Each has their own strengths and weaknesses Might have to convert data...
  15. 15. THE VIRTUAL HUMAN MOVIE SHOWCASING COMPBIOMED RESEARCH ACTIVITIES Total animation length: about 5 minutes Simulations from UCL, UvA, ITMO, BSC, ... Script, art references, lighting, camera movements, ... BSC (Barcelona supercomputing center) in the lead: Overall direction, production, art direction, etc. SURFsara: Production of UvA cell-based blood simulation sequence DISPLAY AT SCIENCE MUSEUM LONDON ON AN IMAX SCREEN...
  16. 16. MATERIALS AND SHADING Subsurface scattering, bump mapping, fresnel shading, ... HDR light probe as single light source X-ray shader for making certain elements semi-transparent    
  17. 17. SIMULATION DATA? Several large simulation datasets (couple of TBs in total) Wanted to use original simulation output as much as possible in work ow Didn't want to convert it all to a di erent format just for visualization → Read directly from HDF5 les BLENDER
  18. 18. blender_import.py Loading geometry from HDF5 les (using h5py module) Geometry de-duplication (remove overlap, using numpy and some C++) Merging per-processor parts Filter geometry (by cell position or cell ID) Place cells side-by-side ... BLENDER PYTHON SCRIPTING TO THE RESCUE! $ blender -P blender_import.py -- [options] <hdf5-files> Options: -p One Blender object per part file -c One Blender object per cell -s All geometry in a single Blender object -w Put cells side-by-side ("wall") -r Assign random colored materials to parts $ blender -P blender_import.py -- -r RBC.01000000.p.*.h5
  19. 19. RENDERING ON CARTESIUS (THE DUTCH NATIONAL SUPERCOMPUTER) 47,776 CPU cores 130 TB memory 7.7 TB storage    1.843 P op/s
  20. 20. All jobs run through Batch system (SLURM) Can only allocate full nodes Usually there is room for short (< 1 hour) jobs Nice if there are lots of idle nodes Can render all frames in an animation in parallel if lucky :) ... but not always room Might have to wait, just like our other users render.py SLURM batch le (render.job) Submit job for rendering RENDERING ON A SUPERCOMPUTER (CARTESIUS) $ blender -P render.py -- <frame> -c <camera> Camera to use -f <format> Output file format, png or exr -g Use grey material override -p Use preview sampling quality -r Render in 50% reduced resolution #!/bin/sh #SBATCH -p short #SBATCH -t 59:00 #SBATCH -C ivy #SBATCH --job-name stentflow3 module load eb module load Blender/2.78c-intel-2016b-Python-3.6.1-pm module load h5py/2.7.0-intel-2016b-Python-3.6.1 frame=$SLURM_ARRAY_TASK_ID blender -b -P ./render.py -- "$@" $frame # Frames 1-150, full quality, save to OpenEXR $ sbatch --array=1-150 render.job # Different camera $ sbatch --array=1-150 render.job -c overview # Low resolution, lower quality, grey material override, PNG $ sbatch --array=1-150 render.job -r -p -g -f png
  21. 21. 1,620 (thin) CPU nodes 24 CPU cores → 24 SBU/hour 64 GPU nodes, each with 2x NVIDIA Tesla K40m 16 CPU cores × 3 → 48 SBU/hour → GPU rendering cheaper if at least 2× faster stent- ow 3 scene frame @ 2048x1080 pixels Blender 2.79 CPU (haswell) GPU   23:35.82 13:47.81 ≈ 1.71 × faster 9.44 SBU 11.04 SBU ≈ 17% more expensive CPU OR GPU RENDERING?
  22. 22. SHOWTIME @ LONDON SCIENCE MUSEUM LATES (27/9)
  23. 23. "... WORTHY OF AN IMAX THEATER." 0:00 / 1:08
  24. 24. RENDERING 8M CELLS @ 1,280 TRIANGLES PER CELL All cells present (even ones on the "inside") All cells of same type (RBC, PLT) have same geometry → Instancing will do the trick!
  25. 25. Hacked XML parser to add new cells element C++ (cells implementation) Render! 10,025,244,672 triangles HACKING STANDALONE CYCLES TO CREATE THE SCENE (INSTANCED GEOMETRY) <state interpolation="smooth" shader="rbc"> <cells scene="cells.bin" type="rbc" n="-1" geometry="rbc_normal_translated.bin" /> </state> <state interpolation="smooth" shader="plt"> <cells scene="cells.bin" type="plt" n="-1" geometry="plt_normal_translated.bin" /> </state> Mesh *mesh = read_cell_mesh(...geometry...); /* Read cell geometry once */ for (int i = 0; i < n; i++) { /* ... read next cell's position and orientation from cells.bin ... */ /* Create object (= instance of the mesh) */ Object *object = new Object(); object->mesh = mesh; object->tfm = state.tfm * transform_translate(make_float3(tx, ty, tz)) * transform_rotate(DEG2RADF(rz), make_float3(0,0,1)) * transform_rotate(DEG2RADF(ry), make_float3(0,1,0)) * transform_rotate(DEG2RADF(rx), make_float3(1,0,0)); scene->objects.push_back(object); } $ cycles --device CPU --output render.png --samples 64 cycles_scene.xml
  26. 26. 0:00 / 1:53
  27. 27. TO SUMMARIZE
  28. 28. THANKS FOR THE ATTENTION! Questions, remarks, ...? This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 675451 (CompBioMed project). The disseminated results re ect the authors views only. The European Commission is not responsible for any use that may be made of the information this presentation contains.             

×