PARAVIEW AND ASTROVIZParallelVisualizationTools for the MassesChristine Corbett MoranInstitute forTheoretical Physics
GOALS OFTHISTALK• Review the state-of-the-art in parallel analysis and visualization• Introduce the ParaView plugin AstroV...
•Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
VISUALIZATION GOALS• Tool for insight, communication, comparison, appreciation
ABSTRACT PROCESS
ABSTRACT PROCESS
ABSTRACT PROCESS
ABSTRACT PROCESS
DETAILED PROCESS• Filtering• Representation• Perception
VISUALIZATION ANDANALYSIS IN ASTROPHYSICS• Parallel capabilities• Wide variety of dataformats• Extensible• Robust• Short l...
•Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
VTK• Open source, multi-platform visualization application• Supports distributed computation models• Extensible modular ar...
VTK AND PARAVIEW• Open-source, multi-platform visualization applications• ParaView is UI on top of theVTK framework• Easil...
• ParaView representsdata as collections ofpoints and cells• Calculations of filterson these points andcells largelyindepen...
PARALLEL CAPABILITIES:READERS• Structured data: eachprocess reads a specificcoordinate extent• Unstructured data: eachproce...
PARALLEL CAPABILITIES: DATADISTRIBUTION• ParaView distributes dataspatially and evenly• Structured data: doneautomatically...
PARALLEL CAPABILITIES:FILTERS• Each process performssame operation on itspiece of data• Where necessary ghostcells are uti...
PARALLEL CAPABILITIES:RENDERING• ParaView uses IceTparallel rendering library• Each process creates animage based on itspa...
D3 FILTERBefore After
GHOST CELLS:EXTERNAL FACES
PARALLEL RENDERINGWITH ICETIceT Users’ Guide and Reference, Kenneth Moreland 2009
PARAVIEW PLUGINS• Easier to create and deploy than directly modifying sourcecode• Interface for developers to write modula...
PLUGIN DEVELOPMENTREQUIREMENTS• ParaView build and source used to compile it• C++ source code to implement plugin’s featur...
PLUGINTYPES• Server-side• Readers and writers• Filters• Client-side• Readers and writers• Object panels• Toolbars• Custom ...
•Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
GUIParaView AstroViz• Manipulate view in 3D• Split view• Link views• Variety of display choices• Additional menu, submenus...
GUI INTERACTION
REALTIME INTERACTION
FILE FORMATSParaView AstroViz• VTK• Comma separated values• XDMF• Saved ParaView state• Tipsy binary• Marked particle files...
FILE FORMATS:C++ IMPLEMENTATION
FILE FORMATS:GUI IMPLEMENTATION
PARALLEL READING• Each process reads from toif and to otherwise
DATA REDUCTIONParaView AstroViz• Reduce data to particlesselected in GUI• Export selected particles andtheir associated da...
DATA REDUCTION
HALO FINDING• A halo is a group of gravitationally bound objects• Friends-of-Friends (FOF) halo finding algorithm• A FOF ha...
ANALYSISParaView AstroViz• Simple functions of input data• Plot data• Statistics• Python scripting• Smooth quantities• Com...
ANALYSIS:FULL SCREENSHOT
KD-TREES• Offers efficient way to• Locate a point in space• Identify N-nearest neighbors• Locate points within a given spat...
KD-TREE
VIRIAL RADIUS• Formally the virial radius the radius at which virial equilibriumholds• Is often approximated as(tensor) (s...
CENTER OF MASS• Summation operation iscommutative• Processes calculate summationindependently• Results combined and finaldi...
PRINCIPLE MOMENTS OFINERTIA
PROFILE• Calculate various physicalquantities as a function of radius• Proceeds by grouping particlesinto equal radius bins
PROFILED QUANTITIES
PARALLEL CAPABILITIESParaView AstroViz• Read certain native formats inparallel• Parallel rendering• Data distribution• Cer...
•Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
• Quantify how much faster an algorithm will run on parallelcomputer with N processes• Where is the time it takes to execu...
• Every non-trivial algorithm has some serial component• As processes are added to the computation of a parallelalgorithm,...
PERFORMANCE BENCHMARKS• GHALO• B1, B2, B3 resolutions with 11 million, 141 million, and 3billion particles respectively• M...
MACHINES• ZBox3 CPU Cluster• 144 nodes with quad core 2.4 GHZ Intel CPUS, 8GB mainmemory per node• Dolphin SCI highspeed i...
READING
ANALYSIS
INTERACTIVEVISUALIZATION
INTERACTIVEVISUALIZATION
LARGE PARTICLE NUMBER
LARGE PARTICLE NUMBER
•Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
DATA REDUCTION
DATA REDUCTION
DATA REDUCTION
DATA REDUCTION
DATA REDUCTION
DATA REDUCTION
DATA REDUCTION
REALTIME ANALYSIS
REALTIME ANALYSIS
REALTIME ANALYSIS
REALTIME ANALYSIS
REALTIME ANALYSIS
SCRIPTED ANALYSIS
MOVIES
MOVIES
•Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
ASTROVIZ: NEXT STEPS• Increased support for hydrodynamical codes (both AMR andSPH) and their common analysis tasks• Increa...
GOALS OFTHISTALK: REVIEW• Review the state-of-the art in parallel analysis and visualization• Introduce the ParaView plugi...
RELEASE INFORMATION• Open source BSD license• Website• http://www.itp.uzh.ch/~corbett/astroviz/astroviz.html• Feedback and...
QUESTIONS?
Big Data Visualization With ParaView
Big Data Visualization With ParaView
Big Data Visualization With ParaView
Big Data Visualization With ParaView
Big Data Visualization With ParaView
Upcoming SlideShare
Loading in...5
×

Big Data Visualization With ParaView

2,626

Published on

ParaView is an open-source graphical user interface for VTK with additional functionality including the capability to perform rendering in parallel and a client-server architecture enabling visualization and analysis to be performed on a server while being viewed and driven from a client. ParaView, like VTK, is open-sourced under a BSD license and its development is overseen by the commercial entity, Kitware, Inc. ParaView is multi-platform, extensible via its plugin architecture, and natively supports many common data analysis tasks and data formats. As it builds upon VTK, any VTK functionality can in principle be invoked. In practice not all VTK functionality is exposed by default but can easily be exposed or extended via the plugin architecture previously mentioned and discussed in more detail below. Exposing VTK functionality is as easy as writing a short XML file. In this talk I present the process of plugging into ParaView to do visualization and analysis of terabytes of data in real time.

Published in: Technology, Education
1 Comment
3 Likes
Statistics
Notes
  • This talk was held at the 7th meeting on May 13 at IBM Zurich by Christine Corbett Moran, University of Zurich.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
2,626
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Big Data Visualization With ParaView"

  1. 1. PARAVIEW AND ASTROVIZParallelVisualizationTools for the MassesChristine Corbett MoranInstitute forTheoretical Physics
  2. 2. GOALS OFTHISTALK• Review the state-of-the-art in parallel analysis and visualization• Introduce the ParaView plugin AstroViz• Focus on generalized techniques of extending ParaView forcustom use cases• Provide performance numbers and use cases
  3. 3. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  4. 4. VISUALIZATION GOALS• Tool for insight, communication, comparison, appreciation
  5. 5. ABSTRACT PROCESS
  6. 6. ABSTRACT PROCESS
  7. 7. ABSTRACT PROCESS
  8. 8. ABSTRACT PROCESS
  9. 9. DETAILED PROCESS• Filtering• Representation• Perception
  10. 10. VISUALIZATION ANDANALYSIS IN ASTROPHYSICS• Parallel capabilities• Wide variety of dataformats• Extensible• Robust• Short learning curveSimAm AstroMD Salsa TipsyFlashViewParaView python VisItSPLASH IDL MayaVi Splotchyt gnuplot IFrIT OpenDX
  11. 11. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  12. 12. VTK• Open source, multi-platform visualization application• Supports distributed computation models• Extensible modular architecture
  13. 13. VTK AND PARAVIEW• Open-source, multi-platform visualization applications• ParaView is UI on top of theVTK framework• Easily extensible via plugin development model
  14. 14. • ParaView representsdata as collections ofpoints and cells• Calculations of filterson these points andcells largelyindependent• ParaView uses dataparallelismPARALLEL COMPUTING INPARAVIEW
  15. 15. PARALLEL CAPABILITIES:READERS• Structured data: eachprocess reads a specificcoordinate extent• Unstructured data: eachprocess reads a specificfraction of the file
  16. 16. PARALLEL CAPABILITIES: DATADISTRIBUTION• ParaView distributes dataspatially and evenly• Structured data: doneautomatically• Unstructured data: donemanually with D3 filter
  17. 17. PARALLEL CAPABILITIES:FILTERS• Each process performssame operation on itspiece of data• Where necessary ghostcells are utilized
  18. 18. PARALLEL CAPABILITIES:RENDERING• ParaView uses IceTparallel rendering library• Each process creates animage based on itspartition of geometry• Processes collectivelycomposite images
  19. 19. D3 FILTERBefore After
  20. 20. GHOST CELLS:EXTERNAL FACES
  21. 21. PARALLEL RENDERINGWITH ICETIceT Users’ Guide and Reference, Kenneth Moreland 2009
  22. 22. PARAVIEW PLUGINS• Easier to create and deploy than directly modifying sourcecode• Interface for developers to write modular software projectsand compile them against ParaView to produce libraries• User can load any number of these libraries at runtime
  23. 23. PLUGIN DEVELOPMENTREQUIREMENTS• ParaView build and source used to compile it• C++ source code to implement plugin’s features• XML files to allow ParaView to use plugin• CMakeLists.txt to set up plugin’s build environment
  24. 24. PLUGINTYPES• Server-side• Readers and writers• Filters• Client-side• Readers and writers• Object panels• Toolbars• Custom views
  25. 25. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  26. 26. GUIParaView AstroViz• Manipulate view in 3D• Split view• Link views• Variety of display choices• Additional menu, submenusand buttons for ease of useof AstroViz features• Each AstroViz feature has anoption panel with usercustomizable options
  27. 27. GUI INTERACTION
  28. 28. REALTIME INTERACTION
  29. 29. FILE FORMATSParaView AstroViz• VTK• Comma separated values• XDMF• Saved ParaView state• Tipsy binary• Marked particle files• Additional ASCII attributes
  30. 30. FILE FORMATS:C++ IMPLEMENTATION
  31. 31. FILE FORMATS:GUI IMPLEMENTATION
  32. 32. PARALLEL READING• Each process reads from toif and to otherwise
  33. 33. DATA REDUCTIONParaView AstroViz• Reduce data to particlesselected in GUI• Export selected particles andtheir associated data• Threshold based on inputattributes or simple functionsof them• Option to load in only certainattributes of particles• Read in particles based on“marked” file• Threshold based on quantitiescomputed with AstroViz’s additionalanalysis capabilities• Halo finding
  34. 34. DATA REDUCTION
  35. 35. HALO FINDING• A halo is a group of gravitationally bound objects• Friends-of-Friends (FOF) halo finding algorithm• A FOF halo is set of objects for which every object in theset is within a linking length scale from at least one otherobject in the set• Purely geometric• One free parameter, the linking length
  36. 36. ANALYSISParaView AstroViz• Simple functions of input data• Plot data• Statistics• Python scripting• Smooth quantities• Compute the virial radius• Center of mass• Cumulative quantities within a radius• Radial, tangential, and circularvelocity• Angular momentum• Principle moments of inertia
  37. 37. ANALYSIS:FULL SCREENSHOT
  38. 38. KD-TREES• Offers efficient way to• Locate a point in space• Identify N-nearest neighbors• Locate points within a given spatialareaFigure reproduced from Dr. BenjaminTyner, Department ofStatistics, Purdue University, 2009.• Build Kd-Tree data structure
  39. 39. KD-TREE
  40. 40. VIRIAL RADIUS• Formally the virial radius the radius at which virial equilibriumholds• Is often approximated as(tensor) (scalar with )
  41. 41. CENTER OF MASS• Summation operation iscommutative• Processes calculate summationindependently• Results combined and finaldivision performed
  42. 42. PRINCIPLE MOMENTS OFINERTIA
  43. 43. PROFILE• Calculate various physicalquantities as a function of radius• Proceeds by grouping particlesinto equal radius bins
  44. 44. PROFILED QUANTITIES
  45. 45. PARALLEL CAPABILITIESParaView AstroViz• Read certain native formats inparallel• Parallel rendering• Data distribution• Certain analysis and datareduction operationsautomatically done in parallel• All AstroViz features capableof running in parallel
  46. 46. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  47. 47. • Quantify how much faster an algorithm will run on parallelcomputer with N processes• Where is the time it takes to execute the algorithm onprocessesPARALLEL COMPUTING
  48. 48. • Every non-trivial algorithm has some serial component• As processes are added to the computation of a parallelalgorithm, a decreasing return is seen on algorithmic speed• If is the fraction of the algorithm that is inherently serialthen the algorithm will instead have the following speedup,upper bounded by , known as Amdahl’s law:PARALLEL COMPUTING
  49. 49. PERFORMANCE BENCHMARKS• GHALO• B1, B2, B3 resolutions with 11 million, 141 million, and 3billion particles respectively• Machines• ZBox3 CPU cluster• Horus GPU cluster
  50. 50. MACHINES• ZBox3 CPU Cluster• 144 nodes with quad core 2.4 GHZ Intel CPUS, 8GB mainmemory per node• Dolphin SCI highspeed interconnects• Horus GPU/CPU Cluster• 16 nodes 2.4 GHZ AMD Opteron Processor 250, 192 GBtotal memory• two NVIDIA Quadro FX 4500 GPUS per node
  51. 51. READING
  52. 52. ANALYSIS
  53. 53. INTERACTIVEVISUALIZATION
  54. 54. INTERACTIVEVISUALIZATION
  55. 55. LARGE PARTICLE NUMBER
  56. 56. LARGE PARTICLE NUMBER
  57. 57. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  58. 58. DATA REDUCTION
  59. 59. DATA REDUCTION
  60. 60. DATA REDUCTION
  61. 61. DATA REDUCTION
  62. 62. DATA REDUCTION
  63. 63. DATA REDUCTION
  64. 64. DATA REDUCTION
  65. 65. REALTIME ANALYSIS
  66. 66. REALTIME ANALYSIS
  67. 67. REALTIME ANALYSIS
  68. 68. REALTIME ANALYSIS
  69. 69. REALTIME ANALYSIS
  70. 70. SCRIPTED ANALYSIS
  71. 71. MOVIES
  72. 72. MOVIES
  73. 73. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  74. 74. ASTROVIZ: NEXT STEPS• Increased support for hydrodynamical codes (both AMR andSPH) and their common analysis tasks• Increased support for observational astrophysics visualizationand analysis tasks including the FITS format and pixel basedoperations• Performance optimizations and additional file formats• Features suggested by community
  75. 75. GOALS OFTHISTALK: REVIEW• Review the state-of-the art in parallel analysis and visualization• Introduce the ParaView plugin AstroViz• Provide performance numbers and use cases
  76. 76. RELEASE INFORMATION• Open source BSD license• Website• http://www.itp.uzh.ch/~corbett/astroviz/astroviz.html• Feedback and feature requests• http://astroviz.uservoice.com/
  77. 77. QUESTIONS?

×