Big Data Visualization With ParaView

  • 2,085 views
Uploaded on

ParaView is an open-source graphical user interface for VTK with additional functionality including the capability to perform rendering in parallel and a client-server architecture enabling …

ParaView is an open-source graphical user interface for VTK with additional functionality including the capability to perform rendering in parallel and a client-server architecture enabling visualization and analysis to be performed on a server while being viewed and driven from a client. ParaView, like VTK, is open-sourced under a BSD license and its development is overseen by the commercial entity, Kitware, Inc. ParaView is multi-platform, extensible via its plugin architecture, and natively supports many common data analysis tasks and data formats. As it builds upon VTK, any VTK functionality can in principle be invoked. In practice not all VTK functionality is exposed by default but can easily be exposed or extended via the plugin architecture previously mentioned and discussed in more detail below. Exposing VTK functionality is as easy as writing a short XML file. In this talk I present the process of plugging into ParaView to do visualization and analysis of terabytes of data in real time.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • This talk was held at the 7th meeting on May 13 at IBM Zurich by Christine Corbett Moran, University of Zurich.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
2,085
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
1
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. PARAVIEW AND ASTROVIZParallelVisualizationTools for the MassesChristine Corbett MoranInstitute forTheoretical Physics
  • 2. GOALS OFTHISTALK• Review the state-of-the-art in parallel analysis and visualization• Introduce the ParaView plugin AstroViz• Focus on generalized techniques of extending ParaView forcustom use cases• Provide performance numbers and use cases
  • 3. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  • 4. VISUALIZATION GOALS• Tool for insight, communication, comparison, appreciation
  • 5. ABSTRACT PROCESS
  • 6. ABSTRACT PROCESS
  • 7. ABSTRACT PROCESS
  • 8. ABSTRACT PROCESS
  • 9. DETAILED PROCESS• Filtering• Representation• Perception
  • 10. VISUALIZATION ANDANALYSIS IN ASTROPHYSICS• Parallel capabilities• Wide variety of dataformats• Extensible• Robust• Short learning curveSimAm AstroMD Salsa TipsyFlashViewParaView python VisItSPLASH IDL MayaVi Splotchyt gnuplot IFrIT OpenDX
  • 11. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  • 12. VTK• Open source, multi-platform visualization application• Supports distributed computation models• Extensible modular architecture
  • 13. VTK AND PARAVIEW• Open-source, multi-platform visualization applications• ParaView is UI on top of theVTK framework• Easily extensible via plugin development model
  • 14. • ParaView representsdata as collections ofpoints and cells• Calculations of filterson these points andcells largelyindependent• ParaView uses dataparallelismPARALLEL COMPUTING INPARAVIEW
  • 15. PARALLEL CAPABILITIES:READERS• Structured data: eachprocess reads a specificcoordinate extent• Unstructured data: eachprocess reads a specificfraction of the file
  • 16. PARALLEL CAPABILITIES: DATADISTRIBUTION• ParaView distributes dataspatially and evenly• Structured data: doneautomatically• Unstructured data: donemanually with D3 filter
  • 17. PARALLEL CAPABILITIES:FILTERS• Each process performssame operation on itspiece of data• Where necessary ghostcells are utilized
  • 18. PARALLEL CAPABILITIES:RENDERING• ParaView uses IceTparallel rendering library• Each process creates animage based on itspartition of geometry• Processes collectivelycomposite images
  • 19. D3 FILTERBefore After
  • 20. GHOST CELLS:EXTERNAL FACES
  • 21. PARALLEL RENDERINGWITH ICETIceT Users’ Guide and Reference, Kenneth Moreland 2009
  • 22. PARAVIEW PLUGINS• Easier to create and deploy than directly modifying sourcecode• Interface for developers to write modular software projectsand compile them against ParaView to produce libraries• User can load any number of these libraries at runtime
  • 23. PLUGIN DEVELOPMENTREQUIREMENTS• ParaView build and source used to compile it• C++ source code to implement plugin’s features• XML files to allow ParaView to use plugin• CMakeLists.txt to set up plugin’s build environment
  • 24. PLUGINTYPES• Server-side• Readers and writers• Filters• Client-side• Readers and writers• Object panels• Toolbars• Custom views
  • 25. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  • 26. GUIParaView AstroViz• Manipulate view in 3D• Split view• Link views• Variety of display choices• Additional menu, submenusand buttons for ease of useof AstroViz features• Each AstroViz feature has anoption panel with usercustomizable options
  • 27. GUI INTERACTION
  • 28. REALTIME INTERACTION
  • 29. FILE FORMATSParaView AstroViz• VTK• Comma separated values• XDMF• Saved ParaView state• Tipsy binary• Marked particle files• Additional ASCII attributes
  • 30. FILE FORMATS:C++ IMPLEMENTATION
  • 31. FILE FORMATS:GUI IMPLEMENTATION
  • 32. PARALLEL READING• Each process reads from toif and to otherwise
  • 33. DATA REDUCTIONParaView AstroViz• Reduce data to particlesselected in GUI• Export selected particles andtheir associated data• Threshold based on inputattributes or simple functionsof them• Option to load in only certainattributes of particles• Read in particles based on“marked” file• Threshold based on quantitiescomputed with AstroViz’s additionalanalysis capabilities• Halo finding
  • 34. DATA REDUCTION
  • 35. HALO FINDING• A halo is a group of gravitationally bound objects• Friends-of-Friends (FOF) halo finding algorithm• A FOF halo is set of objects for which every object in theset is within a linking length scale from at least one otherobject in the set• Purely geometric• One free parameter, the linking length
  • 36. ANALYSISParaView AstroViz• Simple functions of input data• Plot data• Statistics• Python scripting• Smooth quantities• Compute the virial radius• Center of mass• Cumulative quantities within a radius• Radial, tangential, and circularvelocity• Angular momentum• Principle moments of inertia
  • 37. ANALYSIS:FULL SCREENSHOT
  • 38. KD-TREES• Offers efficient way to• Locate a point in space• Identify N-nearest neighbors• Locate points within a given spatialareaFigure reproduced from Dr. BenjaminTyner, Department ofStatistics, Purdue University, 2009.• Build Kd-Tree data structure
  • 39. KD-TREE
  • 40. VIRIAL RADIUS• Formally the virial radius the radius at which virial equilibriumholds• Is often approximated as(tensor) (scalar with )
  • 41. CENTER OF MASS• Summation operation iscommutative• Processes calculate summationindependently• Results combined and finaldivision performed
  • 42. PRINCIPLE MOMENTS OFINERTIA
  • 43. PROFILE• Calculate various physicalquantities as a function of radius• Proceeds by grouping particlesinto equal radius bins
  • 44. PROFILED QUANTITIES
  • 45. PARALLEL CAPABILITIESParaView AstroViz• Read certain native formats inparallel• Parallel rendering• Data distribution• Certain analysis and datareduction operationsautomatically done in parallel• All AstroViz features capableof running in parallel
  • 46. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  • 47. • Quantify how much faster an algorithm will run on parallelcomputer with N processes• Where is the time it takes to execute the algorithm onprocessesPARALLEL COMPUTING
  • 48. • Every non-trivial algorithm has some serial component• As processes are added to the computation of a parallelalgorithm, a decreasing return is seen on algorithmic speed• If is the fraction of the algorithm that is inherently serialthen the algorithm will instead have the following speedup,upper bounded by , known as Amdahl’s law:PARALLEL COMPUTING
  • 49. PERFORMANCE BENCHMARKS• GHALO• B1, B2, B3 resolutions with 11 million, 141 million, and 3billion particles respectively• Machines• ZBox3 CPU cluster• Horus GPU cluster
  • 50. MACHINES• ZBox3 CPU Cluster• 144 nodes with quad core 2.4 GHZ Intel CPUS, 8GB mainmemory per node• Dolphin SCI highspeed interconnects• Horus GPU/CPU Cluster• 16 nodes 2.4 GHZ AMD Opteron Processor 250, 192 GBtotal memory• two NVIDIA Quadro FX 4500 GPUS per node
  • 51. READING
  • 52. ANALYSIS
  • 53. INTERACTIVEVISUALIZATION
  • 54. INTERACTIVEVISUALIZATION
  • 55. LARGE PARTICLE NUMBER
  • 56. LARGE PARTICLE NUMBER
  • 57. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  • 58. DATA REDUCTION
  • 59. DATA REDUCTION
  • 60. DATA REDUCTION
  • 61. DATA REDUCTION
  • 62. DATA REDUCTION
  • 63. DATA REDUCTION
  • 64. DATA REDUCTION
  • 65. REALTIME ANALYSIS
  • 66. REALTIME ANALYSIS
  • 67. REALTIME ANALYSIS
  • 68. REALTIME ANALYSIS
  • 69. REALTIME ANALYSIS
  • 70. SCRIPTED ANALYSIS
  • 71. MOVIES
  • 72. MOVIES
  • 73. •Visualization•VTK and ParaView• AstroViz• Performance• Usage• Next stepsOUTLINE
  • 74. ASTROVIZ: NEXT STEPS• Increased support for hydrodynamical codes (both AMR andSPH) and their common analysis tasks• Increased support for observational astrophysics visualizationand analysis tasks including the FITS format and pixel basedoperations• Performance optimizations and additional file formats• Features suggested by community
  • 75. GOALS OFTHISTALK: REVIEW• Review the state-of-the art in parallel analysis and visualization• Introduce the ParaView plugin AstroViz• Provide performance numbers and use cases
  • 76. RELEASE INFORMATION• Open source BSD license• Website• http://www.itp.uzh.ch/~corbett/astroviz/astroviz.html• Feedback and feature requests• http://astroviz.uservoice.com/
  • 77. QUESTIONS?