VO Course 11: Spatial indexing

709 views

Published on

Introduction to the problem of spatial indexing for spherical coordinate systems, as those used in astronomy. Part of the virtual observatory course by Juan de Dios Santander Vela, as imparted for the MTAF (Métodos y Técnicas Avanzadas en Física, Advanced Methods and Techniques in Physics) Master at the University of Granada (UGR).

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
709
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

VO Course 11: Spatial indexing

  1. 1. Spatial IndexingJuan de Dios Santander Vela (IAA-CSIC)
  2. 2. Why Spatial Indexing?Spherical coordinates Wrap Around Singularities at the Poles Changing Width
  3. 3. Why Spatial Indexing?Spherical coordinates Wrap Around Singularities at the Poles Changing Width
  4. 4. Why Spatial Indexing?Spherical coordinates Wrap Around Singularities at the Poles Changing Width
  5. 5. Why Spatial Indexing?Query on Arbitrary Shapes Object Contours Instrument Footprints Region IntersectionsRegion operations integral to ADQL!
  6. 6. Using partitioningschemesQuery applications Spatial joinsX-Match applicationsDisplay applicationsCore spatial region support
  7. 7. Spherical Coordinates x = r sin q cos f y = r sin q sin f z = r cos q x = sin q cos f y = sin q sin f z = cos q
  8. 8. Typical Query Case: Cone SearchSELECT * FROM TARGET_TABLE WHERE 2 * ASIN( SQRT(SIN(($DEC_C-DEC)/2) * SIN(($DEC_C-DEC)/2) + COS($DEC_C) * COS(DEC) * SIN(($RA_C - RA)/2) * SIN(($RA_C - RA)/2))) <= $SR_CSELECT * FROM RASS_PHOTONS WHERE X*$X_C+Y*$Y_C+Z*$Z_C >= COS($SR_C)
  9. 9. Spatial Indexing ElementsHalf-Space A.K.A NT AI STR CON
  10. 10. Spatial Indexing Elements A POLYGON IS A UNION OF CONVEXESConvex
  11. 11. Spatial Indexing Elements LEVEL 0 LEVEL 1 LEVEL 2HTM Indexing &Trixels
  12. 12. Spatial Indexing ElementsHTM Indexing &Trixels
  13. 13. Indexing SchemesUnordered infinite plane versus ordered finiteplaneDifferent coordinate systems needed for differentthings Cartesian SphericalPlus, discontinuities in both!
  14. 14. Indexing SchemesEasy query/look-up of plane or sphericalcoordinate-based objectsHierarchical (suitable for wavelet, storagelocality…)Adaptability to transformations (sphericalharmonics, wavelet, FFT…)
  15. 15. Indexing SchemesPartitioning HEALPix HTM Q3C Mostly quad-tree-like schemes
  16. 16. HEALPix
  17. 17. HEALPixBy Górsky, Hivon, Banday, et al. (first ESO, laterJPL)Main concerns: Hierarchical structure Equal area partitioning Isolatitude distribution
  18. 18. The Astrophysical Journal, 622:759–771, 2005 April 1HEALPix # 2005. The American Astronomical Society. All rights reserved. Printed in U.S.A. HEALPix: A FRAMEWORK FOR HIGH-RESOLUTION DISCRETIZATION AND FAST ANALYSIS OF DATA DISTRIBUTED ON THE SPHERE K. M. Gorski,1, 2 E. Hivon,3 A. J. Banday,4 B. D. Wandelt,5, 6 F. K. Hansen,7 ´ M. Reinecke,4 and M. Bartelmann8 Received 2004 September 21; accepted 2004 December 10 ABSTRACT HEALPix—the Hierarchical Equal Area isoLatitude Pixelization—is a versatile structure for the pixelization of data on the sphere. An associated library of computational algorithms and visualization software supports fast scientific applications executable directly on discretized spherical maps generated from very large volumes of astronomical data. Originally developed to address the data processing and analysis needs of the present generation of cosmic microwave background experiments (e.g., BOOMERANG, WMAP), HEALPix can be expanded to meet many of the profound challenges that will arise in confrontation with the observational output of future missions and experiments, including, e.g., Planck, Herschel, SAFIR, and the Beyond Einstein inflation probe. In this paper weBy Górsky, Hivon, Banday, et al. (first ESO, later consider the requirements and implementation constraints on a framework that simultaneously enables an efficient discretization with associated hierarchical indexation and fast analysis/synthesis of functions defined on the sphere. We demonstrate how these are explicitly satisfied by HEALPix. Subject headingg: cosmic microwave background — cosmology: observations — methods: statistical sJPL) 1. INTRODUCTION 1. global analysis problems: harmonic decomposition, esti- mation of the power spectrum, and higher order measures of Advanced detectors in modern astronomy generate data at spatial correlations; huge rates over many wavelengths. Of particular interest to us 2. real space morphological analyses, object detection, iden- are those data sets that accumulate measurements distributed on tification, and characterization;Main concerns: the entire sky, or a considerable fraction thereof. Typical exam- 3. the simulation of models of the primary and foreground ples include radio, cosmic microwave background (CMB), sub- sky signals to study instrument performance and calibrate fore- millimeter, infrared, X-ray, and gamma-ray sky maps of diffuse ground separation and statistical inference methods; and emission, and full-sky or wide-area surveys of extragalactic ob- 4. spatial and /or spectral cross-correlation with external data jects. Together with this wealth of gathered information comes sets. an inevitable increase in complexity for data reduction and sci- ence extraction. In this paper we are focused on those issues These tasks, and many others, necessitate a careful definition of related to the distinctive nature of the spherical spatial domain the data models and proper construction of the mathematical Hierarchical structure over which the data reside. Our original motivations arose from framework for data analysis such that the algorithmic and com- work related to the measurement and interpretation of the CMB puting time requirements can be satisfied in order to achieve the anisotropy. The growing complexity of the associated science successful and timely scientific interpretation of the observa- extraction problem can be illustrated by the transition between tions. A particular method of addressing some of these issues is the data sets from various experiments: COBE Differential Mi- described next. crowave Radiometer (DMR) (early1990s, 7 FWHM resolution, $6000 pixel sky maps at three wavelengths), BOOMERANG 2. DISCRETIZED MAPPING AND ANALYSIS ( late 1990s, 120 FWHM, partial sky maps of $200,000 pixels OF FUNCTIONS ON THE SPHERE Equal area partitioning at four wavelengths), WMAP (early 2000s, resolution up to 140 FWHM, $3 million pixel sky maps at five wavelengths), and The analysis of functions on domains with a spherical to- Planck (data expected ca. 2009, resolution up to 50 FWHM, pology occupies a central place in both the physical sciences $50 million pixel sky maps at nine wavelengths). Science ex- and engineering disciplines. This is particularly apparent in the traction from these data sets involves the following: fields of astronomy, cosmology, geophysics, and atomic and nu- clear physics. In many cases the geometry is dictated either by 1 JPL /Caltech, MS 169-327, 4800 Oak Grove Drive, Pasadena, CA 91109. the object under study or by the need to assume and exploit ap- 2 Warsaw University Observatory, Aleje Ujazdowskie 4, 00-478 Warsaw, proximate spherical symmetry to utilize powerful perturbative Isolatitude distribution Poland. techniques. Practical limits for the purely analytical study of 3 IPAC, MS 100-22, Caltech, 1200 East California Boulevard, Pasadena, these problems create an urgent necessity for efficient and ac- CA 91125. 4 Max-Planck-Institut f ur Astrophysik, Karl-Schwarzschild-Strasse 1, curate numerical tools. ¨ Postfach 1317, D-85741 Garching bei Munchen, Germany. ¨ The simplicity of the spherical form belies the intricacy of 5 Department of Physics, University of Illinois, Urbana, IL 61801. global analysis on the sphere. There is no known point set that 6 Department of Astronomy, University of Illinois, 1002 West Green achieves the analog of uniform sampling in Euclidean space Street, Urbana, IL 61801. and allows exact and invertible discrete spherical harmonic de- 7 Institute of Theoretical Astrophysics, University of Oslo, P.O. Box 1029 Blindern, N-0315 Oslo, Norway. compositions of arbitrary but band-limited functions. All ex- 8 ITA, Universitat Heidelberg, Tiergartenstrasse 15, D-69121 Heidelberg, ¨ isting practical schemes proposed for the treatment of such Germany. discretized functions on the sphere introduce some (hopefully 759
  19. 19. HEALPix
  20. 20. HEALPix Astronomy Astrophysics manuscript no. mhg (DOI: will be inserted by hand later) June 20, 2005 Mapping on the HEALPix grid M. R. Calabretta1 and B. F. Roukema2 1 Australia Telescope National Facility, PO Box 76, Epping, NSW 1710, Australia 2 Toru´ Centre for Astronomy, N. Copernicus University, ul. Gagarina 11, PL-87-100 Toru´ , Poland n n Draft dated 2005/06/15 Abstract. The natural spherical projection associated with the Hierarchical Equal Area and isoLatitude Pixelisation, HEALPix, is described and shown to be one of a hybrid class that combines the cylindrical equal-area and Collignon projections, not previously documented in the cartographic literature. Projection equations are derived for the class in general and it is shown that the HEALPix projection suggests a simple method (a) of storing, and (b) visualising data sampled on the grid of the HEALPix pixelisation, and also suggests an extension of the pixelisation that is better suited for these purposes. Potentially useful properties of other members of the class are described, and new triangular and hexagonal pixelisations are constructed from them. Finally, the formalism is defined for representing the celestial coordinate system for any member of the class in the FITS data format. Key words. astronomical data bases: miscellaneous – cosmic microwave background – cosmology: observations – methods: data analysis, statistical – techniques: image processing 1. Introduction The Hierarchical Equal Area and isoLatitude Pixelisation, HEALPix (G´ rski et al. 2004), o↵ers a scheme for distributing o 12N 2 (N 2 ) points as uniformly as possible over the surface of the unit sphere subject to the constraint that the points lie on a relatively small number (4N 1) of parallels of latitude and are equispaced in longitude on each of these parallels. These 120 0 -120 properties were chosen to optimise spherical harmonic analy- 90 -90 sis and other computations performed on the sphere. 0 In fact, HEALPix goes further than simply defining a dis- 90 tribution of points; it also specifies the boundary between ad- 75 jacent points and does so in such a way that each occupies the 60 45 60 same area. Thus HEALPix is described as an equal area pix- 30 elisation. Pixels at the same absolute value of the latitude have 15 the same shape in the equatorial region, though pixel shape dif- 0 fers between latitudes, and with longitude in the polar regions. -15 -30 The boundaries for N = 1 define the 12 base-resolution pix- -45 els and higher-order pixelisations are defined by their regular -60 -60 180 90 -75 0 -90 -180 subdivision. Note, however, that although they are four-sided, HEALPix pixels are not spherical quadrilaterals because their 135 45 -45 -135 edges are not great circle arcs. Fig. 1. The HEALPix class of projections for H = 1, 2, 3 rescaled to HEALPix was originally described purely with reference unit area (top), and the nominative case with H = 4 (bottom) at twice to the sphere, the data itself being stored as a one-dimensional the linear scale and with the top left-hand corner of the graticule “cut array in a FITS binary table (Cotton et al. 1995) with either away” to reveal the underlying cylindrical equal-area projection in the equatorial region. Facets are shown as dashed diamonds. ring or nested organisation, the former being suited for spher- ical harmonic analysis and the latter for nearest-neighbour searches. For visualisation purposes the software that imple- Send o↵print requests to: M. Calabretta, ments HEALPix o↵ers a choice of four conventional projection e-mail: mcalabre@atnf.csiro.au types onto which HEALPix data may be regridded.
  21. 21. HEALPix
  22. 22. HEALPix HEALPix radians measured eastward. sphere are given by the foll North polar cap.— For i Nside , and the pixel-in-r qffiffi i¼I p j¼pþ z¼ ¼ jÀ 2i North equatorial belt.— F i 2Nside , and 1 j 4Ns i ¼ I ð p0 j ¼ ð p0 m z¼ Fig. 3.—Orthographic view of the HEALPix partition of the sphere. The s overplot of equator and meridians illustrates the octahedral symmetry of ¼ j À ; an HEALPix. Light gray shading shows one of the 8 (4 north and 4 south) identical 2 Nside 2
  23. 23. HEALPix HEALPix radians measured eastward. sphere are given by the foll North polar cap.— For i Nside , and the pixel-in-r qffiffi i¼I p j¼pþ z¼ ¼ jÀ 2i North equatorial belt.— F i 2Nside , and 1 j 4Ns i ¼ I ð p0 j ¼ ð p0 m z¼ Fig. 3.—Orthographic view of the HEALPix partition of the sphere. The s overplot of equator and meridians illustrates the octahedral symmetry of ¼ j À ; an HEALPix. Light gray shading shows one of the 8 (4 north and 4 south) identical 2 Nside 2
  24. 24. HEALPix HEALPix radians measured eastward. sphere are given by the foll North polar cap.— For i Nside , and the pixel-in-r qffiffi i¼I p j¼pþ z¼ ¼ jÀ 2i North equatorial belt.— F i 2Nside , and 1 j 4Ns i ¼ I ð p0 j ¼ ð p0 m z¼ Fig. 3.—Orthographic view of the HEALPix partition of the sphere. The s overplot of equator and meridians illustrates the octahedral symmetry of ¼ j À ; an HEALPix. Light gray shading shows one of the 8 (4 north and 4 south) identical 2 Nside 2
  25. 25. HEALPix
  26. 26. HEALPix 0011 0111 1011 1111
  27. 27. HEALPix 10100 10101 10110 10111
  28. 28. HEALPix
  29. 29. HEALPix Fig. 4.—Layout of the HEALPix pixels on the sphere in a cylindrical projection and a demonstration of two possible pixel indexations—one running onisolatitude rings, the other arranged hierarchically or in a nested tree fashion. The top two panels correspond to Nside ¼ 2 in first ring then nested schemes; the bottomtwo panels are for Nside ¼ 4. refers to the colatitude and to the longitude.
  30. 30. HEALPix FITS Astronomy Astrophysics manuscript no. mhg (DOI: will be inserted by hand later) June 20, 2005 Mapping on the HEALPix grid M. R. Calabretta1 and B. F. Roukema2 1 Australia Telescope National Facility, PO Box 76, Epping, NSW 1710, Australia 2 Toru´ Centre for Astronomy, N. Copernicus University, ul. Gagarina 11, PL-87-100 Toru´ , Poland n n Draft dated 2005/06/15 Abstract. The natural spherical projection associated with the Hierarchical Equal Area and isoLatitude Pixelisation, HEALPix, is described and shown to be one of a hybrid class that combines the cylindrical equal-area and Collignon projections, not previously documented in the cartographic literature. Projection equations are derived for the class in general and it is shown that the HEALPix projection suggests a simple method (a) of storing, and (b) visualising data sampled on the grid of the HEALPix pixelisation, and also suggests an extension of the pixelisation that is better suited for these purposes. Potentially useful properties of other members of the class are described, and new triangular and hexagonal pixelisations are constructed from them. Finally, the formalism is defined for representing the celestial coordinate system for any member of the class in the FITS data format. Key words. astronomical data bases: miscellaneous – cosmic microwave background – cosmology: observations – methods: data analysis, statistical – techniques: image processing 1. Introduction The Hierarchical Equal Area and isoLatitude Pixelisation, HEALPix (G´ rski et al. 2004), o↵ers a scheme for distributing o 12N 2 (N 2 ) points as uniformly as possible over the surface of the unit sphere subject to the constraint that the points lie on a relatively small number (4N 1) of parallels of latitude and are equispaced in longitude on each of these parallels. These 120 0 -120 properties were chosen to optimise spherical harmonic analy- 90 -90 sis and other computations performed on the sphere. 0 In fact, HEALPix goes further than simply defining a dis- 90 tribution of points; it also specifies the boundary between ad- 75 jacent points and does so in such a way that each occupies the 60 45 60 same area. Thus HEALPix is described as an equal area pix- 30 elisation. Pixels at the same absolute value of the latitude have 15 the same shape in the equatorial region, though pixel shape dif- 0 fers between latitudes, and with longitude in the polar regions. -15 -30 The boundaries for N = 1 define the 12 base-resolution pix- -45 els and higher-order pixelisations are defined by their regular -60 -60 180 90 -75 0 -90 -180 subdivision. Note, however, that although they are four-sided, HEALPix pixels are not spherical quadrilaterals because their 135 45 -45 -135 edges are not great circle arcs. Fig. 1. The HEALPix class of projections for H = 1, 2, 3 rescaled to HEALPix was originally described purely with reference unit area (top), and the nominative case with H = 4 (bottom) at twice to the sphere, the data itself being stored as a one-dimensional the linear scale and with the top left-hand corner of the graticule “cut array in a FITS binary table (Cotton et al. 1995) with either away” to reveal the underlying cylindrical equal-area projection in the equatorial region. Facets are shown as dashed diamonds. ring or nested organisation, the former being suited for spher- ical harmonic analysis and the latter for nearest-neighbour searches. For visualisation purposes the software that imple- Send o↵print requests to: M. Calabretta, ments HEALPix o↵ers a choice of four conventional projection e-mail: mcalabre@atnf.csiro.au types onto which HEALPix data may be regridded.
  31. 31. HEALPix FITS M. R. Calabretta: Mapping on the HEALPix grid 5 75 is very much an artifact of the distortions inherent in the pro- 60 45 jection. Because the -coordinate varies non-linearly with θ, 30 15 on the sphere they are actually biased to one side of the pixel. 0 Hence some degree of bias is unavoidable. -15 -30 -45 -60 -75 2.3.2. Hexagonal – H = 3 120 0 -120 The division into 3 × 120◦ suggests hexagonal base-resolution √ Fig. 5. HEALPix projection for H = 3 scaled in by 1/ 3 whereupon pixels. Although the familiar “honeycomb” structure shows it becomes three consecutive hexagons. that it is possible to tile the plane with hexagons, neverthe- less there is no bounded tesselation of hexagons by hexagons. That is, a hexagonal region may not be cut out of a honey- by equilateral triangles. This pixelisation may be defined by √ comb tesselation without cutting the individual elements. Thus rescaling the HEALPix projection with H = 6 by 3 in it may seem surprising that a hexagonal pixelisation can be con- so that the half-facets become equilateral triangles. Such a lin- structed from the HEALPix projection for H = 3 with scaled √ ear scaling does not affect the projection’s equal area property. by 1/ 3. The boundary of this projection, as seen in Fig. 5, is What were previously half-facets may now be identified with reduced to that of three sequential hexagons. This boundary is 36 new, triangular base-resolution pixels that may be subdi- then used conceptually as a “pie-cutter” on a honeycomb tesse- vided in a hierarchical way as for HEALPix, as depicted in lation of the right scale. Pixels that are cut can be made whole Fig. 4. It is interesting to note that this subdivision is natu- again by borrowing from adjacent facets, much as the square rally hierarchical - the number of pixels varies exponentially as pixelisation in Fig. 3 does. √ 36 × 4N−1 where N is the hierarchy level. In the H = 4 pixelisa- Rescaling Eqs. (13) and (14) gives H◦ / 3 = (2.7, 2.5, tion the exponential hierarchy must be engineered by doubling 2.0, 1.5, 1.5, 1.7, 1.8, 1.8) for θ = (0, 15, 30, θ×, 45, 60, 75, 90). N. Thus the rescaled H = 3 projection does not achieve confor- The conformal latitude computed for H = 6 with this mality at any latitude, it does well close to the equator, but ◦ extra -scaling is θ◦ = 30. 98, indicating that the projec- degrades at mid-latitudes. In the polar regions the 30◦ angle tion becomes conformal at the latitude that bisects the hemi- between meridians and parallels along the edge of the facets sphere by area. Applying Eqs. (13) and (14) with the extra √ is further from the ideal of 90◦ than the 45◦ for the unscaled scaling gives 3H◦ = (8.2, 7.6, 6.1, 4.5, 4.6, 5.1, 5.3, 5.4) for projections. θ = (0, 15, 30, θ×, 45, 60, 75, 90), again indicating less distor- tion in the polar regions than H = 4. It also does better in the polar zone away from the centreline because the 60◦ angle 2.4. HPX: HEALPix in FITS along the edge of the polar facets more closely matches the true In this section the HEALPix projections are described in the angle of 90◦ on the sphere. Overall, this pixelisation performs same terms as the projections defined in Calabretta Greisen adequately at low latitudes and does better than the H = 4 pix- (2002). elisations at mid to high latitudes. HEALPix projections will be denoted in FITS with algo- This rescaling of the H = 6 projection is reminiscent of rithm code HPX in the CTYPE ia keywords for the celestial axes. Tegmark’s (1996) icosahedral projection composed of 20 equi- As data storage has become much less of an issue in recent lateral triangles; the problem of indexing the subdivisions of its years we do not consider it necessary to create an analogue of triangular facets was solved in the implementation of the cor- the CUBEFACE keyword to cover HPX. However, if HEALPix responding pixelisation. In the present context the iso-latitude data is repackaged into the pseudo-quadcube layout shown property is still present but modified somewhat from the dia- in Fig. 3 the CUBEFACE storage mechanism is applicable for mond pixelisation of H = 4. However, if the pixel centres are H = 4 and will be treated properly by  (Calabretta, 1 moved up or down from the centroid by 12 of the height of 1995). the equilateral triangles to the point half-way between the base Since the HEALPix projections are constructed with the and apex then they fall onto a regular grid sampled more fre- origin of the native coordinate system at the reference point, quently in x than y. This provides some of the same benefits as we set the H = 4 double-pixelisation. (φ0 , θ0 )HEALPix = (0, 0). (15) However, although the displacement is small, there is a pos- sibility that it will introduce statistical biases so the full con- The projection equations and their inverses, re-expressed in the sequences should be investigated for a particular application. form required by FITS, are now summarised formally. These potential biases may be minimised by making the pixel In the equatorial zone where | sin θ | ≤ 2/3: size sufficiently small, and the fact that the bias between ad- x = φ, (16) jacent pairs of pixels is in opposite senses will tend to cancel 270◦ them over a region encompassing a sufficient number of pixels. = sin θ, (17) H It should also be remembered that although the pixel locations appear to be at the centre of the pixel boundary in the projec- in the polar zones, where | sin θ | 2/3: tion of the diamond, square, and triangular pixelisations this x = φc + (φ − φc ) σ, (18)
  32. 32. HEALPix FITS SIMPLE = T /FITS FORMAT BITPIX = 16 /DUMMY PRIMARY HEADER NAXIS = 0 /NO DATA IS ASSOCIATED WITH THIS HEADER EXTEND = T /EXTENSIONS MAY (WILL!) BE PRESENT RESOLUTN= 9 /RESOLUTION INDEX PIXTYPE = HEALPIX /PIXEL ALGORIGTHM ORDERING= NESTED /ORDERING SCHEME NSIDE = 512 /RESOLUTION PARAMETER FIRSTPIX= 0 /FIRST PIXEL (0 BASED) LASTPIX = 3145727 /LAST PIXEL (0 BASED) COMMENT THIS FILE CONTAINS A WMAP INTENSITY (I) RES 9 SKYMAP FOR THE COMMENT FREQUENCY BAND INDICATED BY THE FREQ KEYWORD. COMMENT FIVE YEARS OF DATA WENT INTO THE MAP. DATE = 2008-01-02T18:32:00 /FILE CREATION DATE (YYYY-MM- DDTHH:MM:SS UT) TELESCOP= WMAP /WILKINSON MICROWAVE ANISOTROPY PROBE REFERENC= WMAP EXPLANATORY SUPPLEMENT: HTTP:// LAMBDA.GSFC.NASA.GOV/ / STOKES = I / FREQ = KA BAND /FREQUENCY BAND RELEASE = DR3 /WMAP RELEASE END
  33. 33. HEALPix FITS XTENSION= BINTABLE /BINARY TABLE EXTENSION BITPIX = 8 /8-BIT BYTES NAXIS = 2 /2-DIMENSIONAL BINARY TABLE NAXIS1 = 8 /WIDTH OF TABLE IN BYTES NAXIS2 = 3145728 /NUMBER OF ROWS IN TABLE PCOUNT = 0 /SIZE OF SPECIAL DATA AREA GCOUNT = 1 /ONE DATA GROUP (REQUIRED KEYWORD) TFIELDS = 2 /NUMBER OF FIELDS IN EACH ROW TTYPE1 = TEMPERATURE /LABEL FOR FIELD 1 TFORM1 = E /DATA FORMAT OF FIELD: 4-BYTE REAL TUNIT1 = MK /PHYSICAL UNIT OF FIELD 1 TTYPE2 = N_OBS /LABEL FOR FIELD 2 TFORM2 = E /DATA FORMAT OF FIELD: 4-BYTE REAL TUNIT2 = COUNTS /PHYSICAL UNIT OF FIELD 2 EXTNAME = ARCHIVE MAP TABLE /NAME OF THIS BINARY TABLE EXTENSION PIXTYPE = HEALPIX /PIXEL ALGORIGTHM ORDERING= NESTED /ORDERING SCHEME NSIDE = 512 /RESOLUTION PARAMETER FIRSTPIX= 0 /FIRST PIXEL (0 BASED) LASTPIX = 3145727 /LAST PIXEL (0 BASED) END
  34. 34. XTENSION= BINTABLE /BINARY TABLE EXTENSIONBITPIX = 8 /8-BIT BYTESNAXIS = 2 /2-DIMENSIONAL BINARY TABLENAXIS1 = 8 /WIDTH OF TABLE IN BYTESNAXIS2 = 3145728 /NUMBER OF ROWS IN TABLEPCOUNT = 0 /SIZE OF SPECIAL DATA AREAGCOUNT = 1 /ONE DATA GROUP (REQUIRED KEYWORD)TFIELDS = 2 /NUMBER OF FIELDS IN EACH ROWTTYPE1 = TEMPERATURE /LABEL FOR FIELD 1TFORM1 = E /DATA FORMAT OF FIELD: 4-BYTE REALTUNIT1 = MK /PHYSICAL UNIT OF FIELD 1TTYPE2 = N_OBS /LABEL FOR FIELD 2TFORM2 = E /DATA FORMAT OF FIELD: 4-BYTE REALTUNIT2 = COUNTS /PHYSICAL UNIT OF FIELD 2EXTNAME = ARCHIVE MAP TABLE /NAME OF THIS BINARY TABLEEXTENSIONPIXTYPE = HEALPIX /PIXEL ALGORIGTHMORDERING= NESTED /ORDERING SCHEMENSIDE = 512 /RESOLUTION PARAMETERFIRSTPIX= 0 /FIRST PIXEL (0 BASED)LASTPIX = 3145727 /LAST PIXEL (0 BASED)END
  35. 35. HTM
  36. 36. HTMBy Szalay, Gray, et al.Hierarchical, multiresolution search system withsupport for region operationsBasis for spatial support in MS SQL Server, SDSS
  37. 37. Indexing the Sphere with the Hierarchical Triangular MeshHTM Alexander S. Szalay1, Jim Gray2, George Fekete1 Peter Z. Kunszt3, Peter Kukol2, and Ani Thakar1 1. Dept of Physics and Astronomy, The Johns Hopkins University, Baltimore 2. Microsoft Research Bay Area Research Center, San Francisco 3. CERN, Geneva Abstract: We describe a method to subdivide the surface of a sphere into spherical triangles of similar, but not identical, shapes and sizes. The Hierarchical Triangular Mesh (HTM) is a quad-tree that is particularly good at supporting searches at different resolutions, from arc seconds to hemispheres. The subdivision scheme is universal, providing the basis for addressing and for fast lookups. The HTM provides the basis for an efficient geospatial indexing scheme in relational databases where the data have an inherent location on either the celestial sphere or the Earth. The HTM index is superior to cartographical methods using coordinates with singularities at the poles. We also describe a way to specify surface regions that efficiently represent spherical query areas. This article presents the algorithms used to identify the HTM triangles covering such regions. 1. IntroductionBy Szalay, Gray, et al. Many science and commercial applications must catalog and search objects in three-dimensional space. In Earth and Space Science, the coordinate system is usually spherical, so the object’s position is given by its location on the surface of the unit sphere and its distance from the origin. Queries on catalogs of such objects often involve constraints on these coordinates, often in terms of complex regions that imply complicated spherical trigonometry.Hierarchical, multiresolution search system with There is a great interest in a universal, computer-friendly index on the sphere, especially in astronomy, where the ancient index of stellar constellations is still in common use, and in earth sciences, where people use maps having complicated spherical projections. The spatial index presented here transformssupport for region operations regions of the sphere to unique identifiers (IDs). These IDs can be used both as an identifier for an area and as an indexing strategy. The transformation uses only elementary spherical geometry to identify a certain area. It provides universality, which is essential for cross-referencing across different datasets. The comparisons are especially well-suited for computers because they replace transcendental functions with a few multiplications and additions.Basis for spatial support in MS SQL Server, SDSS The technique to subdivide the sphere into spherical triangles presented here is recursive. At each level of recursion, the triangle areas are roughly the same (within a factor of 2), which is a major advantage over the usual spherical coordinate system projections with singularities at the poles. Also, in areas with high data density, the recursion process can be applied to a higher level than in areas where data points are rare. This enables uneven data distributions to be organized into equal-sized bins. A similar scheme for subdividing the sphere was advocated by Barrett [1]. The idea of the current implementation of the HTM was described in Kunszt et al [2]. Short, Fekete et al [3,4,5] used an icosahedron-based mesh for Earth sciences applications. Quad trees and geometrical indexing are discussed in the books of Samet in detail [6,7]. Lee and Samet [8] use an identical triangulation, but a different numbering scheme. Goodchild [9,10] and Song et al [11] created a similar triangulation of the sphere, called the Discrete Global Grid, with precisely equal areas, using small circles for the hierarchical boundaries. Gray et al [12] described linking the HTM to a relational database system. 1
  38. 38. ( 0, 1, 0 ) := v2 (2.1) ( −1, 0, 0 ) := v3 ( 0, −1, 0 ) := v4 HTM ( 0, 0, −1 ) := v5The first eight nodes of the HTM index are defined as these eight spherical triangles named by using Ssouth and N for north, and then numbering the faces clockwise: Figure 1 The Hierarchal Triangular Mesh (HTM) is a recursive decomposition of the sphere. The figures above show the sequence of subdivisions, starting from the octahedron, down to level 5 corresponding to 8192 spherical triangles. The triangles have been plotted as planar ones for simplicity.
  39. 39. HTM triangles is 15 percent, and the ratio of the average arc length to the canonical size /2n is about 10 10 8 8 frequency 6 6 4 4 2 2 0 0 0.6 0.8 1 1.2 1.4 1.6 0.6 0.8 1 1.2 1.4 1.6 relative area relative arc length Figure 3 The distribution of relative spherical trixel areas and arc lengths around the mean at HTM depth 7. The scatter arises mainly from the difference in size of the center triangle from the corner ones at depth 2. Subsequent subdivisions smear out the ratios some, but the curvature plays a diminishing role at higher depths.efining the GeometryTM index must support intersections with arbitrary spherical regions. Given a region, we want toHTM trixels of a given depth that cover it. We call this set of trixels the region’s HTM cover. This presents the geometry primitives that define spherical areas. These primitives can easily be
  40. 40. HTM vs. HEALPix Planck Splitting the sky - HTM HEALPix. Motivation HTM - searches in spherical space HEALPix - spherical images • Index the sphere • Numerical analysis - • support spherical trigono- • convolutions local/global metrics. kernel • Support data binning • Fourier analysis with • Hierarchical for storage spherical harmonics • power spectrum estimation • Topological analysis • extreme searches • neighbour searches • Minkowski functionals • To aid above should have • equal area pixels • iso latitude distribution • Hierarchical for storage William O’Mullane 20/7/00 2/20
  41. 41. HTM vs. HEALPix Planck Splitting the sky - HTM HEALPix. Special Considerations Anomalies are inevitable with any tessellation. HEALPix HTM • Pixel shape varies - not a problem with • Pixel shape varies high enough sampling • Pixel area varies - makes many numeri- • need to take account of posi- cal calculations difficult. tion on the sphere (3 rules) • no rings • number of pixels in rings vary • similar neighbour orientation problem • Most pixels have 8 neighbours - 8 have 7 neighbours • Nested X,Y orientation of base pixel neighbours varies These are mainly problems for developers not particularly for users Advanced libraries exist for both schemes although target audiences are different. William O’Mullane 20/7/00 12/20
  42. 42. Q3C
  43. 43. Q3CBy Kortosov Barkunov (Sternberg AstronomicalInstitute)Implementation of a faster, open-source, HTM-likepartitioning Not equal areaAlternative to pgSphere, not well documented yet
  44. 44. Q3C SPHERE PARTITIONING
  45. 45. Q3C CONESEARCH PIXELISATION
  46. 46. Using partitioningschemesDisplay applicationsQuery applications Spatial joinsX-Match applications
  47. 47. Partitioning Cone Search
  48. 48. Partitioning Cone SearchSELECT * FROM TARGET_TABLE WHERE 2 * ASIN( SQRT(SIN(($DEC_C-DEC)/2) * SIN(($DEC_C-DEC)/2) + COS($DEC_C) * COS(DEC) * SIN(($RA_C - RA)/2) * SIN(($RA_C - RA)/2))) = $SR_CSELECT * FROM RASS_PHOTONS WHERE X*$X_C+Y*$Y_C+Z*$Z_C = COS($SR_C)
  49. 49. Partitioning Cone SearchSELECT * FROM RASS_PHOTONS WHERE PIXEL_ID IN RESULTING_PIXEL_QUERY AND X*$X_C+Y*$Y_C+Z*$Z_C = $COS_SR
  50. 50. Using Spatial Indexing inDatabasesJuan de Dios Santander Vela (IAA-CSIC)
  51. 51. Things to show SE BA SYHTM and Transact SQL (MS SQL)PostgreSQL and pgSphereHands on on PostgreSQL
  52. 52. HTM SQL API
  53. 53. Main Page Classes Class List Class MembersHTM SQL API Spherical Htm.Sql Spherical.Htm Sql Spherical.Htm.Sql Class ReferenceEncapsulates the HTM procedures for use from SQL2005. More...List of all members.Static Public Member Functions static SqlString fHtmVersion () fHtmVersion() returns the version number of this htm library as a string static long fHtmXyz (double x, double y, double z) fHtmXyz(x,y,z) returns the 20-deep HtmID of the given cartesian point. There are no error cases. All vectors are nomalized and 0,0,0 maps to 1,0,0 static long fHtmEq (double ra, double dec) fHtmEq(ra,dec) returns the 20-deep HtmID of the given equatorial point. There are no error cases. all RA, folded to [0..360] and dec to [-90...90] static long fHtmLatLon (double lat, double lon) fHtm(lat,lon) returns the 20-deep HtmID of the given location. There are no error cases. all RA, folded to [0..360] and dec to [0...90] static SqlDouble fDistanceEq (double ra1, double dec1, double ra2, double dec2) double fDistanceEq(ra1,dec1,ra2,dec2) returns distance in ArcMinutes between two points. static SqlDouble fDistanceLatLon (double lat1, double lon1, double lat2, double lon2) double fDistanceLatLon(lat1,lon1,lat2,lon2) returns distance in ArcMinutes between two points. static SqlDouble fDistanceXyz (double x1, double y1, double z1, double x2, double y2, double z2) double fDistanceXyz(x1,y1,z1,x2,y2,z2) returns distance in ArcMinutes between two points. static SqlString fHtmToString (SqlInt64 HtmID) fHtmToString(HtmID) returns varchar(max)a string describing the HtmID static IEnumerable fHtmLatLonToXyz (SqlDouble lat, SqlDouble lon) XyzTable(x,y,z) fHtmLatLonToXyz(lat, lon) converts an lat,lon point to a table with one row containing the cartesian point (x,y,z). static IEnumerable fHtmEqToXyz (SqlDouble ra, SqlDouble dec) XyzTable(x,y,z) fHtmEqToXyz(ra, dec) converts an equitorial point to a table with one row containing the cartesian point (x,y,z). static IEnumerable fHtmXyzToLatLon (SqlDouble x, SqlDouble y, SqlDouble z) LatLonTable(lat,lon) fHtmXyzToLatLon(x,y,z) converts the cartesian point (x,y,z) to a table with one row containing the equivalent lat,lon point. static IEnumerable fHtmXyzToEq (SqlDouble x, SqlDouble y, SqlDouble z) EqTable(ra, dec) fHtmXyzToEq(x,y,z) converts the cartesian point (x,y,z) to a table with one row containing the equivalent equitorial (ra,dec) point. static IEnumerable fHtmToCenterPoint (SqlInt64 HtmID) fHtmToCenterPoint(HtmID) converts an HTM triangle ID to an (x,y,z) vector of the HTM triangle centerpoint. and returns that vector as the only row of a table. static IEnumerable fHtmToCornerPoints (SqlInt64 HtmID) fHtmToCornerPoints(HtmID) converts an HTM triangle ID to an table of three (x,y,z) vectors of the HTM triangle corners. static IEnumerable fHtmCoverCircleLatLon (SqlDouble lat, SqlDouble lon, SqlDouble radiusArcMin) fHtmCoverCircleLatLon(ra,dec,radiusArcMin) returns a trixel table (a list) covering
  54. 54. fHtmRegionToNormalFormString(coverspec) converts a region definiton to normal formParameters: coverspec a string satisfying the region syntax regionSpec := REGION {areaSpec}* | areaSpecHTM SQL API circleSpec := CIRCLE J2000 ra dec radArcMin | CIRCLE LATLON lat lon radArcMin | CIRCLE [CARTESIAN ] x y z radArcMin rectSpec := RECT LATLON {lat lon}2 | RECT J2000 {ra dec }2 | RECT [CARTESIAN ] {x y z }2 polySpec := POLY LATLON {lat lon}3+ | POLY J2000 {ra dec }3+ | POLY [CARTESIAN ] {x y z }3+ hullSpec := CHULL LATLON {lat lon}3+ | CHULL J2000 {ra dec }3+ | CHULL [CARTESIAN ] {x y z }3+ convexSpec := CONVEX LATLON {lat lon}* | CONVEX J2000 {ra dec }* | CONVEX [CARTESIAN ]{x y z }* areaSpec := circleSpec | rectSpec | polySpec | hullSpec | convexSpecReturns: region String SqlString region in normal form ( a union of convexes) This looks something like REGION CONVEX 1 0 0 0 0 1 0 0 CONVEX 0 0 1 0 create function fHtmRegionToNormalFormString(@region nvarchar(max)) returns table (HtmID_start bigint, HtmID_end bigint) as external name HTM.Sql.fHtmToNormalForm select dbo.fHtmRegionToNormalFormString(CIRCLE J2000 195 0 1)--------------------------------------------------------------static SqlString Spherical.Htm.Sql.fHtmRegionError ( SqlString coverspec ) [static]fHtmRegionError(coverspec) returns diagnostic message appropriate for a bad region stringParameters: coverspec a string satisfying the region syntaxReturns: diagnostic message: varchar(max) OK if ok else message and a syntax string. create function fHtmRegionError(@region nvarchar(max)) returns varchar(max) as external name HTM.Sql.fHtmRegionError print dbo.fHtmRegionError(CIRCLE LATLON 195 )See also: fHtmCover()the main routine
  55. 55. HTM SQL API Main Page Classes Class List Class Members Spherical.Htm.Sql Member List This is the complete list of members for Spherical.Htm.Sql including all inherited members. Spherical.Htm.Sql, fDistanceEq fDistanceEq(double ra1, double dec1, double ra2, double dec2) Spherical.Htm.Sql [static] fDistanceLatLon fDistanceLatLon(double lat1, double lon1, double lat2, double lon2) Spherical.Htm.Sql [static] fDistanceXyz fDistanceXyz(double x1, double y1, double z1, double x2, double y2, double Spherical.Htm.Sql [static] z2) fHtmCoverCircleEq fHtmCoverCircleEq(SqlDouble ra, SqlDouble dec, SqlDouble radiusArcMin) Spherical.Htm.Sql [static] fHtmCoverCircleLatLon fHtmCoverCircleLatLon(SqlDouble lat, SqlDouble lon, SqlDouble Spherical.Htm.Sql [static] radiusArcMin) fHtmCoverCircleXyz fHtmCoverCircleXyz(SqlDouble x, SqlDouble y, SqlDouble z, SqlDouble Spherical.Htm.Sql [static] radiusArcMin) fHtmCoverList fHtmCoverList(SqlString coverspec) Spherical.Htm.Sql [static] fHtmCoverRegion fHtmCoverRegion(SqlString coverspec) Spherical.Htm.Sql [static] fHtmCoverRegionSelect fHtmCoverRegionSelect(SqlString coverspec, SqlString kind) Spherical.Htm.Sql [static] fHtmCoverTypedRegion fHtmCoverTypedRegion(SqlString coverspec) Spherical.Htm.Sql [static] fHtmEq fHtmEq(double ra, double dec) Spherical.Htm.Sql [static] fHtmEqToXyz fHtmEqToXyz(SqlDouble ra, SqlDouble dec) Spherical.Htm.Sql [static] fHtmLatLon fHtmLatLon(double lat, double lon) Spherical.Htm.Sql [static] fHtmLatLonToXyz fHtmLatLonToXyz(SqlDouble lat, SqlDouble lon) Spherical.Htm.Sql [static] fHtmRegionError fHtmRegionError(SqlString coverspec) Spherical.Htm.Sql [static] fHtmRegionToNormalFormString fHtmRegionToNormalFormString(SqlString coverspec) Spherical.Htm.Sql [static] fHtmRegionToTable fHtmRegionToTable(SqlString coverspec) Spherical.Htm.Sql [static] fHtmToCenterPoint fHtmToCenterPoint(SqlInt64 HtmID) Spherical.Htm.Sql [static] fHtmToCornerPoints fHtmToCornerPoints(SqlInt64 HtmID) Spherical.Htm.Sql [static] fHtmToString fHtmToString(SqlInt64 HtmID) Spherical.Htm.Sql [static] fHtmVersion fHtmVersion() Spherical.Htm.Sql [static] fHtmXyz fHtmXyz(double x, double y, double z) Spherical.Htm.Sql [static] fHtmXyzToEq fHtmXyzToEq(SqlDouble x, SqlDouble y, SqlDouble z) Spherical.Htm.Sql [static] fHtmXyzToLatLon fHtmXyzToLatLon(SqlDouble x, SqlDouble y, SqlDouble z) Spherical.Htm.Sql [static] NOW ALL IN C# FOR MS SQL SERVER, BUT…
  56. 56. HTM SQL API
  57. 57. HTM SQL API
  58. 58. HTM Regions HTMSqlRegions.html 20/01/10 8:56 coverSpec = polySpec | rectSpec | circleSpec | regionSpec | hullSpec circleSpec = CIRCLE J2000 [Ln] ra dec radArcMin | CIRCLE CARTESIAN [ln] x y z radArcMin coverSpec = polySpec | rectSpec | circleSpec | regionSpec | hullSpec circleSpec = CIRCLE J2000 [Li] ra dec rad | CIRCLE CARTESIAN [Li] x y z rad rectSpec = RECT J2000 {ra dec}4 | RECT CARTESIAN {x y z }4 polySpec = POLY [J2000] {ra dec} | POLY CARTESIAN {x y z}3+ hullSpec = CHULL J2000 {ra dec}3+ (Cartesian not implemented currently) convexSpec = CONVEX { x y z D}+ regionSpec = REGION {convexSpec}+
  59. 59. pgSphere objects
  60. 60. pgSphere objects SPOINT FORMAT STRINGS spoint (0.1,-0.2) spoint ( 10.1d, -90d) spoint ( 10d 12m 11.3s, -13d 14m) spoint ( 23h 44m 10s, -1.4321 ) CONSTRUCTOR FUNCTIONS SPOINT(0.1,-0.2) SPOINT(RADIANS(10.1D),RADIANS(-90)) SPOINT(RADIANS(RA_FIELD),RADIANS(DECL_FIELD))
  61. 61. pgSphere objects SCIRCLE FORMAT STRINGS SCIRCLE (0D, 90D), 5D SCIRCLE (0D, 90D), 0.5 CONSTRUCTOR FUNCTIONS SCIRCLE(SPOINT(0.1,-0.2),0.2) SCIRCLE(SPOINT(RADIANS(50.0),RADIANS(-59.0)),RADIANS(1)) SCIRCLE(SPOINT(RADIANS(TARGET_RA), RADIANS(TARGET_DECL), RADIANS(SR))
  62. 62. pgSphere objects SELLIPSE FORMAT STRINGS SELLIPSE { 10D, 5D } , ( 20D, 0D ), 90D CONSTRUCTOR FUNCTIONS SELLIPSE(SPOINT(0.1,-0.2), 0.3,0.1, 0.2) SELLIPSE(SPOINT(RADIANS(RA),RADIANS(DECL)), RADIANS(ERR_MAJ),RADIANS(ERR_MIN),RADIANS(ERR_ANG))
  63. 63. pgSphere objects USE CONSTRUCTORS!
  64. 64. pgSphere operatorsbinary boolean angular distance (- ) equality (=) unary scalar inclusion and overlap (see next length (of line or slide) circumference; @-@)binary scalar center (@@)
  65. 65. t (1 row)Inclusion and overlap 5.3. Contain and overlap On sphere, an equality relationship is rarely used. There are frequently object a contained by object b? or Does object a overlap object b? pgS queries using binary operators returning true or false: Table 3. Contain and overlap operators operator operator returns true, if @ the left object is contained by right object ˜ the left object contains right object !@ the left object is not contained by right object !˜ the left object does not contain right object the objects overlap each other ! the objects do not overlap each other An overlap or contain operator does not exist for all combinations of d instance, scircle @ spoint is useless because a spherical point can nev spherical circle. Example 25. Is the left circle contained by the right circle?
  66. 66. Spatial Indexing Primer create procedure safcat.fConeSearch(in ra double, in decl double, in sr double) begin ! declare decl_start double; ! declare decl_end double; ! declare ra_start double; ! declare ra_end double; ! declare ra_sr double; ! declare nx double; ! declare ny double; ! declare nz double; ! set nx = cos(radians(ra)) * cos(radians(decl)); ! set ny = sin(radians(ra)) * cos(radians(decl)); ! set nz = sin(radians(decl)); ! set decl_start = decl - abs(sr); ! if decl_start -90 thenExample ! ! ! ! ! set decl_start = -90.0; end if ; set decl_end = decl + abs(sr); if decl_end 90 then ! ! set decl_end = 90.0; ! end if ; ! if abs(cos(radians(decl))) 0.00001 then Cone Search with ! ! set ra_start = 0; ! ! set ra_end = 360; ! else ! ! set ra_sr = degrees(2 * asin( sin(radians(sr)/2) / cos(radians(decl)))); RA and Decl ! ! set ra_start = ra - ra_sr; ! ! set ra_end = ra + ra_sr; ! end if ; ! if ra_start 0 then restrictions ! ! select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test ! ! where decl between decl_start and decl_end ! ! and (ra between ra_start+360 and 360 or ! ! ra between 0 and ra_end) ! ! and ((nx*cx+ny*cy+nz*cz) cos (radians(sr))) ! else ! ! if ra_end 360 then ! ! ! select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test ! ! ! where decl between decl_start and decl_end ! ! ! and (ra between 0 and ra_end - 360 or ! ! ! ra between ra_start and 360) ! ! ! and ((nx*cx+ny*cy+nz*cz) cos (radians(sr))) ! ! else ! ! ! select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test ! ! ! where decl between decl_start and decl_end ! ! ! and ra between ra_start and ra_end ! ! ! and ((nx*cx+ny*cy+nz*cz) cos (radians(sr))) ! ! end if; ! end if; end ;
  67. 67. Spatial Indexing Primer create procedure safcat.fConeSearch(in ra double, in decl double, in sr double) begin ! declare decl_start double; ! declare decl_end double; ! declare ra_start double; ! declare ra_end double; ! declare ra_sr double; ! declare nx double; ! declare ny double; ! declare nz double; ! set nx = cos(radians(ra)) * cos(radians(decl)); ! set ny = sin(radians(ra)) * cos(radians(decl)); ! set nz = sin(radians(decl)); ! set decl_start = decl - abs(sr); ! if decl_start -90 thenExample ! ! ! ! ! set decl_start = -90.0; end if ; set decl_end = decl + abs(sr); if decl_end 90 then ! ! set decl_end = 90.0; ! end if ; ! if abs(cos(radians(decl))) 0.00001 then Cone Search with ! ! set ra_start = 0; ! ! set ra_end = 360; ! else ! ! set ra_sr = degrees(2 * asin( sin(radians(sr)/2) / cos(radians(decl)))); RA and Decl ! ! set ra_start = ra - ra_sr; ! ! set ra_end = ra + ra_sr; ! end if ; ! if ra_start 0 then restrictions ! ! select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test ! ! where decl between decl_start and decl_end ! ! and (ra between ra_start+360 and 360 or ! ! ra between 0 and ra_end) ! ! and ((nx*cx+ny*cy+nz*cz) cos (radians(sr))) ! else ! ! if ra_end 360 then ! ! ! select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test ! ! ! where decl between decl_start and decl_end ! ! ! and (ra between 0 and ra_end - 360 or ! ! ! ra between ra_start and 360) ! ! ! and ((nx*cx+ny*cy+nz*cz) cos (radians(sr))) ! ! else ! ! ! select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test ! ! ! where decl between decl_start and decl_end ! ! ! and ra between ra_start and ra_end ! ! ! and ((nx*cx+ny*cy+nz*cz) cos (radians(sr))) ! ! end if; ! end if; end ;
  68. 68. Spatial Indexing PrimerExample select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test where decl between decl_start and decl_end Cone Search with and ra between ra_start and ra_end and ((nx*cx+ny*cy+nz*cz) cos (radians(sr))) RA and Decl restrictions
  69. 69. Spatial Indexing Primer HTMindexImp index = (HTMindexImp) new HTMindexImp(20, 5); Vector3d vect = new Vector3d(ra, decl); double dist = Math.cos(radians(sr)); Convex convex = new Convex(); Constraint constraint = new Constraint(vect, dist); convex.add(constraint); convex.simplify(); Domain domain = new Domain(); domain.add(convex); domain.setOlevel(20); HTMrange htmRange = new HTMrange();Example domain.intersect(index, htmRange, false); StringBuilder queryHtmTable = new StringBuilder( select row_id, ra, decl, cx, cy, cz, b, v, htm20n”+ “from htm20testnwheren); htmRange.reset(); Cone Search with boolean isFirst = true; long[] range = htmRange.getNext(); while ((range != null) (range[0] 0)) { HTM String betweenClause; if (isFirst) { betweenClause = String.format( (htm20 between %014d and %014d)n, range[0], range[1]); isFirst = false; } else { betweenClause = String.format( or (htm20 between %014d and %014d)n, range[0], range[1]); } queryHtmTable.append(betweenClause); range = htmRange.getNext(); } String finalQuery; if (htmRange.nranges() 0) { finalQuery = queryHtmTable.toString(); } else { finalQuery = select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test where htm20 0; }
  70. 70. Spatial Indexing Primer select row_id, ra, decl, cx, cy, cz, b, v, htm20 from htm20test where (htm20 between 13469017440256 and 13469084549119) or (htm20 between 13469239738368 and 13469239803903) or (htm20 between 13469239869440 and 13469239934975) or (htm20 between 13469239951360 and 13469239952383) or (htm20 between 13469239952896 and 13469239952896) or (htm20 between 13469239954944 and 13469239954944) or (htm20 between 13469239954946 and 13469239954946) or (htm20 between 13469239954947 and 13469239954947) or (htm20 between 13469260709888 and 13469260775423) or (htm20 between 13469260775424 and 13469260840959) or (htm20 between 13469260939264 and 13469260940287)Example or (htm20 between 13469260941568 and 13469260941568) or (htm20 between 13469260942592 and 13469260942592) or (htm20 between 13469260942593 and 13469260942593) or (htm20 between 13469260942595 and 13469260942595) or (htm20 between 13469269098496 and 13469269102591) Cone Search with or (htm20 between 14568529068032 and 14568596176895) or (htm20 between 14568751366144 and 14568751431679) or (htm20 between 14568751497216 and 14568751562751) or (htm20 between 14568751579136 and 14568751580159) HTM or (htm20 between 14568751580672 and 14568751580672) or (htm20 between 14568751582720 and 14568751582720) or (htm20 between 14568751582722 and 14568751582722) or (htm20 between 14568751582723 and 14568751582723) or (htm20 between 14568772337664 and 14568772403199) or (htm20 between 14568772403200 and 14568772468735) or (htm20 between 14568772567040 and 14568772568063) or (htm20 between 14568772569344 and 14568772569344) or (htm20 between 14568772570368 and 14568772570368) or (htm20 between 14568772570369 and 14568772570369) or (htm20 between 14568772570371 and 14568772570371) or (htm20 between 14568780726272 and 14568780730367) or (htm20 between 15668040695808 and 15668107804671) or (htm20 between 15668262993920 and 15668263059455) or (htm20 between 15668263124992 and 15668263190527) or (htm20 between 15668263206912 and 15668263207935) or (htm20 between 15668263208448 and 15668263208448) or (htm20 between 15668263210496 and 15668263210496) or (htm20 between 15668263210498 and 15668263210498) or (htm20 between 15668263210499 and 15668263210499) or (htm20 between 15668283965440 and 15668284030975)
  71. 71. Is this Really Faster?
  72. 72. Sybase IQ TM Query PlanQuery:Version: 15.2.0.5604/100518/P/GA/Enterprise Linux64 - x86_64 - 2.6.9-67.0.4.ELsmp/64bitQuery Tree | 392 rows #03 Root SQL CONESEARCH | 392 rows #04 Scrolling Cursor Store | 392 rows #08 Parallel Combiner QUERY PLAN || || 5596 rows #02 Filter ||| ||| 2056400 rows #01 Leaf safcat.htm20testQuery Timings Timing Legend Condition Execution Prepare 1st Fetch Subsequent Fetches Complete Elapsed Time (sec) 1.362 2.724 4.086 5.448 6.81 8.172 9.534 10.896 12.258 13.62#03 Root ...... .. ...#04 Scrolling Cursor Store . ...... .. ...#08 Parallel Combiner .. . . . . . . .. ...#02 Filter ... . . . . . . . . . ..#01 Leaf .... . . . . . . . . . .. 3 . . . . 08 08 08 08 . . Threads 2 . 1 .. . . . . . . .. 100 . . 95 . 90 . 85 . 80 . 75 . . 70 . 65 . 60 . 55 . . CPU % . . . . .. 50 . . 45 . 40 . 35 . 30 . 25 . . 20 . 15 . 10 . 5 . . Wall Time 00:01:02.363 00:01:05.088 00:01:07.813 00:01:10.538 00:01:13.263Query Text select htm20test.row_id,htm20test.ra,htm20test.decl,htm20test.cx,htm20test.cy,htm20test.cz,htm20test.b,htm20test.v,htm20test.htm20 from htm20test where nx*htm20test.cx+ny*htm20test.cy+nz*htm20test.cz cos(radians(sr)) and htm20test.ra between ra_start and ra_end and htm20test.decl between decl_start and decl_endQuery Detail #03 Root Child Node 1 #04 Generated Result Rows 392 Estimated Result Rows 640000 User Name safcat (SA connHandle: 100 SA connID: 15)
  73. 73. Sybase IQ TM Query PlanQuery:Version: 15.2.0.5604/100518/P/GA/Enterprise Linux64 - x86_64 - 2.6.9-67.0.4.ELsmp/64bitQuery Tree HTM | 1 row #03 Root | 1 row QUERY PLAN #04 Scrolling Cursor Store | 1 row #01 Aggregation Leaf safcat.htm20testQuery Timings Timing Legend Condition Execution Prepare 1st Fetch Subsequent Fetches Complete Elapsed Time (sec) 6.429 12.858 19.287 25.716 32.145 38.574 45.003 51.432 57.861 64.29#03 Root ......... ..#04 Scrolling Cursor Store . . . . . . . . . . ..#01 Aggregation Leaf .. . . . . . . . . . . . 3 . . 01 . . . . . . . . .. Threads 2 . . 1 ... . . . . . . . . .. 100 . 95 . 90 . 85 . 80 . . 75 . . 70 . 65 . 60 . 55 . CPU % . ....... . .. 50 . 45 . 40 . 35 . . 30 . . 25 . 20 . 15 . 10 . . 5 . . Wall Time 23:59:58.046 00:00:10.905 00:00:23.764 00:00:36.623 00:00:49.482Query Text select count(row_id)
  74. 74. References LinksHEALPix: A Framework for Indexing the Sphere with theHigh-Resolution Hierarchical Triangular MeshDiscretization and FastAnalysis […] Splitting the sky - HTM  HEALPixHEALPix Primer Q3C, Quad Tree Cube - TheSpatial indexing in a new sky-indexing conceptrelational database usingHEALPix pgSphere documentationMapping on the HEALPix grid

×