GoogleSky Status at Google


Published on

Team presentation at Google on April 11, 2006

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

GoogleSky Status at Google

  1. 1. Google Sky Status and Goals
  2. 2. Data Resources Photo Photo Photo CCD Source FITS FITS FITS JP2 Status 16 bit 8 bit 16 bit 32 bit Range 0.7” 0.5” 1” 0.45” Plate scale 21.0 mag 21.0 mag 21.0 mag 23.5 mag Depth JF (N) JFN (S) JFN (N) JF (S) JFN (N) JFN (S) ugriz Bands 40000 40000 40000 8000 Coverage COSMOS PMM DSS SDSS
  3. 3. Data Resources <ul><li>SDSS </li></ul><ul><ul><li>8000 sq degrees, CCD, 23 mag, ugriz, 0.5” </li></ul></ul><ul><ul><li>Pittsburgh (jpegs), FNAL (fits) </li></ul></ul><ul><ul><li>Scanning across the sky (great circles) </li></ul></ul><ul><ul><li>Ver1: original 0.25x0.25 degree patches </li></ul></ul><ul><ul><li>Ver2: individual fields </li></ul></ul><ul><ul><li>Ver3: original fits images (ugr, gri, riz) </li></ul></ul><ul><ul><li>Licensing started </li></ul></ul><ul><li>DPOSS/UKSchmidt </li></ul><ul><ul><li>All-sky, Photo, 21 mag, JFN, 1” </li></ul></ul><ul><li>PMM </li></ul><ul><ul><li>All-Sky, 21 mag, JF, 1” </li></ul></ul>
  5. 5. Fusion <ul><li>Enterprise Box </li></ul><ul><ul><li>Delivered (fusion and server) </li></ul></ul><ul><ul><ul><li>200 sq degrees ingested and served </li></ul></ul></ul><ul><ul><ul><li>SDSS reprocessed as compressed JPEG2000 (single resolution) </li></ul></ul></ul><ul><ul><ul><li>SDSS 280,000 regions, 8000 sq degrees 1300x1900 pixels (11m) </li></ul></ul></ul><ul><ul><ul><li>DSS 180,000 regions, 2500x2500 pixels (16m) </li></ul></ul></ul><ul><ul><li>Tested all sky </li></ul></ul><ul><ul><ul><li>WMAP </li></ul></ul></ul>
  6. 6. Current Status <ul><li>Issues </li></ul><ul><ul><li>Timing for fusion </li></ul></ul><ul><ul><ul><li>350 frames (0.1% of SDSS) </li></ul></ul></ul><ul><ul><ul><ul><li>274986 tiles touched </li></ul></ul></ul></ul><ul><ul><ul><ul><li>2 days 17 hrs > 2000 days? </li></ul></ul></ul></ul><ul><ul><ul><ul><li>2-3 hours to build pyramid of 14 levels </li></ul></ul></ul></ul><ul><ul><li>Merging assets </li></ul></ul><ul><ul><ul><li>Limit on number and resolution of assets </li></ul></ul></ul><ul><ul><ul><ul><li>All assets same resolution, must be broken into individual assets </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Merged at project level </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Auto mask -> feathering between assest (dark bands) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>No mask -> does not build all resolution levels (appear at differnet zooms) </li></ul></ul></ul></ul><ul><ul><li>Artifacts </li></ul></ul><ul><ul><ul><li>Detail at different zooms </li></ul></ul></ul><ul><ul><ul><li>Diffraction spikes etc – what will public accept </li></ul></ul></ul><ul><ul><ul><li>Image quality at poles </li></ul></ul></ul>
  7. 7. Generic Reduction Pipeline <ul><li>Simplify Astronomer’s Ingestion </li></ul><ul><ul><li>Assumes World Coordinate System </li></ul></ul><ul><ul><ul><li>wcsOverlap </li></ul></ul></ul><ul><ul><ul><ul><li>Calculates overlap between images </li></ul></ul></ul></ul><ul><ul><ul><li>wcsTanProj </li></ul></ul></ul><ul><ul><ul><ul><li>Project onto tangent plane (base image and overlaps) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Background subtract </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Lanczos interpolation (windowed sinc function) </li></ul></ul></ul></ul><ul><ul><ul><li>imageColor </li></ul></ul></ul><ul><ul><ul><ul><li>Applies color mapping for input images (Lupton algorithm) </li></ul></ul></ul></ul>I ≡ (r + g + b)/3 R = r ∗ f(I)/I G = g ∗ f(I)/I B = b ∗ f(I)/I
  8. 8. Rotation and background
  9. 9. Color Definition SDSS DSS JFN DSS JF extrapolate
  10. 11. Pipeline Requirements <ul><li>Base Map Processing Estimates (no optimization) </li></ul><ul><ul><li>Fits reduction pipeline </li></ul></ul><ul><ul><ul><li>2-3 hrs per plate per color </li></ul></ul></ul><ul><ul><ul><li>10-20,000 hrs </li></ul></ul></ul><ul><ul><li>Fusion </li></ul></ul><ul><ul><ul><li>24-50,000 hours </li></ul></ul></ul><ul><li>Space Estimates </li></ul><ul><ul><li>DSS </li></ul></ul><ul><ul><ul><li>7 TB (FITS) </li></ul></ul></ul><ul><ul><ul><li>1.5 TB (1 arcsec, JP2) </li></ul></ul></ul><ul><ul><li>SDSS </li></ul></ul><ul><ul><ul><li>10 TB (FITS) </li></ul></ul></ul><ul><ul><ul><li>1.2 TB (0.45 arcsec, JP2) </li></ul></ul></ul><ul><li>Optimization </li></ul><ul><ul><li>Generate initial images on the fusion grid? </li></ul></ul><ul><ul><li>Optimal input image size? </li></ul></ul><ul><ul><li>Low priority queues </li></ul></ul>
  11. 12. Actions <ul><li>Defining Coordinate System </li></ul><ul><ul><li>ICRS (J2000) </li></ul></ul><ul><ul><ul><li>GIS: -180 to 180 deg </li></ul></ul></ul><ul><ul><ul><li>Astro 0 to 360 deg </li></ul></ul></ul><ul><ul><ul><li>New coordinate grid on sky, new client? </li></ul></ul></ul><ul><ul><li>Projection </li></ul></ul><ul><ul><ul><li>North up East left? </li></ul></ul></ul><ul><li>Color Balancing </li></ul><ul><ul><li>Image scaling for bright stars </li></ul></ul><ul><ul><ul><li>Images appear noisy at low zoom </li></ul></ul></ul><ul><ul><ul><li>Recreate images at each zoom level </li></ul></ul></ul><ul><ul><li>Balance between images/surveys </li></ul></ul><ul><ul><ul><li>Need to optimize overlaps of DSS and SDSS </li></ul></ul></ul><ul><ul><ul><li>Colors for astrophysics or public? </li></ul></ul></ul>
  12. 13. KML and Catalogs <ul><li>Prototype of press release and catalog query </li></ul><ul><ul><li>HST, SIAP, DBMS </li></ul></ul><ul><li>Scaling </li></ul><ul><ul><li>Querying images/points </li></ul></ul><ul><ul><ul><li>How many points, images in cache </li></ul></ul></ul><ul><ul><ul><li>How many kml objects can be served </li></ul></ul></ul><ul><ul><ul><li>Need for addition astrophysics tags </li></ul></ul></ul><ul><ul><ul><li>What altitudes to we view a set of points </li></ul></ul></ul><ul><ul><li>Query based on field of view </li></ul></ul>
  13. 14. Issues <ul><li>Data Sources </li></ul><ul><ul><li>What catalogs/overlays for public </li></ul></ul><ul><ul><ul><li>Hubble press releases, ACS </li></ul></ul></ul><ul><ul><ul><li>Bayer, NGC/UGC, RC3, Clusters, Constellations </li></ul></ul></ul><ul><li>Layers </li></ul><ul><ul><li>Layers of expertise for users </li></ul></ul>
  14. 15. Deploying Google Sky <ul><li>How to get to Beta </li></ul><ul><ul><li>New Client or New database </li></ul></ul><ul><ul><ul><li>EC can do this </li></ul></ul></ul><ul><ul><li>How many base maps </li></ul></ul><ul><ul><li>Base map transparency? </li></ul></ul><ul><ul><li>What functionality in initial beta </li></ul></ul><ul><ul><ul><li>Coordinate system and grid </li></ul></ul></ul><ul><ul><li>Excluding ads </li></ul></ul><ul><ul><li>Licensing status </li></ul></ul><ul><ul><li>What is the process to deploy to data centers </li></ul></ul>
  15. 16. New Features <ul><li>Hierarchy of images </li></ul><ul><ul><li>Small user – single image (OK) </li></ul></ul><ul><ul><li>Large user – base map (OK) </li></ul></ul><ul><ul><ul><li>More than one base map </li></ul></ul></ul><ul><ul><li>Intermediate user </li></ul></ul><ul><ul><ul><li>WMS standard interface </li></ul></ul></ul><ul><ul><ul><li>Fast but needs GS to cache images </li></ul></ul></ul><ul><ul><ul><li>Persistent and rapid update of images </li></ul></ul></ul><ul><ul><ul><li>How will things scale </li></ul></ul></ul>
  16. 17. New Features <ul><li>Time/Frequency Domain </li></ul><ul><ul><li>Real time events </li></ul></ul><ul><ul><ul><li>Press release (linked via CNN etc) </li></ul></ul></ul><ul><ul><ul><li>RSS feeds of new events </li></ul></ul></ul><ul><ul><ul><li>Sharing of events from users (comets, asteroids etc) </li></ul></ul></ul><ul><ul><li>Imaging surveys </li></ul></ul><ul><ul><ul><li>Multiple realizations </li></ul></ul></ul><ul><ul><ul><ul><li>SDSS 25 repeats over 300 sq degree </li></ul></ul></ul></ul><ul><ul><ul><ul><li>LSST/PanSTARRS 100s over 20,000 sq degrees </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Equivalent to different wavelengths (Xray, Near-IR) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Movie of the dynamic sky </li></ul></ul></ul></ul><ul><ul><li>Embedding into Google Earth? </li></ul></ul><ul><ul><ul><li>Provide sky around GE </li></ul></ul></ul>
  17. 18. VO Interfaces <ul><li>SIAP </li></ul><ul><ul><li>Importing links to kml </li></ul></ul><ul><li>Catalogs </li></ul><ul><ul><li>XML tags for astronomy </li></ul></ul><ul><ul><li>Relation with VOTables? </li></ul></ul><ul><ul><li>Scaling to 10^6 sources </li></ul></ul>
  18. 19. Short-term Critical Actions <ul><li>Low priority processing queues </li></ul><ul><ul><li>Need 40,000+ cpu hours </li></ul></ul><ul><ul><li>MapReduce (already reduced) </li></ul></ul><ul><ul><li>Which resources are available </li></ul></ul><ul><li>Ingestion/Magrathean </li></ul><ul><ul><li>Enterprise box can store and serve final products (network disk) </li></ul></ul><ul><ul><li>Enterprise box cant process images </li></ul></ul>