This document provides an overview of visualizing big imaging data in radio astronomy. It discusses: 1) Facilities like Pawsey Supercomputing Centre and ICRAR that provide computational resources for processing and visualizing large astronomy data. 2) Common astronomy image formats like FITS and emerging "big data" formats like JPEG2000 that allow for multi-resolution and streaming visualization. 3) The SkuareView framework that implements remote visualization of JPEG2000 encoded astronomy data using JPIP by streaming different resolutions and regions of interest without downloading full datasets. 4) A demo of using SkuareView to interactively visualize multi-TB radio astronomy datasets stored in the cloud.

1. Visualisation of Big
Imaging Data:
Radio Astronomy
Case
Slava Kitaeff

2. Contents
• Pawsey, ICRAR, computers and telescopes
• Astronomy image formats and visualisation
software
• The era of Big Data in astronomy
• JPEG2000 and JPIP
• SkuareView – new astronomy remote
visualisation framework and tool
• Demo
2

3. 3
3
Pawsey, ICRAR, computers and telescopes.

4. Pawsey Supercomputing Centre
4
Pawsey is the government-supported high-performance compu7ng
na7onal facility (Perth, Western Australia) that supports researchers
in Western Australia and across Australia through providing the
infrastructure for the computa7onal research workflows. This
includes
• supercomputers
• cloud compu7ng
• data storage
• visualisa7on

5. HPC@Pawsey
5
• 48 blades x 4 nodes x 2 CPUs (Intel Xeon
E5-2690V3 “Haswell” 2.6 GHz) x 12-cores =
35,712 cores
• 1.1 PetaFLOP
• Interconnect - Cray Aries
• Local storage – 3PB Cray Sonexion 1600
Lustre appliance
Magnus
#41 in TOP500 (November 2014)

6. HPC@Pawsey
6
• 118 compute blades, each of which has four
nodes
• Each node supports two, 10-core Intel Xeon
E5-2960V2 “Ivy Bridge” processors
opera7ng at 3.00 GHz
• Total of 9,440 cores
• ~200 TeraFLOPS of compute power.
• Interconnect - Cray Aries
Galaxy
Zeus & Zythos
• 39 nodes in various configura7ons
• Zythos is the largest node: SGI UV2000
system with 6TB shared memory, 264 Intel
Xeon processor cores and 4 NVIDIA K20
GPUs.

7. NeCTAR
l NeCTAR (National eResearch
Collaboration Tools and Research)
l NeCTAR is an Australian Government
project to build infrastructure specifically for
the needs of Australian researchers
l NecTAR is a $47 million dollar, Australian
Government project conducted as part the
Super Science initiative and financed by the
Education Investment Fund
l NeCTAR has built:
l New virtual laboratories
l A research cloud
l eResearch tools
l Hosting services

8. RDS/RDSI
Australian National Data Storage
In Perth
~4 Petabytes disk storage
(GPFS), plus
>35 Perabyte DMF Tape storage

9. ICRAR
• The International Centre for
Radio Astronomy Research is a
collaborative centre that is
international in scope and that
achieves research excellence in
astronomical science and
engineering.
• ICRAR is an equal joint venture
between Curtin University and
The University of Western
Australia with funding support
from the State Government of
Western Australia
9

10. ASKAP/MWA/SKA
10
2013 Harley Wood Winter School,

11. 11
11
Astronomy Image Formats
and
Visualisation Software

12. Astronomy Visualisation
Astronomy datasets are n-dimensional
• An electro-magnetic wave is described by Amp(RA,
DEC, spectral/velocity/energy, polarization, time
[phase])
• Project a n-dimensional object on a 2-dimensional
plane
• Add other dimensions through other means
• No other dimensions: projection of data, slices
• Time/movies
• Projection can also show combinations of dimensions, rotation
of cubes, volume rendering/opaqueness
• Collapse can be in different ways, e.g. moment maps, peak
flux maps, medians, etc.
• Can be combined, e.g. brightness/hue
• Contours, markers, vectors
• Polarization is used e.g. in 3d-movies
12

13. SAOImage
13

14. Alladin Sky Atlas - Lite
14

15. NRAO casaviewer
15

16. Astronomy Image Formats
• Flexible Image Transport System (FITS)
• CASA Measurement Set (and Image Tables)
• HDF5 (LOFAR)
• others
• No PNG, JPG, TIFF etc, as they are poor in
metadata handling
16

17. Spectral-imaging data-cube
17
• Right Ascension
• Declina7on
• Velocity (frequency/wavelength)
• Polarisa7on
• Sky Model
• Beam map

18. 18
Neutral Hydrogen (HI) in Universe
18

19. 19
Cosmos HI Large Extragalactic Survey
(CHILES)
19
VLA in B array and covering a redshil
range from z=0 to z=0.45
Full data-cube is 500GB

20. 20
20
The Era of Big Data in Astronomy

21. SKA1 data sizes/volumes
21
• CHILES cube 0.5 TB
• ASKAP DINGO cube 1 TB

22. HDD capacity
Moore’s law for HDD
• ~10 times every 5 years
• 10TB HDD in a today’s desktop
• 100TB HDD/SSD/(?) by 2020

23. Another problem: Network speed
• Moore’s law for network I/O
• ~10 times every 6-10 years
• 1-10Gb in desktop/server today
• 100Gb by ~2025
Capacity (Gb/s)
0.001
0.010
0.100
1.000
10.000
100.000
1990 1995 2000 2005 2010 2015 2020 2025
Download of 22TB SKA data-cube
Today
• at 1 Gb/s: ~60 hours
by 2020
• at 10 Gb/s: ~6 hours
The data is likely has to stay in the archive,
and we need to be able to work with it remotely.

24. 24
“Must have’s” to enable SKA scale visualisation
24
• Remote visualisation from archive or cluster
• Multiple representations of data
• Entirely different data organisations

25. “Must have’s” to enable visualisation
• Multiple resolutions without penalties
• Lossles & lossy compression
to save the bandwidth
• Steaming progressively
instead of cutting out
• Comprehensive support
for metadata
25

26. Current formats and frameworks can’t do it!
26

27. One of few alternatives
27

28. 28
Presentation Title (Edit in File > 'Page Setup’ > ‘Header/footer’)
28
JPEG2000 & JPIP technology

29. JPEG2000 encoding
29

30. JPIP – interaction protocol
30
Part 9 of JPEG2000 standard

31. Distributed client-server architecture
31
SkuareView
Client
SkuareView
Client
SkuareView
Server
JPX
component
JPX
component
SkuareView
Server
JPX JPX
Proxy
JPX merger

32. JPEG 2000 Key Benefits
Superior compression performance (CDF
5/3 for lossless and CDF 9/7 for lossy
compression) at low computational
requirements.
Availability of multi-component
transforms including arbitrary wavelet
transforms, arbitrary linear transforms
(e.g., KLT, block-wise KLT, etc.) with both
reversible and irreversible versions.
Superior compression efficiency and
graceful degradation (no blocking
artifacts, visually lossless compression)
http://www.aware.com/biometrics
100:1 JPEG 2000100:1 JPEG
AstroHPC’12, June 19, 2012,
Delft, The Netherlands
PERFORMANCE AND EFFICIENCY

33. JPEG 2000 Key Benefits
SCALABILITY: MULTIPLE VERSIONS
OUT OF A SINGLE COMPRESSED
IMAGE
• Multiple fidelity/resolution representation.
• Progressive transmission/recovery by fidelity or
resolution.
• Several mechanisms to support spatial random
access image regions at varying degrees of
granularity.
• Easy proxy generation.
• Bandwidth optimization and adaptive
transmission (only what’s needed)
• Different parts of the same image can be stored
using different quality (e.g. ROI at highest
quality).
33
AstroHPC’12, June 19, 2012,
Delft, The Netherlands
LOW QUALITY AREA

34. JPEG 2000 Key Benefits
FORMAT AND ACCESS
• Support of volumetric image cubes
through JP3D and 3D volumetric
compression.
34
Bruylants et al, 2007
AstroHPC’12, June 19, 2012,
Delft, The Netherlands

35. 35
JPEG 2000 Key Benefits
35
• Store existing metadata
headers
– FITS
– WCS
• Provenance
• Cataloguing
– Supports complex
geometries
– Comments/labels/links to
other files
Powerful metadata support

36. Almost any data can be compressed
Lossless
• FITS – 16.97MB
• JPEG2000 – 1.68MB
• Ratio 1:10
• FITS - 6.9 MB;
• JPEG2000 - 2.3 MB
• Ratio 1:3

37. Almost any data can be compressed
Most could be
compressed lossely to
least 1:20 ratio showing
no visually noticeable
degradation
Lossy (target PSNR=44.5 dB), ra7o 1:20, Original
• 1:100s ratio can be
achieved with adaptive
quality
Lossy

38. Almost any data can be compressed
Cosmological simulations

39. What’s the damage if lossy?
39
• ~1:10 – no difference for given precision
(<0.1%, at quanta7sa7on step 10-4)
• ~1:20 – visually lossless
• great benefits for the source finding

40. 40
40
SkuareView
New Astronomy Remote Visualisation
Framework and Tool

41. Data Reduction Pipeline
Raw data from
antennas
Channeliza7on
(PFB) Beam-forming Correla7on Calibra7on Imaging Cleaning
Spectral-imaging
data-cube
Polariza7on Map
Con7nuum Map
Catalogues
Process #1
940 – 944
MHz
MS
FITS
JPX
Processes
#2…119
4MHz
chunks
MS
FITS
JPX
Process
#120
1416-1420
MHz
MS
FITS
JPX
Cube.jpx
Cluster/Cloud

42. SkuareView implements
42
SkuareView
Client
SkuareView
Client
SkuareView
Server
JPX
component
JPX
component
SkuareView
Server
JPX JPX
Proxy
JPX
merger

43. 43
Interactive
CHILES talk-fest 19/04/2016 | JT Malarecki
43

44. Demo
The data is in AWS (US, Oregon)
1) MWA GLEAM: rgb_map_hp_trim.jpx (167MB,
raw data ~769M)
2) CHILES in AWS (500 GB as FITS, ~120GB as
JPX)
Data cubes: 120 chunks (4MHz, 256 channels)
Full data cube: cube.jpx (virtually joined chunks,
25088 channels (5 chunks are being reprocessed)
44

45. SkuareView Framework
45
AstroHPC’12, June 19, 2012,
Delft, The Netherlands

46. Astronomy Data Services at Pawsey
46