All-sky astronomy with HiPS and MOCs

Tutorial 1 presented during ADASS XXVIII, Sunday, November 11 2018, 13:00-15:00

All-sky astronomy with HiPS and MOCs was Tutorial 1 presented during ADASS XXVIII, Sunday, November 11 2018, 13:00-15:00

This page

Instructor: Sébastien Derriere (CDS).


The main goal of this tutorial is to teach participants how to use recent Virtual Observatory standards allowing exploration and querying of all-sky datasets. The Hierarchical Progressive Survey (HiPS) and the Multi-Order Coverage map (MOC) can be used by data providers to expose their datasets (images or catalogues), and astronomers can use them to perform complex queries on all-sky datasets. Participants will create image and catalogue HiPS, learn how to compare them to reference datasets, and share them in a web page. Advanced usage with the Table Access Protocol and astropy/MOCpy will also be shown.

Primary learning objectives

Requirements for participants

Please download these data samples before attending the tutorial !

Test images and catalogues (~200MB total – will be available on USB sticks during the tutorial if needed):


1. Generate HiPS image survey and MOC

You can use Aladin Desktop to generate a HiPS image survey from a collection of calibrated FITS images. As an example, we will use a small sample of images from the ISOPHOT 170 micron Serendipity Survey (Stickel et al. 2007), covering a fraction of the galactic plane.

If you have not downloaded the data sample yet, ask a tutor to retrieve it from a USB stick.

Make sure the 6 sample FITS images are located in a single folder, named /data/ISOPHOT for example.

/data/ISOPHOT$ ls
PSSA030+00.fits  PSSA050+00.fits  PSSA070+00.fits  PSSA090+00.fits  PSSA110+00.fits  PSSA130+00.fits

1.1 HiPS image

Launch Aladin desktop, by running the appropriate launcher, or executing:

java -Xmx2048m -Xms1024m -jar Aladin.jar

In Aladin Desktop, go to :

Tool > Generate a HiPS based on... > An image collection (FITS, JPEG or PNG)...

Generate a HiPS based on an image collection

You need to indicate the name of the directory containing the Source files, and the Target directory where the hierarchy of files of the HiPS will be stored. You can provide a name for the dataset.

Click on See one original image..., and notice that blank pixels have value 0, so we use this value in the form for the option Alternate blank FITS value.

Alternate blank FITS value is 0

Click Next >, and start the processing with the default parameters.

Processing the HiPS with default parameters

Click START. This will generate the FITS tiles of an image HiPS. Tip: if the contrast or the dynamic range of pixel values is wrong, you can adjust it by zooming in on a region with valid pixels, clicking on the pixel button Aladin Desktop pixel tool icon, and clicking on Local cut. Once finished, we will generate PNG versions of the tiles by pressing Build in the following panel:

Post-processing : build the HiPS

The cuts will control the contrast of the PNG tiles. A Min cut close to zero, and a Max cut between 150 and 600 should give acceptable results.

We will ignore the next two sections (4- Publish result, and Generate RGB) in this tutorial. More details on generating image HiPS, fine-tuning the process, providing relevant metadata, and ultimately distributing the result can be found on the page Generate your HiPS in 10 steps.

You can now visualize your image HiPS in Aladin Desktop. Try to zoom in/out, pan around, and compare this HiPS to one of the original individual FITS image.

Visualize your image HiPS in Aladin Desktop

1.2 MOC for an image survey

A MOC file has been automatically generated: try to visualize it by loading it as a local file (Ctrl+O) in the ISOPHOTHiPS directory: MOC.fits.

The default MOC has been computed at order 10. Overlay the Healpix grid to check this: (Overlay > HEALPix grid or Alt+W), and zoom in.

Overlay the HEALPix grid

Zoom in the HEALPix grid to level 10

We can create a more detailed MOC, for example at level 11.

Coverage > Generate a MOC based on... > An image collection.

Generate a MOC based on an image collection

Indicate the directory containing the original images. Check Scanning image pixel values, Alternative BLANK value=0, and choose MOC resolution Order 11.

Alternative BLANK value=0 for the MOC

Overlay the two MOCs and compare the resolution and properties (e.g. sky coverage) by hovering your mouse over the plane in the stack.

2. Catalogue HiPS

You can create a progressive version of a catalogue with the Hipsgen-cat tool. Detailed instructions are available here. We will work with the MSX6C Infrared point source catalogue (Egan et al., 2003), V/114 in VizieR. The msx6_gp table contains 431711 rows, and has been extracted in VOTable base64 format in the file: vizier_votable.b64 in the data samples. Generate the catalogue HiPS by running the following command:

java -jar Hipsgen-cat.jar -cat MSX6C -in vizier_votable.b64 -f VOT -ra _RAJ2000 -dec _DEJ2000 -lM 6 -score B2 -desc -out /path_to/MSX6C

This could take a couple of minutes: you might want to start section 3 while this is completing. The command will create a progressive version of the catalogue, with a maximum HEALPix order 6 (-lM 6), where the sources with the highest values of B2 are shown first, and lower values only when one zooms in locally. Overlay this progressive catalogue by opening the top directory /path_to/MSX6C as a local file in Aladin desktop (Ctrl+O).

Overlay the progressive catalogue

3. Comparing with other surveys

HiPS can be computed for many different surveys, with a wide range of spatial coverages and spatial resolutions, and allows comparison and exploration of terabyte-sized datasets. The data collections tree in Aladin desktop provides access to over 300 image HiPS. Try to load the progressive versions of some reference image surveys:

You can compare these various surveys by changing the opacity of the overlaid images (click on the plane name, and use the opac. Slider), or you can use the multi-view and match buttons below the main panel Aladin Desktop multiview buttons to compare the view and align them.

Type Gaia in the select field below the collections data tree: this will filter sources related to Gaia, and choose the progressive access to the Gaia DR2 catalogue.

Access the progressive Gaia DR2 catalogue

Try to zoom in until all Gaia DR2 sources in the field of view are displayed (the progress bar turns full green).

HiPS can be created for different data products, such as density maps of flux maps.

Try to load a Gaia DR2 color flux map computed from flux values in the catalogue: Collections > Ancillary > Gaia DR2 > Color flux map

Gaia DR2 color flux map

HiPS can also be applied to make progressive data cubes !

4. Queries by MOC

One can perform logical operations on MOCs (such as union, intersection), or use MOCs to perform complex spatial queries. Load in Aladin desktop:

And compute the intersection of the 3 MOCs: Coverage > Logical operations

Compute the intersection of the 3 MOCs

This will compute a new MOC describing the sky coverage covered by ISOPHOT, HST-V and GALEX. This MOC can be used to query catalogues, even very large ones. Make sure (in the properties) that the area is small, select the MOC plane, select the Gaia DR2 catalogue in the collections tree, and query it by region & MOC. You should get a few thousand sources (4263) out of the 1.7 billion.

Query the Gaia DR2 catalogue by MOC

5. Publishing with Aladin Lite

With Aladin Lite, you can easily embed HiPS in any web page. All you need is the directory structure of the HiPS to be accessible on a web server. You can generate example code to be integrated in an HTML web page on this page.

Check the Aladin Lite API, which explains how to integrate custom image and catalogue HiPS in Aladin Lite.

We will try to create a simple HTML web page to embed the ISOPHOT HiPS and the progressive MSX6C catalogue you created earlier. Depending on your level of expertise in running http servers, you can choose to do it using the easy method, or the (hopefully not so) hard one:

Easy method

A copy of the HiPS data have already been installed on a web server, with the following base URLs:

Save the following piece of code as an HTML file on your computer, for example /path/to/test.html

    <meta name="description" content="Test page for ADASS 2018 Tutorial 1" />
    <!-- insert this snippet where you want Aladin Lite viewer to appear and after the loading of jQuery -->
    <div id="aladin-lite-div" style="width:700px;height:400px;"></div>
    <script type="text/javascript" src="" charset="utf-8"></script>
    <script type="text/javascript">
    let aladin;
    A.init.then(() => {
        aladin = A.aladin('#aladin-lite-div', {survey: "P/DSS2/color", fov:60}, target: "Cepheus A"});
        aladin.setImageSurvey(aladin.createImageSurvey("ISOPHOT", "ISOPHOT / ADASS tutorial", "", "equatorial", 10, {imgFormat: 'png'})); // setting a custom HiPS
        aladin.addCatalog(A.catalogHiPS("", useProxy=false))

As you can see, the javascript code uses these URLs to load the data into Aladin Lite. You can test your HTML file by opening it in a browser, with a local URL: file:///path/to/test.html (there are 3 / after file:).

Hard(?) method

Accessing the data locally on your computer requires to run a local http server. You can start a simple http server on your local machine with Python 3, running the following command in the directory you want to serve on http:

~/path/to/tutorial$ python3 -m http.server
Serving HTTP on port 8000 ( ...

This will return the URL (here where the contents of the directory ~/path/to/tutorial is made accessible.

Adapt the piece of code given in the Easy method to access the local resources, for example: Save the HTML file, e.g. test.html in the directory where you started the http server, and open it in a browser, e.g. .

You should get a result similar to the following:

For more details on sharing your custom HiPS with the community, see: Generate your HiPS in 10 steps.

6. Adanced usage of HiPS and MOCs

6.1 Healpix in TAP/ADQL queries

We will try to run a TAP query to the Gaia DR2 catalogue to visualize the rotation pattern of the Large Magellanic Cloud. You can do TAP queries (written in ADQL) on Gaia DR2 data in different ways:

6.1.1 Find the average proper motions in the LMC area

Go to the TAP VizieR web interface for Gaia DR2, or search Gaia in the main TAP VizieR interface. Select I/345/gaia2 and click on Construct your query at the top right.

TAP VizieR for the Gaia DR2 catalogue

Selection of the I/345/gaia2 table in TAP VizieR

Enter the following ADQL query to retrieve the average proper motions within 1° of LMC.

SELECT avg(pmra) as avpmra, avg(pmdec) as avpmdec
FROM "I/345/gaia2"
WHERE CONTAINS(POINT('ICRS',ra,dec),CIRCLE('ICRS',80.8942,-69.7561,1.))=1
AND sqrt(pmra*pmra+pmdec*pmdec)<30
Try it !
Loading Querying : vizier

You can run the query. Or just click "Preview" for this simple case:

Result of the ADQL query

6.1.2 Proper motions in different regions of the LMC

We will try to do a map of proper motions values in the LMC, averaging values on different Healpix cells. In Aladin, locate the Gaia DR2 catalogue, and query by criteria:

Query by criteria in Aladin Desktop

This will open a window where you can type a TAP query:

Run a TAP query in Aladin Desktop

Make sure to choose the ASYNC mode, and type and run the following query

SELECT avg(ra) as ra, avg(dec) as dec, avg(pmra) as pmra, avg(pmdec) as pmdec,
   avg(parallax) as parallax, avg(radial_velocity) as radial_velocity, healpix(ra, dec, 8) AS h
FROM "I/345/gaia2"
WHERE CONTAINS(POINT('ICRS',ra,dec),CIRCLE('ICRS',80.8942,-69.7561,2.5))=1 
   AND sqrt(pmra*pmra+pmdec*pmdec)<30
Try it !
Loading Querying : vizier

This may take a few minutes to execute. While the query is being processed, you can prepare the last step below. It the query fails to complete, you can download a copy of the VOTable output here

6.1.3 Visualize it in Aladin

Aladin Filters can be used to filter out and/or change the appearance of sources in catalogue planes. Create a new Filter in Aladin Catalog > Create a filter. Go to advanced mode, and use the above values as average pmra and pmdec, yielding the following syntax:

  draw pm(2000*(${pmra}-1.8697), 2000*(${pmdec}-0.4273)) rainbow(${pmra}*${pmra}+${pmdec}*${pmdec}, 2, 5)

Once the TAP query is complete, you should see the following result:

Visualization of the results in Aladin Desktop

6.2 Programmatic use in Python

There is a Python interface to the CDS MOCServer, a server containing MOC and metadata informations for a large number of datasets. You can query the MOCServer with the query_region() method, and the region parameter can be a mocpy.MOC object. This allows you to find all datasets intersecting any MOC.

We will use Python commands to do the following :

  1. Download custom and official MOCs.
  2. Compute their intersection.
  3. Visualize them using pyplot.
  4. Find a selection of VizieR infrared catalogues containing data in the MOC that we have created in section 4.
  5. Visualize MOCs with Aladin Lite using ipyaladin, and overlay catalogues.
  6. Query a catalogue by MOC.

In order to ease the execution for those not familiar with Python, the commands are presented in a Jupyter notebook. Each ADASS participant has been attributed a username, in the following form: completelastname_i, where completelastname is your family name (all lower case, no spaces, but - have been kept if present) and i is the initial of your first name, preceded by a _.

Launch the notebook, and press Shift+Return to execute each step.

The online notebook will not be supported after ADASS.

It is archived in the CDS github repository.