Tutorial 1 - ADASS XXVIII

Sunday, November 11 2018, 13:00-15:00

Instructor: Sébastien Derriere (CDS).



Introduction

The main goal of this tutorial is to teach participants how to use recent Virtual Observatory standards allowing exploration and querying of all-sky datasets. The Hierarchical Progressive Survey (HiPS) and the Multi-Order Coverage map (MOC) can be used by data providers to expose their datasets (images or catalogues), and astronomers can use them to perform complex queries on all-sky datasets. Participants will create image and catalogue HiPS, learn how to compare them to reference datasets, and share them in a web page. Advanced usage with the Table Access Protocol and astropy/MOCpy will also be shown.

Primary learning objectives:

  • How to generate HiPS and MOCs from example datasets, for both image and catalogue data.
  • How to visualize all-sky datasets from the full-sky view, down to the original full resolution, and how to share and publish them (for example with Aladin lite).
  • How to compare, combine and query large datasets, using CDS tools like Aladin desktop or scripts in Python.

Requirements for participants:

  • Personal laptop with at least 1GB of available disk space for data storage and processing, Wifi or network access.
  • Software - you can run the tutorial with Windows, Linux or MAC, provided you have installed:

Plan

  1. Introduction slideshow (10'). Presenting the goals of the tutorial, what HiPS and MOCs are, the tools that will be used, and the tutorial structure.
  2. Generate HiPS image survey and MOC from a collection of individual images (15')
  3. Catalogue HiPS (15'). Generate HiPS from catalogue for progressive access.
  4. Comparing with other surveys (10'). Visualization of the two HiPS in Aladin desktop, comparison with other existing surveys. Use of very large datasets like Pan-STARRS, HST images or Gaia DR2 to demonstrate the scalability.
  5. Queries by MOC (15'). Use Aladin desktop to compute the intersection of several MOCs. Use the result to query other VizieR catalogues.
  6. Publishing with Aladin Lite (15'). Embedding in a webpage with Aladin lite.
  7. Adanced usage of HiPS and MOCs
    1. Healpix in TAP/ADQL queries (20'). Visualize the rotation of the LMC in Gaia DR2 data.
    2. Programmatic use in Python (20'). astroquery.cds and MOCPy.
  8. Final questions ?

1. Generate HiPS image survey and MOC

You can use Aladin Desktop to generate a HiPS image survey from a collection of calibrated FITS images. As an example, we will use a small sample of images from the ISOPHOT 170 micron Serendipity Survey (Stickel et al. 2007), covering a fraction of the galactic plane.

If you have not downloaded the data sample yet, ask a tutor to retrieve it from a USB stick.

Make sure the 6 sample FITS images are located in a single folder, named /data/ISOPHOT for example.

/data/ISOPHOT$ ls
PSSA030+00.fits  PSSA050+00.fits  PSSA070+00.fits  PSSA090+00.fits  PSSA110+00.fits  PSSA130+00.fits

1.1 HiPS image

Launch Aladin desktop, by running the appropriate launcher, or executing: java -Xmx2048m -Xms1024m -jar Aladin.jar

In Aladin Desktop, go to Tool > Generate a HiPS based on... > An image collection (FITS, JPEG or PNG)...

You need to indicate the name of the directory containing the Source files, and the Target directory where the hierarchy of files of the HiPS will be stored. You can provide a name for the dataset.

Click on See one original image..., and notice that blank pixels have value 0, so we use this value in the form for the option Alternate blank FITS value.

Click Next>, and start the processing with the default parameters.

Click START. This will generate the FITS tiles of an image HiPS.

Tip: if the contrast or the dynamic range of pixel values is wrong, you can adjust it by zooming in on a region with valid pixels, clicking on the pixel button , and clicking on Local cut.

Once finished, we will generate PNG versions of the tiles by pressing Build in the following panel:

The cuts will control the contrast of the PNG tiles. A Min cut close to zero, and a Max cut between 150 and 600 should give acceptable results.

We will ignore the next two sections (4- Publish result, and Generate RGB) in this tutorial. More details on generating image HiPS, fine-tuning the process, providing relevant metadata, and ultimately distributing the result can be found on the page Generate your HiPS in 10 steps.

You can now visualize your image HiPS in Aladin Desktop. Try to zoom in/out, pan around, and compare this HiPS to one of the original individual FITS image.

1.2 MOC for an image survey

A MOC file has been automatically generated: try to visualize it by loading it as a local file (Ctrl+O) in the ISOPHOTHiPS directory: MOC.fits.

The default MOC has been computed at order 10. Overlay the Healpix grid to check this: (Overlay > HEALPix grid or Alt+W), and zoom in.

We can create a more detailed MOC, for example at level 11. Coverage > Generate a MOC based on... > An image collection

.

Indicate the directory containing the original images. Check "Scanning image pixel values", Alternate blank value=0, and choose MOC resolution Order 11.

.

Overlay the two MOCs and compare the resolution and properties (e.g. sky coverage) by hovering your mouse over the plane in the stack.

2. Catalogue HiPS

You can create a progressive version of a catalogue with the Hipsgen-cat tool. Detailed instructions are available here.

We will work with the MSX6C Infrared point source catalogue (Egan et al., 2003), V/114 in VizieR. The msx6_gp table contains 431711 rows, and has been extracted in VOTable base64 format in the file: vizier_votable.b64 in the data samples.

Generate the catalogue HiPS by running the following command:

java -jar Hipsgen-cat.jar -cat MSX6C -in vizier_votable.b64 -f VOT -ra _RAJ2000 -dec _DEJ2000 -lM 6 -score B2 -desc -out /path_to/MSX6C

This could take a couple of minutes: you might want to start section 3 while this is completing. The command will create a progressive version of the catalogue, with a maximum HEALPix order 6 (-lM 6), where the sources with the highest values of B2 are shown first, and lower values only when one zooms in locally.

Overlay this progressive catalogue by opening the top directory /path_to/MSX6C as a local file in Aladin desktop Ctrl+O.

3. Comparing with other surveys

HiPS can be computed for many different surveys, with a wide range of spatial coverages and spatial resolutions, and allows comparison and exploration of terabyte-sized datasets.

The data collections tree in Aladin desktop provides access to over 300 image HiPS. Try to load the progressive versions of some reference image surveys:

  • Optical > PanSTARRS > DR1 color
  • Optical > HST > HST-V
  • Infrared > HERSCHEL > Herschel SPIRE color

You can compare these various surveys by changing the opacity of the overlaid images (click on the plane name, and use the opac. Slider), or you can use the multi-view and match buttons below the main panel to compare the view and align them.

Type Gaia in the select field below the collections data tree: this will filter sources related to Gaia, and choose the progressive access to the Gaia DR2 catalogue.

Try to zoom in until all Gaia DR2 sources in the field of view are displayed (the progress bar turns full green).

Note that HiPS can be created for different data products, such as density maps of flux maps. Try to load a Gaia DR2 color flux map computed from flux values in the catalogue: Collections > Ancillary > Gaia DR2 > Color flux map

Remark: HiPS can also be applied to make progressive data cubes !

4. Queries by MOC

One can perform logical operations on MOCs (such as union, intersection), or use MOCs to perform complex spatial queries.

Load in Aladin desktop:

  • The ISOPHOT MOC we computed earlier at order 11 (should still be in your stack)
  • The HST V band MOC (Collections > Image > Optical > HST > HLA > HST-V > coverage)
  • The Galex MOC (Collections > Image > UV > GALEX > GALEX GR6 AIS color > coverage)

And compute the intersection of the 3 MOCs: Coverage > Logical operations

This will compute a new MOC describing the sky coverage covered by ISOPHOT, HST-V and Galex. This MOC can be used to query catalogues, even very large ones. Make sure (in the properties) that the area is small, select the MOC plane, select the Gaia DR2 catalogue in the collections tree, and query it "by region & MOC". You should get a few thousand sources (4263) out of the 1.7 billion.

5. Publishing with Aladin Lite

With Aladin Lite, you can easily embed HiPS in any web page. All you need is the directory structure of the HiPS to be accessible on a web server.

You can generate example code to be integrated in an HTML web page on this page.

Check the Aladin Lite API described here: https://aladin.u-strasbg.fr/AladinLite/doc/API/, which explains how to integrate custom image and catalogue HiPS in Aladin Lite.

We will try to create a simple HTML web page to embed the ISOPHOT HiPS and the progressive MSX6C catalogue you created earlier. Depending on your level of expertise in running http servers, you can choose to do it using the easy method, or the (hopefully not so) hard one:

Easy method

A copy of the HiPS data have already been installed on a web server, with the following base URLs:

Save the following piece of code as an HTML file on your computer, for example /path/to/index.html

<html>
<head>
    <!-- include Aladin Lite CSS file in the head section of your page -->
    <link rel="stylesheet" href="http://aladin.u-strasbg.fr/AladinLite/api/v2/latest/aladin.min.css" />
</head>
<body>     
    <!-- you can skip the following line if your page already integrates the jQuery library -->
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.12.1.min.js" charset="utf-8"></script>
     
    <!-- insert this snippet where you want Aladin Lite viewer to appear and after the loading of jQuery -->
    <div id="aladin-lite-div" style="width:700px;height:400px;"></div>
    <script type="text/javascript" src="http://aladin.u-strasbg.fr/AladinLite/api/v2/latest/aladin.min.js" charset="utf-8"></script>
    <script type="text/javascript">
        var aladin = A.aladin('#aladin-lite-div', {survey: "P/DSS2/color", fov:60, target: "Cepheus A"});
        aladin.setImageSurvey(aladin.createImageSurvey("ISOPHOT", "ISOPHOT / ADASS tutorial", "http://cds.unistra.fr/adass2018/ISOPHOTHiPS/", "equatorial", 10, {imgFormat: 'png'})); // setting a custom HiPS
	aladin.addCatalog(A.catalogHiPS("http://cds.unistra.fr/adass2018/MSX6C/", useProxy=false))
    </script>
</body>
</html>

As you can see, the javascript code uses these URLs to load the data into Aladin Lite. You can test your HTML file by opening it in a browser, with a local URL: file:///path/to/index.html (there are 3 / after file:).

Hard(?) method

Accessing the data locally on your computer requires to run a local http server.

You can start a simple http server on your local machine with Python 3, running the following command in the directory you want to serve on http: python3 -m http.server

This will return the corresponding URL, for example:

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

Adapt the piece of code given in the Easy method to access the local resources, for example: http://0.0.0.0:8000/ISOPHOTHiPS/. Save the HTML file, e.g. test.html in the directory where you started the http server, and open it in a browser, e.g. http://0.0.0.0:8000/test.html .

You should get a result similar to the following:

For more details on sharing your custom HiPS with the community, see: Generate your HiPS in 10 steps.

6. Adanced usage of HiPS and MOCs

6.1 Healpix in TAP/ADQL queries

We will try to run a TAP query to the Gaia DR2 catalogue to visualize the rotation pattern of the Large Magellanic Cloud.

You can do TAP queries (written in ADQL) on Gaia DR2 data in different ways:

6.1.1 Find the average proper motions in the LMC area

Go to the TAP VizieR web interface: http://tapvizier.u-strasbg.fr/adql/?I/345 for Gaia DR2, or search Gaia in the main TAP VizieR interface http://tapvizier.u-strasbg.fr/adql/.

Select I/345/gaia2 and click on Construct your query at the top right.

Enter the following ADQL query to retrieve the average proper motions within 1° of LMC.

SELECT avg(pmra) as avpmra, avg(pmdec) as avpmdec
FROM "I/345/gaia2"
WHERE CONTAINS(POINT('ICRS',ra,dec),CIRCLE('ICRS',80.8942,-69.7561,1.))=1
AND sqrt(pmra*pmra+pmdec*pmdec)<30

You can run the query. Or just click "Preview" for this simple case:

6.1.2 Proper motions in different regions of the LMC

We will try to do a map of proper motions values in the LMC, averaging values on different Healpix cells. In Aladin, locate the Gaia DR2 catalogue, and query by criteria:

This will open a window where you can type a TAP query:

Make sure to choose the ASYNC mode, and type and run the following query

SELECT avg(ra) as ra, avg(dec) as dec, avg(pmra) as pmra, avg(pmdec) as pmdec, avg(parallax) as parallax, avg(radial_velocity) as radial_velocity,
healpix(ra, dec, 8) AS h
FROM "I/345/gaia2"
WHERE CONTAINS(POINT('ICRS',ra,dec),CIRCLE('ICRS',80.8942,-69.7561,2.5))=1 and sqrt(pmra*pmra+pmdec*pmdec)<30
group by h

This may take a few minutes to execute. While the query is being processed, you can prepare the last step below. It the query fails to complete, you can download a copy of the VOTable output here

6.1.3 Visualize it in Aladin

Aladin Filters can be used to filter out and/or change the appearance of sources in catalogue planes.

Create a new Filter in Aladin Catalog > Create a filter. Go to advanced mode, and use the above values as average pmra and pmdec, yielding the following syntax:

${pmra}*${pmra}+${pmdec}*${pmdec}<6{
  draw pm(2000*(${pmra}-1.8697), 2000*(${pmdec}-0.4273)) rainbow(${pmra}*${pmra}+${pmdec}*${pmdec}, 2, 5)
}

Once the TAP query is complete, you should see the following result:

6.2 Programmatic use in Python

There is a Python interface to the CDS MOCServer, a server containing MOC and metadata informations for a large number of datasets. https://astroquery.readthedocs.io/en/latest/cds/cds.html. You can query the MOCServer with the query_region() method, and the region parameter can be a mocpy.MOC object. This allows you to find all datasets intersecting any MOC.

We will use Python commands to do the following :

  1. Download custom and official MOCs.
  2. Compute their intersection.
  3. Visualize them using pyplot.
  4. Find a selection of VizieR infrared catalogues containing data in the MOC that we have created in section 4.
  5. Visualize MOCs with Aladin Lite using ipyaladin, and overlay catalogues.
  6. Query a catalogue by MOC.

In order to ease the execution for those not familiar with Python, the commands are presented in a Jupyter notebook. Each ADASS participant has been attributed a username, in the following form: completelastname_i, where completelastname is your family name (all lower case, no spaces, but - have been kept if present) and i is the initial of your first name, preceded by a _

Launch the notebook, and press Shift+Return to execute each step.

Remark: the online notebook will not be supported after ADASS. You might want to download the list of commands of the Jupyter notebook adass2018.ipynb in order to use it after ADASS is finished.