Publishing data with Saada (images, spectra, catalogues)

Abstract

Saada transforms a set of heterogeneous FITS files or VOTables of various categories (images, tables, spectra....) into a powerful database deployed on the Web and provides standard VO interfaces to the data (ConeSearch, Simple Image Access, Simple Spectrum Access). The database (a SaadaDB) is located on your host and stays independent of any external server. This job does not require to write any line of code.

In this session, participants will go through the following steps : creating a Saada DataBase, ingesting the data files in the database, assigning metadata, registering the service in a VO registry. Finally, participants will be able to access the data from various VO tools using standard interfaces provided by the Saada system.

Participants are highly encouraged to bring their own datasets, as the ultimate goal of the session is to teach how they can easily deploy VO services for their own purposes.

Supported systems

Saada has been successfully tested under:

• MacOS 10.4
• Linux 32 and 64-bits
• Windows XP
• Windows Vista (with difficulties: see Saada Web page)

External References

• Saada

Advisors (CDS)

• Laurent Michel
• Thomas Boch
• Sébastien Derrière

Software Requirements

Additionally to the common workshop software requirements, this session requires:

• PostgreSQL 8.xx
  • Linux
  • Windows
  • MacOSX
  • Note: The administrator role must be accessible (password known)

Step by Step Tutorial

Notice

• The Saada package delivered with the workshop material is more advanced than the 1.5.beta3 available on line. We stronly suggest to use it instead of the public one.
• The Aladin 5.028 beta version must be used to handle VOTables with associations
• Worshop material (software and data) can be found here

Session Case

• A set of data focused on M33 is provided for the workshop.
  • 5 images (XMM, DSS, NVSS, EGRET and HST)
  • 38640 sources in 5 tables (XMM, bright UV, SN remnants, variable stars and planetary nebulae)
  • 335 spectra (UV and X)
• Following configuration parameters are related to this dataset.

PostgresQL Installation:

The details of the installation procedure depends on the plaform, but the sketch always remains the same. Shell commands are replaced on Windows with adapted clicking. The database system initialisation is made by the Windows installer.

• Software installation (sudo or root)
• Database system initialisation (sudo or root): initdb -D data_directory . Give data_directory enough room to host your databases.
• Server startup (sudo or root): postgres -D data_directory
• Account setup: see Saada Web page. Roles can be created either by command lines or by using the Pgadmin utility. This utility is included within the Windows package. It must be installed by hand on others platforms.
  • Create the roles used by the data loader. This role must have administrator privileges createuser -s [-p password] saadmin. Password is optional.
  • Create the role used by Tomcat (Web interface). This role just requires a connect permission: createuser -l [-p password] reader.Password is optional.

Tomcat Installation

• see Saada Web page.
  • Once the installation is complete start the server (TOMCAT_HOME/bin/startup.sh. On Windows, Tomcat must be started with the task monitor start->programs->Tomcat->Monitor Tomcat
  • Try to connect http://localhost:8080

Setup of the Workshop Directory

• Create the following directories (doc):
  • saada Saada distribution installation directory
  • saadadbs Directories where database created by Saada (the SaadaDBs) will be installed. This directory must not be the same as saada directory
  • repository Directory where ingested files are copied in order to feed up the download facility.

Saada Installation

• Download the Saada distribution (jar file SaadaXXX.jar) from http://amwdb.u-strasbg.fr/saada/DCA2008/software
• Run that jar file : java -jar SaadaXXX.jar
• Choose the saada directory created above as target for Saada install

Create an empty SaadaDB.

• Go in install_dir/Saada1.5.beta3/bin and run ./newsaadadb (see here)
• The database creation is a step by step process. System Parameters (directories, PSQL ...) values are given above. Parameters specific to the SaadaDB are given below:
  • Step 1:

SaadaDB name	e.g. DCAWorkshop
SaadaDB Installation Directory	saadadbs
SaadaDB Repository	repository

• • Step 5 (click Add for each attribute):

User Attribute Definition for images	Survey (String)	All images will be tagged with the name of the survey they come from
User Attribute Definition for spectra	Domain (String)	All spectra will be tagged with their spectral range
User Attribute Definition for entries	ObjectClass (String)	All table entries will be tagged with their object class

• • Step 6

Gobal Spectral unit

Wavelength (nm)

The dispersion range of input spectra will be converted (as far as possible) in nm. This conversion relies on light speed in vacuum.

Populate the SaadaDB.

• Go in install_dir/saadadbs/DCAWorkshop/bin and run ./saadmintool (see complete doc here)
• Create a data collection M33
• Loader parameters for images.
  • Select the=Data Loader Configuration tab=
  • Select image
  • Open the data file you want to load with the "Data Sample= button
  • Fill the parameter as show below. Quoted mapping parameters are taken as constants. Others are keyword names., Apply and Save
• Right click on the image node on the dta tree
• Select Load using the current configuration

Config Name	Class Name	Instance Name	Ignored att.	Survey
DSS	DSS (Fusion)	'DSS',OBJCTRA,OBJCTDEC	AMD*	'DSS'
EPIC	EPIC_Img (Fusion)	'XMM',OBJECT,OBSERVER	EM*	'XMM'
NVSS	NVSS (Fusion)	TELESCOP,INSTRUME,OBSRA,OBSDEC		'NVSS'
HST_IRAF	HST_IRAF(Fusion)	OBJECT,ORIGIN		'IRAF'

• Loader parameters for spectra.

Config Name	Class Name	Instance Name	extension	Domain	Position	Spectral Range
HST	HST (Fusion)	TARGNAME	#0	'UV'	RA_TARG,DEC_TARG mapping=only	range='1085 1605' unit=Angstrom mapping=only
RGS	RGS_Spc (Fusion)	OBJECT	SPECTRUM	'X-ray'	no mapping	range='0.6 6' unit=Kev mapping=only
EPIC	EPIC_Spc (Fusion)	OBJECT	SPECTRUM	'X-ray'	no mapping	range='0.2 12' unit=Kev mapping=only

• Loader parameters for tables.

Config Name	Class Name	EntryName	extension	ObjectClass	Position	Position Error
XMM_Cat	XMM_Catalog (Fusion)	IAUNAME		'X-ray Source'	RA_SIAP,DEC_SIAP mapping=only	RADEC_ERR
PN	PlanetaryNebulae (Fusion)	'PN',_RAJ2000,_DEJ2000	#0	'Planetary Nebulae'
SNRemnants	SNRemnants (Fusion)	'SNR',_RAJ2000,_DEJ2000	#1	'SN Remnant'
UVBright	UVBright (Fusion)	'UVB',_RAJ2000,_DEJ2000	#0	'UV Bright'
VariableStar	VariableStar (Fusion)	'Var',_RAJ2000,_DEJ2000	#1	'Variable Star

• Relationships.

Name	Starting Collection	EndingCollection	Qualifier	Correlator
SourcesInField	M33.IMAGE	M33.ENTRY		Trivial (All images linked with all entries) e.g. p.oidsaada > 0
CandidateSpectra	M33.ENTRY	M33.SPECTRUM	distance	1st spectrum at less than 10 arcsec

SaadaDB access.

Your database can be accessed by the Web once it has been deployed (Deply Web App. button on the admin tool).

• The base address of the SaadaDB is http://localhost:8080/DCAWorkshop
• The Web interface relies on SaadaQL queries (see doc [[http://amwdb.u-strasbg.fr/saada/spip.php?rubrique124][hre])
• Simple queries can be edited from the form on the bottom if the data page.
• The Refine Query opens a smart editor of SaadaQL.
• Query Examples:
  • Select any entries at less than 1 arcmin from the M33 center

        Select ENTRY  From * In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }

• • Select any entries at less than 1 arcmin from the M33 center and having one candidate spectrum

        Select ENTRY  From * In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }
        WhereRelation {
             matchPattern {
                     CandidateSpectra
             }
        }

• • Select variable stars at less than 1 arcmin from the M33 center and having one candidate spectrum

        Select ENTRY  From VariableStarEntry In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }
        WhereRelation {
             matchPattern {
                     CandidateSpectra
             }
        }

VO Publishing

The SaadaDB comes with 3 VO services (one SIAP for images, one Cone Search for sources and one SSA for spectra). These services can accessed by the voportal page http://localhost:8080/DCAWorkshop/voportal Notice that your database can only be accessed from your local machine because its base URL was set as http://localhost.... It should be run on a machine with a fixed IP address to be visible from other PCs.

• VO access from the Web interface:
  • VO services can accessed by the voportal page http://localhost:8080/DCAWorkshop/voportal
  • Each Saada collection can be seen as one VO resource as well as the whole database (collection ANY).
  • Saada VO queries always return VOtables.
  • The XML button makes VOtables to be displayed on the browser
  • The Aladin button starts the Aladin applet with he returned table. This feature does not support associations.

• VO access from the GLU
  • A local registry (a gludic can used by Aladin). All resources declared in that gludic will be seen by your local Aladin as any other VO resource.
  • GLU marks (similar to registry records) are delivered by the vo portal when the Glu Mark box is checked.
  • Copy these marks in a file (e.g. gludic.txt)
  • Run aladin with the following command: java -jar Aladin.jar -glufile=/PATHTO/gludic.txt
  • VO resources of your SaadaDB will appear and can be queried from the load window (menu File-->Open).

• VO access from the registry
  • The Registry Record check box returns an XML template of a valid registry record. This template can be completed by hand and declared in a registry. Your SaadaDB becomes then a VO resource.
  • Notice that being declared in a VO registry supposes your services to have a world wide access and a high disponibility.
  • Valid registry records can be downloaded here
    • Update these files with your identity
    • Cut and past file content here (user name: registry , password : test ) and submit.
    • The declared resources are visible from the Aladin All VO frame :
      • Go in menu File-->Open
      • Click in the All VO tab, and then click on Detailed list
      • In the new window, select Query workshop registry and click Refresh
      • You should see your registered services in the displayed list
      • Select and query one of your service using target M33
    • Published services can be accessed by any client (e.g. Aladin) running in local.

Meta Data Mapping

Once data are ingested in the SaadaDB, units, UCDs and Utypes can be mapped on keywords (see doc here). The mapping is set class by class:

• Double click on the class icon on the data tree. That opens the meta data tab.
• Right click on the metadata tab and select Tag Meta Data
• Drag and Drop metadata from the relevent tree to the metadata row.
• Right click on the metadata tab and select Save Meta Data
• Deploy the Web interface again.

These metadata are used by SaadaQL to express constraints:

• Set UCD phot.mag;em.opt.V on the column Vmag on the VariableStar Class .
• Set this keyword as queriable (means queriable by UCD)
• Do the same for the UVBright class.
• Try the following query:

          Select ENTRY  From * In M33
          WhereUCD {
                  [phot.mag;em.opt.V] [] (19.7,20) [unit]
          }

---

• People intersted in using Saada can subscribe the mail list (low rate - spam free)
• Session Review
• DCA_Workshop_2008__saada.pdf: Introduction to Saada for the hands-on session