Publishing data with Saada (images, spectra, catalogues)

Abstract

Saada transforms a set of heterogeneous FITS files or VOTables of various categories (images, tables, spectra....) into a powerful database deployed on the Web and provides standard VO interfaces to the data (ConeSearch, Simple Image Access, Simple Spectrum Access). The database (a SaadaDB) is located on your host and stays independent of any external server. This job does not require to write any line of code.

In this session, participants will go through the following steps : creating a Saada DataBase, ingesting the data files in the database, assigning metadata, registering the service in a VO registry. Finally, participants will be able to access the data from various VO tools using standard interfaces provided by the Saada system.

Participants are highly encouraged to bring their own datasets, as the ultimate goal of the session is to teach how they can easily deploy VO services for their own purposes.

Supported systems

Saada has been successfully tested under:

  • MacOS 10.4
  • Linux 32 and 64-bits
  • Windows XP
  • Windows Vista (with difficulties: see Saada Web page)

External References

Advisors (CDS)

  • Laurent Michel
  • Thomas Boch
  • Sébastien Derrière

Software Requirements

Additionally to the common workshop software requirements, this session requires:

Step by Step Tutorial

Notice

  • The Saada package delivered with the workshop material is more advanced than the 1.5.beta3 available on line. We stronly suggest to use it instead of the public one.
  • The Aladin 5.028 beta version must be used to handle VOTables with associations
  • Worshop material (software and data) can be found here

Session Case

  • A set of data focused on M33 is provided for the workshop.
    • 5 images (XMM, DSS, NVSS, EGRET and HST)
    • 38640 sources in 5 tables (XMM, bright UV, SN remnants, variable stars and planetary nebulae)
    • 335 spectra (UV and X)
  • Following configuration parameters are related to this dataset.

PostgresQL Installation:

The details of the installation procedure depends on the plaform, but the sketch always remains the same. Shell commands are replaced on Windows with adapted clicking. The database system initialisation is made by the Windows installer.
  • Software installation (sudo or root)
  • Database system initialisation (sudo or root): initdb -D data_directory . Give data_directory enough room to host your databases.
  • Server startup (sudo or root): postgres -D data_directory
  • Account setup: see Saada Web page. Roles can be created either by command lines or by using the Pgadmin utility. This utility is included within the Windows package. It must be installed by hand on others platforms.
    • Create the roles used by the data loader. This role must have administrator privileges createuser -s [-p password] saadmin. Password is optional.
    • Create the role used by Tomcat (Web interface). This role just requires a connect permission: createuser -l [-p password] reader.Password is optional.

Tomcat Installation

  • see Saada Web page.
    • Once the installation is complete start the server (TOMCAT_HOME/bin/startup.sh. On Windows, Tomcat must be started with the task monitor start->programs->Tomcat->Monitor Tomcat
    • Try to connect http://localhost:8080

Setup of the Workshop Directory

  • Create the following directories (doc):
    • saada Saada distribution installation directory
    • saadadbs Directories where database created by Saada (the SaadaDBs) will be installed. This directory must not be the same as saada directory
    • repository Directory where ingested files are copied in order to feed up the download facility.

Saada Installation

  • Download the Saada distribution (jar file SaadaXXX.jar) from http://amwdb.u-strasbg.fr/saada/DCA2008/software
  • Run that jar file : java -jar SaadaXXX.jar
  • Choose the saada directory created above as target for Saada install

Create an empty SaadaDB.

  • Go in install_dir/Saada1.5.beta3/bin and run ./newsaadadb (see here)
  • The database creation is a step by step process. System Parameters (directories, PSQL ...) values are given above. Parameters specific to the SaadaDB are given below:
    • Step 1:
SaadaDB name e.g. DCAWorkshop
SaadaDB Installation Directory saadadbs
SaadaDB Repository repository
    • Step 5 (click Add for each attribute):
User Attribute Definition for images Survey (String) All images will be tagged with the name of the survey they come from
User Attribute Definition for spectra Domain (String) All spectra will be tagged with their spectral range
User Attribute Definition for entries ObjectClass (String) All table entries will be tagged with their object class

    • Step 6
Gobal Spectral unit Wavelength (nm) The dispersion range of input spectra will be converted (as far as possible) in nm. This conversion relies on light speed in vacuum.

Populate the SaadaDB.

  • Go in install_dir/saadadbs/DCAWorkshop/bin and run ./saadmintool (see complete doc here)
  • Create a data collection M33
  • Loader parameters for images.
    • Select the=Data Loader Configuration tab=
    • Select image
    • Open the data file you want to load with the "Data Sample= button
    • Fill the parameter as show below. Quoted mapping parameters are taken as constants. Others are keyword names., Apply and Save
  • Right click on the image node on the dta tree
  • Select Load using the current configuration

Config NameClass Name Instance NameIgnored att.Survey
DSSDSS (Fusion) 'DSS',OBJCTRA,OBJCTDECAMD*'DSS'
EPICEPIC_Img (Fusion) 'XMM',OBJECT,OBSERVEREM*'XMM'
NVSSNVSS (Fusion) TELESCOP,INSTRUME,OBSRA,OBSDEC'NVSS'
HST_IRAFHST_IRAF(Fusion) OBJECT,ORIGIN'IRAF'

  • Loader parameters for spectra.
Config NameClass NameInstance Name extensionDomain PositionSpectral Range
HSTHST (Fusion)TARGNAME #0'UV' RA_TARG,DEC_TARG mapping=only range='1085 1605' unit=Angstrom mapping=only
RGSRGS_Spc (Fusion)OBJECT SPECTRUM'X-ray' no mapping range='0.6 6' unit=Kev mapping=only
EPICEPIC_Spc (Fusion)OBJECT SPECTRUM'X-ray' no mapping range='0.2 12' unit=Kev mapping=only

  • Loader parameters for tables.
Config NameClass NameEntryName extensionObjectClass PositionPosition Error
XMM_CatXMM_Catalog (Fusion)IAUNAME 'X-ray Source' RA_SIAP,DEC_SIAP mapping=onlyRADEC_ERR
PNPlanetaryNebulae (Fusion)'PN',_RAJ2000,_DEJ2000 #0'Planetary Nebulae'
SNRemnantsSNRemnants (Fusion)'SNR',_RAJ2000,_DEJ2000 #1'SN Remnant'
UVBrightUVBright (Fusion)'UVB',_RAJ2000,_DEJ2000 #0'UV Bright'
VariableStarVariableStar (Fusion)'Var',_RAJ2000,_DEJ2000 #1'Variable Star

  • Relationships.
NameStarting CollectionEndingCollection QualifierCorrelator
SourcesInField M33.IMAGEM33.ENTRY Trivial (All images linked with all entries) e.g. p.oidsaada > 0
CandidateSpectra M33.ENTRYM33.SPECTRUM distance1st spectrum at less than 10 arcsec

SaadaDB access.

Your database can be accessed by the Web once it has been deployed (Deply Web App. button on the admin tool).

  • The base address of the SaadaDB is http://localhost:8080/DCAWorkshop
  • The Web interface relies on SaadaQL queries (see doc [[http://amwdb.u-strasbg.fr/saada/spip.php?rubrique124][hre])
  • Simple queries can be edited from the form on the bottom if the data page.
  • The Refine Query opens a smart editor of SaadaQL.
  • Query Examples:
    • Select any entries at less than 1 arcmin from the M33 center
        Select ENTRY  From * In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }

    • Select any entries at less than 1 arcmin from the M33 center and having one candidate spectrum
        Select ENTRY  From * In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }
        WhereRelation {
             matchPattern {
                     CandidateSpectra
             }
        }

    • Select variable stars at less than 1 arcmin from the M33 center and having one candidate spectrum
        Select ENTRY  From VariableStarEntry In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }
        WhereRelation {
             matchPattern {
                     CandidateSpectra
             }
        }

VO Publishing

The SaadaDB comes with 3 VO services (one SIAP for images, one Cone Search for sources and one SSA for spectra). These services can accessed by the voportal page http://localhost:8080/DCAWorkshop/voportal Notice that your database can only be accessed from your local machine because its base URL was set as http://localhost.... It should be run on a machine with a fixed IP address to be visible from other PCs.

  • VO access from the Web interface:
    • VO services can accessed by the voportal page http://localhost:8080/DCAWorkshop/voportal
    • Each Saada collection can be seen as one VO resource as well as the whole database (collection ANY).
    • Saada VO queries always return VOtables.
    • The XML button makes VOtables to be displayed on the browser
    • The Aladin button starts the Aladin applet with he returned table. This feature does not support associations.

  • VO access from the GLU
    • A local registry (a gludic can used by Aladin). All resources declared in that gludic will be seen by your local Aladin as any other VO resource.
    • GLU marks (similar to registry records) are delivered by the vo portal when the Glu Mark box is checked.
    • Copy these marks in a file (e.g. gludic.txt)
    • Run aladin with the following command: java -jar Aladin.jar -glufile=/PATHTO/gludic.txt
    • VO resources of your SaadaDB will appear and can be queried from the load window (menu File-->Open).

  • VO access from the registry
    • The Registry Record check box returns an XML template of a valid registry record. This template can be completed by hand and declared in a registry. Your SaadaDB becomes then a VO resource.
    • Notice that being declared in a VO registry supposes your services to have a world wide access and a high disponibility.
    • Valid registry records can be downloaded here
      • Update these files with your identity
      • Cut and past file content here (user name: registry , password : test ) and submit.
      • The declared resources are visible from the Aladin All VO frame :
        • Go in menu File-->Open
        • Click in the All VO tab, and then click on Detailed list
        • In the new window, select Query workshop registry and click Refresh
        • You should see your registered services in the displayed list
        • Select and query one of your service using target M33
      • Published services can be accessed by any client (e.g. Aladin) running in local.

Meta Data Mapping

Once data are ingested in the SaadaDB, units, UCDs and Utypes can be mapped on keywords (see doc here). The mapping is set class by class:

  • Double click on the class icon on the data tree. That opens the meta data tab.
  • Right click on the metadata tab and select Tag Meta Data
  • Drag and Drop metadata from the relevent tree to the metadata row.
  • Right click on the metadata tab and select Save Meta Data
  • Deploy the Web interface again.

These metadata are used by SaadaQL to express constraints:

  • Set UCD phot.mag;em.opt.V on the column Vmag on the VariableStar Class .
  • Set this keyword as queriable (means queriable by UCD)
  • Do the same for the UVBright class.
  • Try the following query:
          Select ENTRY  From * In M33
          WhereUCD {
                  [phot.mag;em.opt.V] [] (19.7,20) [unit]
          }


Topic attachments
I Attachment History Action Size Date Who Comment
Pdfpdf DCA_Workshop_2008__saada.pdf r1 manage 2485.2 K 26 Jun 2008 - 17:00 LaurentMICHEL Introduction to Saada for the hands-on session
Edit | Attach | Watch | Print version | History: r24 < r23 < r22 < r21 < r20 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r24 - 29 Jun 2008 - LaurentMICHEL
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback