DcaJune2008CDSSaada < EuroVODCA

---+ Publishing data with Saada (images, spectra, catalogues)

---++ Abstract
Saada transforms a set of heterogeneous FITS files or VOTables of various categories (images, tables, spectra....) into a powerful database deployed on the Web and provides standard VO interfaces to the data (!ConeSearch, Simple Image Access, Simple Spectrum Access). The database (a !SaadaDB) is located on your host and stays independent of any external server. This job does not require to write any line of code.

In this session, participants will go through the following steps : creating a Saada !DataBase, ingesting the data files in the database, assigning metadata, registering the service in a VO registry.
Finally, participants will be able to access the data from various VO tools using standard interfaces provided by the Saada system.

Participants are highly encouraged to bring their own datasets, as the ultimate goal of the session is to teach how they can easily deploy VO services for their own purposes.

  
---+++ Supported systems

Saada has been successfully tested under:
   * <nop>MacOS 10.4
   * Linux 32 and 64-bits
   * Windows XP
   * Windows Vista (with difficulties: see Saada [[http://amwdb.u-strasbg.fr/saada/spip.php?article195][Web page]])

---+++ External References
   * [[http://amwdb.u-strasbg.fr/saada/][Saada]]


---+++ Advisors (CDS)
   * Laurent Michel
   * Thomas Boch
   * Sébastien Derrière


---+++ Software Requirements
Additionally to the [[DcaJune2008Software][common workshop software requirements]], this session requires:
   * [[http://www.postgresql.org/][PostgreSQL]] 8.xx
      * [[http://www.postgresql.org/ftp/binary/v8.3.1/linux/][Linux]]
      * [[http://www.postgresql.org/ftp/binary/v8.3.1/win32/][Windows]]
      * [[http://www.postgresqlformac.com/][MacOSX]]
      * Note: The administrator role must be accessible (password known)

---++ Step by Step Tutorial

---+++ Notice
   * The Saada package delivered with the workshop material is more advanced than the 1.5.beta3 available on line. We stronly suggest to use it instead of the public one.
   * The [[http://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=get&id=Aladin-beta.jnlp][Aladin 5.028 beta version]] must be used to handle <nop>VOTables with associations
   * Worshop material (software and data) can be found [[http://amwdb.u-strasbg.fr/saada/DCA2008/][here]]

---+++ Session  Case
   * A set of data focused on M33 is provided for the workshop.
      * 5 images (XMM, DSS, NVSS,  EGRET and HST)
      * 38640 sources in 5 tables (XMM, bright UV, SN remnants, variable stars and planetary nebulae)
      * 335 spectra (UV and X)
   * Following configuration parameters are related to this dataset.

---+++ <nop>PostgresQL Installation:
The details of the installation procedure depends on the plaform, but the sketch always remains the same. Shell commands are replaced  on Windows with adapted clicking. The database system initialisation is made by the Windows installer.
   * Software installation (sudo or root)
   * Database system initialisation (sudo or root): <code>initdb -D data_directory</code> . Give <code>data_directory</code> enough room to host your databases.
   * Server startup (sudo or root): <code>postgres -D data_directory</code>
   * Account setup: see Saada [[http://amwdb.u-strasbg.fr/saada/spip.php?article156][Web page]]. Roles can be created either by command lines or by using the <code>Pgadmin</code> utility. This utility is included within the Windows package. It must be installed by hand on others platforms.
      * Create the roles used by the data loader. This role must have administrator privileges <code>createuser  -s [-p password] saadmin</code>. Password is optional. 
      * Create the role used by Tomcat (Web interface). This role just requires a connect permission: <code>createuser -l [-p password] reader</code>.Password is optional.

---+++ Tomcat Installation
   * see Saada [[http://amwdb.u-strasbg.fr/saada/spip.php?article158][Web page]]. 
      * Once the installation is complete start the server (<code>TOMCAT_HOME/bin/startup.sh</code>. On Windows, Tomcat must be started with the task monitor  <code>start->programs->Tomcat->Monitor Tomcat</code>
      * Try to connect <code>http://localhost:8080</code>

---+++ Setup of the Workshop Directory
   * Create the following directories ([[http://amwdb.u-strasbg.fr/saada/spip.php?article157][doc]]):
      * =saada= Saada distribution installation directory
      * =saadadbs= Directories where database created by Saada (the <nop>SaadaDBs) will be installed. This directory <b>must not</b> be the same as <CODE>saada</CODE> directory
      * =repository= Directory where ingested files are copied in order to feed up the download facility.

---+++ Saada Installation
   * Download the Saada distribution (jar file =<nop>SaadaXXX.jar=) from =http://amwdb.u-strasbg.fr/saada/DCA2008/software=
   * Run that jar file : =java -jar <nop>SaadaXXX.jar=
   * Choose the =saada= directory created above as target for Saada install

---+++ Create an empty <nop>SaadaDB.
   * Go in =install_dir/Saada1.5.beta3/bin= and run =./newsaadadb= (see [[http://amwdb.u-strasbg.fr/saada/spip.php?article159][here]])
   * The database creation is a step by step process. System Parameters (directories, PSQL ...) values are given above. Parameters specific to the <nop>SaadaDB are given below:
      * Step 1:
|<nop>SaadaDB name| e.g. =DCAWorkshop=|
|<nop>SaadaDB Installation Directory|=saadadbs=|
|<nop>SaadaDB Repository|=repository=|
      * Step 5 (click Add for each attribute):
<table border cellspacing=1>
|<I>User Attribute Definition for images|<CODE>Survey (String)|All images will be tagged with the name of the survey they come from|
|<I>User Attribute Definition for spectra|<CODE>Domain (String)|All spectra will be tagged with their spectral range|
|<I>User Attribute Definition for entries|<CODE>ObjectClass (String)|All table entries will be tagged with their object class|

      * Step 6
|Gobal Spectral unit|=Wavelength (nm)=|The dispersion range of input spectra will be converted (as far as possible) in =nm=. This conversion relies on light speed in vacuum.|

---+++ Populate the <nop>SaadaDB.
   * Go in =install_dir/saadadbs/DCAWorkshop/bin= and run =./saadmintool= (see complete doc [[http://amwdb.u-strasbg.fr/saada/spip.php?rubrique123][here]])
   * Create a data collection <code>M33</code>
   * Loader parameters for images. 
      * Select the=Data Loader Configuration tab=
      * Select image
      * Open the data file you want to load with the "Data Sample= button
      * Fill the parameter as show below. Quoted mapping parameters are taken as constants. Others are keyword names., =Apply= and =Save=
   * Right click on the image node on the dta tree
   * Select =Load using the current configuration=

<table border=1 cellspacing=1>
<TR>
<TD>Config Name</TD><TD>Class Name</TD>
<TD>Instance Name</TD><TD>Ignored att.</TD><TD>Survey</TD>
</TR><TR>
<TD>DSS</TD><TD><code>DSS (Fusion)</TD>
<TD><code>'DSS',OBJCTRA,OBJCTDEC</TD><TD><code>AMD*</TD><TD><code>'DSS'</TD>
</TR><TR>
<TD>EPIC</TD><TD><code>EPIC_Img (Fusion)</TD>
<TD><code>'XMM',OBJECT,OBSERVER</TD><TD><code>EM*</TD><TD><code>'XMM'</TD>
</TR><TR>
<TD>NVSS</TD><TD><code>NVSS (Fusion)</TD>
<TD><code>TELESCOP,INSTRUME,OBSRA,OBSDEC</TD><TD><code></TD><TD><code>'NVSS'</TD>
</TR><TR>
<TD>HST_IRAF</TD><TD><code>HST_IRAF(Fusion)</TD>
<TD><code>OBJECT,ORIGIN</TD><TD><code></TD><TD><code>'IRAF'</TD>
</TR>
</TABLE>
 
   * Loader parameters for spectra. 
<table border=1 cellspacing=1>
<TR>
<TD nowrap>Config Name</TD><TD nowrap>Class Name</TD><TD nowrap>Instance Name</TD>
<TD nowrap>extension</TD><TD nowrap>Domain</TD>
<TD nowrap>Position</TD><TD nowrap>Spectral Range</TD>
</TR><TR>
<TD>HST</TD><TD><code>HST (Fusion)</TD><TD><code>TARGNAME
</TD>
<TD><code>#0</TD><TD><code>'UV'</TD>
<TD><code>RA_TARG,DEC_TARG mapping=only</TD>
<TD><code>range='1085 1605' unit=Angstrom mapping=only<TD></TD>
</TR><TR>
<TD>RGS</TD><TD><code>RGS_Spc (Fusion)</TD><TD><code>OBJECT
</TD>
<TD><code>SPECTRUM</TD><TD><code>'X-ray'</TD>
<TD><code>no mapping</TD>
<TD><code>range='0.6 6' unit=Kev mapping=only<TD></TD>
</TR><TR>
<TD>EPIC</TD><TD><code>EPIC_Spc (Fusion)</TD><TD><code>OBJECT
</TD>
<TD><code>SPECTRUM</TD><TD><code>'X-ray'</TD>
<TD><code>no mapping</TD>
<TD><code>range='0.2 12' unit=Kev mapping=only<TD></TD>
</TR>
</TABLE>

   * Loader parameters for tables. 
<table border=1 cellspacing=1>
<TR>
<TD nowrap>Config Name</TD><TD nowrap>Class Name</TD><TD nowrap>EntryName</TD>
<TD nowrap>extension</TD><TD nowrap>ObjectClass</TD>
<TD nowrap>Position</TD><TD nowrap>Position Error</TD>
</TR><TR>
<TD nowrap>XMM_Cat</TD><TD nowrap><CODE>XMM_Catalog (Fusion)</TD><TD nowrap><CODE>IAUNAME</TD>
<TD nowrap><code></TD><TD nowrap><CODE>'X-ray Source'</TD>
<TD nowrap><code>RA_SIAP,DEC_SIAP mapping=only</TD><TD nowrap><code>RADEC_ERR</TD>
</TR><TR>
<TD nowrap>PN</TD><TD nowrap><CODE>PlanetaryNebulae (Fusion)</TD><TD nowrap><CODE>'PN',_RAJ2000,_DEJ2000</TD>
<TD nowrap><code>#0</TD><TD nowrap><CODE>'Planetary Nebulae'</TD>
<TD nowrap><code></TD><TD nowrap></TD></TR>
</TR><TR>
<TD nowrap>SNRemnants</TD><TD nowrap><CODE>SNRemnants (Fusion)</TD><TD nowrap><CODE>'SNR',_RAJ2000,_DEJ2000</TD>
<TD nowrap><code>#1</TD><TD nowrap><CODE>'SN Remnant'</TD>
<TD nowrap><code></TD><TD nowrap></TD>
</TR><TR>
<TD nowrap>UVBright</TD><TD nowrap><CODE>UVBright (Fusion)</TD><TD nowrap><CODE>'UVB',_RAJ2000,_DEJ2000</TD>
<TD nowrap><code>#0</TD><TD nowrap><CODE>'UV Bright'</TD>
<TD nowrap><code></TD><TD nowrap></TD>
</TR><TR>
<TD nowrap>VariableStar</TD><TD nowrap><CODE>VariableStar (Fusion)</TD><TD nowrap><CODE>'Var',_RAJ2000,_DEJ2000</TD>
<TD nowrap><code>#1</TD><TD nowrap><CODE>'Variable Star</TD>
<TD nowrap><code></TD><TD nowrap></TD>
</TR>
</TABLE>


   * Relationships.
<table border=1 cellspacing=1>
<TR>
<TD nowrap>Name</TD><TD nowrap>Starting Collection</TD><TD nowrap>EndingCollection</TD>
<TD nowrap>Qualifier</TD><TD nowrap>Correlator</TD>
</TR><TR> 
<TD nowrap>SourcesInField</TD>
<TD nowrap><code>M33.IMAGE</TD><TD nowrap><code>M33.ENTRY</TD>
<TD nowrap><code></TD><TD>Trivial  (All images linked with all entries) e.g. <code>p.oidsaada > 0</TD>
</TR><TR> 
<TD nowrap>CandidateSpectra</TD>
<TD nowrap><code>M33.ENTRY</TD><TD nowrap><code>M33.SPECTRUM</TD>
<TD nowrap><code>distance</TD><TD>1st spectrum at less than 10 arcsec</TD>
</TR>
</TABLE>

---+++ <nop>SaadaDB access.

Your database can be accessed by the Web once it has been deployed (=Deply Web App.= button on the admin tool).
   * The base address of the <nop>SaadaDB is [[http://localhost:8080/DCAWorkshop][http://localhost:8080/DCAWorkshop]]
   * The Web interface relies on <nop>SaadaQL queries (see doc [[http://amwdb.u-strasbg.fr/saada/spip.php?rubrique124][hre])
   * Simple queries can be edited from the form on the bottom if the data page.
   * The =Refine Query= opens a smart editor of <nop>SaadaQL.
   * Query Examples:
      * Select any entries at less than 1 arcmin from the M33 center
<PRE><code>        Select ENTRY  From * In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }</code></PRE>

      * Select any entries at less than 1 arcmin from the M33 center and having one candidate spectrum
<PRE><code>        Select ENTRY  From * In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }
        WhereRelation {
             matchPattern {
                     CandidateSpectra
             }
        }</code></PRE>

      * Select variable stars  at less than 1 arcmin from the M33 center and having one candidate spectrum
<PRE><code>        Select ENTRY  From VariableStarEntry In M33
        WherePosition{ isInCircle("M33",1.0,J2000,FK5) }
        WhereRelation {
             matchPattern {
                     CandidateSpectra
             }
        }</code></PRE>

---+++ VO Publishing

The <nop>SaadaDB comes with 3 VO services (one SIAP for images, one Cone Search for sources and one SSA for spectra). These services  can accessed by the voportal page =http://localhost:8080/DCAWorkshop/voportal=
Notice that your database can only be accessed from your local machine because its base URL was set as =http://localhost...=. It should be run on a machine with a fixed IP address to be visible from other PCs.

   * VO access from the Web interface:
      * VO services  can accessed by the voportal page =http://localhost:8080/DCAWorkshop/voportal=
      * Each Saada collection can be seen as one VO resource as well as the whole database (collection =ANY=).
      * Saada VO queries always return <nop>VOtables.
      * The =XML= button makes <nop>VOtables to be displayed on the browser
      * The =Aladin= button starts the Aladin applet with he returned table. This feature does not support associations.

   * VO access from the GLU
      * A local registry (a =gludic= can used by Aladin). All resources declared in that =gludic= will be seen by your local Aladin as any other VO resource. 
      * GLU marks (similar to registry records) are delivered by the vo portal when the =Glu Mark= box is checked.
      * Copy these marks in a file (e.g. =gludic.txt=)
      * Run aladin with the following command: =java -jar Aladin.jar -glufile=/PATHTO/gludic.txt=
      * VO resources of your <nop>SaadaDB will appear and can be queried from the =load= window (menu =File-->Open=).  

   * VO access from the registry
      * The =Registry Record= check box returns an XML template of a valid registry record. This template can be completed by hand and declared in a registry. Your <nop>SaadaDB becomes then a VO resource.
      * Notice that being declared in a VO registry supposes your services to have a world wide access and a high disponibility.
      * Valid registry records can be downloaded [[http://amwdb.u-strasbg.fr/saada/DCA2008/resources/registry][here]] 
         * Update these files with your identity
         * Cut and past file content [[http://vops1.hq.eso.org:8080/registry/admin/editEntry.jsp][here]] (user name: registry , password : test ) and  submit.
         * The declared resources are visible from the Aladin =All VO= frame :
            * Go in menu =File-->Open=
            * Click in the =All VO= tab, and then click on =Detailed list=
            * In the new window, select =Query workshop registry= and click =Refresh=
            * You should see your registered services in the displayed list
            * Select and query one of your service using target =M33=
         * Published services can be accessed by any client (e.g. Aladin) running  in local.

---+++ Meta Data Mapping

Once data are ingested in the <nop>SaadaDB, units, UCDs and Utypes can be mapped on keywords (see doc [[http://amwdb.u-strasbg.fr/saada/spip.php?article179][here]]). The mapping is set class by class:
   * Double click on the class icon on the data tree. That opens the meta data tab.
   * Right click on the metadata tab and select =Tag Meta Data=
   * Drag and Drop metadata from the relevent tree to the metadata row.
   * Right click on the metadata tab and select =Save Meta Data= 
   * Deploy the Web interface again.

These metadata are used by <nop>SaadaQL to express constraints:
   * Set UCD <CODE>phot.mag;em.opt.V</code> on the column Vmag on the <code>VariableStar</code> Class .
   * Set this keyword as <code>queriable</code> (means queriable by UCD)
   * Do the same for the <code>UVBright</code> class.
   * Try the following query:
<PRE><code>          Select ENTRY  From * In M33
          WhereUCD {
                  [phot.mag;em.opt.V] [] (19.7,20) [unit]
          }</code></PRE>

---
   * People intersted in using Saada can subscribe the [[http://amwdb.u-strasbg.fr/saada/spip.php?article56][mail list]] (low rate - spam free)
   * [[http://cds.u-strasbg.fr/twikiDCA/bin/view/EuroVODCA/DcaJune2008ShowcaseAndReview][Session Review]]
   * [[%ATTACHURL%/DCA_Workshop_2008__saada.pdf][DCA_Workshop_2008__saada.pdf]]: Introduction to Saada for the hands-on session
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	DCA_Workshop_2008__saada.pdf	r1	manage	2485.2 K	26 Jun 2008 - 17:00	LaurentMICHEL	Introduction to Saada for the hands-on session
This topic: EuroVODCA > DcaJune2008Workshop > DcaJune2008CDSSaada
Topic revision: r24 - 29 Jun 2008 - LaurentMICHEL
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback