Tags:
create new tag
, view all tags
-- GillesLandais - 2017-07-08

Alertes du superviseur GLU

Note: complete documentation are available on /home/cds-survie

voir aussi Alertsesame

GLU ALERT IN RED ARE CRTITICAL

GLU ALERT IN ORANGE ARE CRITIC

(*) login as postgres : postgres unix account has no password. To login as postgres need to log before as root :
sudo su
su – postgres

Glu alert

Error

TODO

QueryCatDaemon.axel

server : axel

Big catalogue access


only for FX format

Note :It could be unavailable temporarily

Restarting the service :

test on server if started:
systemctl status cds.catfile.service


stop the service
systemctl stop cds.catfile.service


start the service
systemctl start cds.catfile.service

QueryCatDaemon.cfa

server : CfA mirror

Big catalogue access in CfA

only for FX format

If service is down, VizieR switch automatically to QueryCatDaemon.axel
(this is just an optimisation)

Restart vizier docker container :

docker stop vizier2
docker rm vizier2
(cd dockers ;./start-vizier-docker)

contact ADS : Alberto Accomazzi

QueryCatDaemon.ch

server : China mirror

(vizier.china-vo.org)

Big catalogue access in china

only for FX format

If service is down, VizieR switch automatically to QueryCatDaemon.axel
(this is just an optimisation)

Restart the service :
connect chineese server and execute :
~cds/init.d/QueryCatDaemon.sh

QueryCatDaemon.iucaa

server : IUCAA mirror
(14.139.108.115)

Big catalogue access in India and Japan

only for FX format

If service is down, VizieR switch automatically to QueryCatDaemon.axel
(this is just an optimisation)

Restart vizier docker container :

docker stop vizier2
docker rm vizier2
(cd dockers ;./start-vizier-docker)

contact ADS : Ajit Kembhavi

QueryCatDaemon.vizier

server : vizier

Big catalogues access (local access)

only for FX format

If service is down, VizieR switch automatically to QueryCatDaemon.axel

(this is just an optimisation)

test is started :
/etc/init.d/queryCatDaemon status

start/stop :
/etc/init.d/queryCatDaemon start/stop

Viz.monbigcat

server : axel

Big catalogues access

FX and Ochsenbein format

Note :It could be unavailable temporarily

Note : failures means that some big catalogues are unavailable (but not all)!


- FX format:
see
QueryCatDaemon.axel


- Ochsenbein format:
test :
systemctl status aserver.service

start:
systemctl start aserver.service

Viz.mondb

server : vizdata

VizieR database access (prod)

Note : failure doesn't mean that VizieR is unreachable because VizieR can switch automatically to mirror using the configuration file in vizier server: /home/cds/interfaces.

VizieR try the first line, if it fails, the second, …

If all access are unreachable, the crontab will redirect (15minutes) queries to a mirror.

If slow, it could be due to a pic of activity: see the supervisor :

http://cdsarc.u-strasbg.fr/spv-bin/Vsystem.cgi

Else,

TODO :

resart postgres :

/etc/init.d/postgresql-pcds stop
/etc/init.d/postgresql-pcds start

If problem pesrists , it is certainly due to persistent temporary tables :


list temporary tables per schema
~postgres/sql/clean_temporary_table.sh -l

The number of temporary tables should not overpasses 1000.
if limit is reached, start :
clean_temporary_table.sh -e schema_name

Health tests scripts:
~postgres/sql/stat_usage.sh

Test if connection is back
http://http://vizier.u-strasbg.fr/local/viz-bin/VizieR
if « db1_open(metaviz@PCDSPOOL) »
is in the debug part , then it is ok

Viz.mondb.local

server : cdsarc
VizieR local database

(used by diocumentalist for ingestion)

TODO : see action on Viz.mondb but applied on cdsarc server instead of vizdata

Viz.mondbpool

server : vizdata

VizieR database access (prod)

idem that Viz.mondb

Idem that Viz.mondb

Viz.montap

server : tapvizier

TAPVizieR service

Restart tomcat service :
/etc/init.d/tomcat stop
/etc/init.d/tomcat start

Attention : des bloquages peuvent apparaître au redémarage de tomcat (voir le bouclage dans les logs tomcat) –
dans ce case executer la commande de nettoyage des jobs TAP:

(cd /var/local/TAP-repository/tapvizier1/prod; ./clean.py > tmp.tmp; sh tmp.tmp)

VizU.cds

server : vizier

VizieR HTTP service

Test the database connection

(see Viz.mondb)

test the server access via http/ssh

http://vizier.u-strasbg.fr

VizU.adac
VizU.cadc
VizU.cfa
VizU.ch
VizU.hw
VizU.iucaa
VizU.saao
VizU.uk

server : (VizieR) mirrors

Vizier mirror

Contact the mirror administrator

VizieR.AssocData.ObsTAP

servers : cdsarc, vizdata

Assoc data service

Restart tomcat service (cdsarc)

/etc/init.d/tomcat7 stop
/etc/init.d/tomcat7 start


If problem persists :

Test the database : (vizdata)

test : (login as postgres (*))
psql -p 5433 saadavizier


restart :
/etc/init.d/postgresql-saada stop
/etc/init.d/postgresql-saada start

VizieR.AssocData.SIA

server : cdsarc, vizdata

Assoc data service (SIA)

See VizieR.AssocData.ObsTAP

getcat

server : cdsarc

ReadMe access

Too much connection

or HTTP not reachable

Note :It could be unavailable temporarily

Test the server access via http/ssh :

http://cdsarc.u-strasbg.fr

GLU ALERT IN RED ARE CRTITICAL

(*) login as postgres : postgres unix account has no password. To login as postgres need to log before as root :
sudo su
su – postgres

Glu alert

Error

TODO

QueryCatDaemon.axel

server : axel

Big catalogue access


only for FX format

Note :It could be unavailable temporarily

Restarting the service :

test on server if started

systemctl status cds.catfile.service

stop the service

systemctl stop cds.catfile.service

start the service

systemctl start cds.catfile.service

QueryCatDaemon.cfa

server : CfA mirror

Big catalogue access in CfA

only for FX format

Connect to cfa and execute :

~/init.d/QueryCatDaemon.sh

QueryCatDaemon.ch

server : China mirror

Big catalogue access in china

only for FX format

Connect to China and execute :

~/init.d/QueryCatDaemon.sh

QueryCatDaemon.vizier

server : vizier

Big catalogues access (local access)

only for FX format

If service is down, VizieR switch automatically to QueryCatDaemon.axel

(this is just an optimisation)

test is started :

/etc/init.d/queryCatDaemon status

start/stop :

/etc/init.d/queryCatDaemon start/stop

Viz.monbigcat

server : axel

Big catalogues access

FX and Ochsenbein format

Note :It could be unavailable temporarily

Note : failures means that some big catalogues are unavailable (but not all)!

- FX format :

see QueryCatDaemon.axel

- Ochsenbein format :

test :

systemctl status aserver.service

start :

systemctl start aserver.service

Viz.mondb

server : vizdata

VizieR database access (prod)

Note : failure doesn't mean that VizieR is unreachable because VizieR

can switch automatically to a mirror using the file vizier:/home/cds/interface (section PCDSPOOL).

VizieR try the first line, if it fails, the second, …

If all access are unreachable, the crontab will redirect (15minutes) queries to a mirror.

If slow, it could be the result of too much queries : see the supervisor :

http://cdsarc.u-strasbg.fr/spv-bin/Vsystem.cgi

Else,

TODO :

resart postgres :

/etc/init.d/postgresql-pcds stop

/etc/init.d/postgresql-pcds start

If problem pesrists , it is certainly due to persistent temporary tables :

todo : (login as postgres(*))

list temporary tables per schema
~postgres/sql/clean_temporary_table.sh -l

number of temporry tables should not exceed 1000.

if exceed, start (for the schema impacted and guiven by previous command)

clean_temporary_table.sh -e schema

Health tests scripts:

~postgres/sql/stat_usage.sh

Test if connection is back

http://http://vizier.u-strasbg.fr/local/viz-bin/VizieR

if « db1_open(metaviz@PCDSPOOL) »

is in the debug part , then it is ok

Viz.mondb.local

server : cdsarc

VizieR local database

(used by diocumentalist for ingestion)

TODO : see action on Viz.mondb but applied on cdsarc server instead of vizdata

Viz.mondbpool

server : vizdata

VizieR database access (prod)

idem thal Viz.mndb

Idem that Viz.mondb

Viz.montap

server : tapvizier

TAPVizieR service

Restart tomcat service :

/etc/init.d/tomcat stop

/etc/init.d/tomcat start

Note : the service must be restarted if IRODS is restart !!!

VizU.cds

server : vizier

VizieR HTTP service

Test the database connection
(see Viz.mondb)

test the server access via http/ssh

http://vizier.u-strasbg.fr

VizU.adac

VizU.cadc

VizU.cfa

VizU.ch

VizU.hw

VizU.iucaa

VizU.saao

VizU.uk

server : (VizieR) mirrors

Vizier mirror

Contact the mirror administrator

VizieR.AssocData.ObsTAP

servers : cdsarc, vizdata

Assoc data service

Restart tomcat service (cdsarc)

/etc/init.d/tomcat7 stop

/etc/init.d/tomcat7 start

If problem persists :

Test the database : (vizdata)

test : (login as postgres (*))

psql -p 5433 saadavizier

restart :

/etc/init.d/postgresql-saada stop

/etc/init.d/postgresql-saada start

VizieR.AssocData.SIA

server : cdsarc, vizdata

Assoc data service (SIA)

See VizieR.AssocData.ObsTAP

getcat

server : cdsarc

ReadMe access

Too much connection

or HTTP not reachable

Note :It could be unavailable temporarily

Test the server access via http/ssh :

http://cdsarc.u-strasbg.fr

Topic revision: r3 - 2019-06-27 - GillesLandais
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback