Some XMatch Tests

Test1

Algo: corrélation simple à 1 arcsec
Machine: cdsxmatch
Nombre de sources:
- usnob1: 206 261 950 (x24 pour la taille en octets)
- twomass 115 556 628 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 203.032 s => 1 000 000 src.s-1
Création du kdtree: 39.779 s
Lecture des sources 2MASS+recherche des candidats USNOB1: 372.677 s => 310 000 src.s-1
Total: 10 minutes 17 seconds
Mémoire vive occupé 17Go.
Nombre de threads: 16 en parallèles sur les 2 quad cores.

Test2

Algo: corrélation simple à 1 arcsec
Machine: blazar
Nombre de sources:
- usnob1: 206 261 950 (x24 pour la taille en octets)
- twomass 115 556 628 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 280.8 s => 736 650 src.s-1
Création du kdtree: 228.651 s
Lecture des sources 2MASS+recherche des candidats USNOB1: 983.0 s => 117 555 src.s-1
Total: 24 minutes 55 seconds
Mémoire vive occupé 14Go.
Nombre de threads: 16 en parallèles sur les 4 processeurs de 6 coeurs.

Test3

Algo: corrélation avec erreurs de positions
Machine: cdsxmatch
Nombre de sources:
- usnob1: 206 261 950 (x24 pour la taille en octets)
- twomass 115 556 628 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 171.4 s => 1 200 000 src.s-1
Création du kdtree: 34.1 s
Lecture des sources 2MASS+recherche des candidats USNOB1: 995.6 s => 116 000 src.s-1
Total: 20 minutes 03 seconds
Mémoire vive occupé 17Go.
Nombre de threads: 16 en parallèles sur les 2 quad cores.

Test4

Algo: corrélation avec erreurs de positions
Machine: blazar
Nombre de sources:
- usnob1: 206 261 950 (x24 pour la taille en octets)
- twomass 115 556 628 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 235.6 s => 877 710 src.s-1
Création du kdtree: 208.1 s
Lecture des sources 2MASS+recherche des candidats USNOB1: 1413.2 s --> 23.5min => 81 700 src.s-1
Total: 31 minutes 00 seconds
Mémoire vive occupé 14Go.
Nombre de threads: 16 en parallèles sur les 4 processeurs de 6 coeurs.

Test5

Algo: corrélation simple à 1 arcsec
Machine: blazar
Nombre de sources:
- usnob1: 1 045 175 762 (x24 pour la taille en octets)
- twomass 470 992 970 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 2992.5 s --> 50 min => 349 265 src.s-1
Création du kdtree: 1049.7 s --> 17.5 min
Lecture des sources 2MASS+recherche des candidats USNOB1: 3817.6 s --> 63.6 min --> 1h => 123 300 src.s-1
Total: 131 minutes 12 seconds --> 2h11min
Mémoire vive occupé 75Go.
Nombre de threads: 24 en parallèles sur les 4 processeurs de 6 coeurs.

Test6

Algo: corrélation simple à 1 arcsec
Machine: blazar
Nombre de sources:
- usnob1: 1 045 175 762 (x24 pour la taille en octets)
- twomass 470 992 970 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 2666.9 s --> 44 min => 391 900 src.s-1
Création du kdtree: 1087.8 s --> 18.1 min
Lecture des sources 2MASS+recherche des candidats USNOB1: 4597.3 s --> 76.6 min --> 1h16 => 102 450 src.s-1
Total: 139 minutes 24 seconds --> 2h19min
Mémoire vive occupé 75Go.
Nombre de threads: 24 en parallèles sur les 4 processeurs de 6 coeurs.

Test7

Algo: corrélation simple à 1 arcsec
Machine: cdsxmatch
Nombre de sources:
- usnob1: 206 261 950 (x24 pour la taille en octets)
- twomass 115 556 628 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 175.7 s
Création du kdtree: 191.4 s
Lecture des sources 2MASS+recherche des candidats USNOB1: 2513.7 s
Total: 48 minutes 02 seconds
Mémoire vive occupé 17Go.
Nombre de threads: 1.

Test8

Algo: corrélation simple à 1 arcsec
Machine: blazar
Nombre de sources:
- usnob1: 206 261 950 (x24 pour la taille en octets)
- twomass 115 556 628 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 333.4
Création du kdtree: 558.4 s
Lecture des sources 2MASS+recherche des candidats USNOB1: 4524.6 s
Total: 90 minutes 19 seconds
Mémoire vive occupé 14Go.
Nombre de threads: 1.

Test9

Algo: corrélation simple à 1 arcsec
Machine: cdsxmatch
Nombre de sources:
- usnob1: 206 261 950 (x24 pour la taille en octets)
- twomass 115 556 628 (x24 pour la taille en octets)
Lecture et mise en mémoire de l'USNOB1: 173.7 s
Création du kdtree: 33.2 s
Lecture des sources 2MASS+recherche des candidats USNOB1: 257.1 s
Total: 07 minutes 45 seconds
Mémoire vive occupé 17Go.
Nombre de threads: 16.
REMARQUE: nouvelle gestion des Threads avec des queues de 10 000 sources par Thread.
NOMBRE DE CONTREPARTIES: environ 17 000 000.
AUTRES RESULTATS:
- 08 min 05 s avec des listes de 1 000 sources.
- 08 min 06 s avec des listes de 100 000 sources.
- 08 min 19 s avec des listes de 25 000 sources.
dans chaque cas, la différence se fait plutôt sur les temps variables de lecture disque et la création du kdtree, ma recherche de candidats et plus ou moins constante en fonction de la taille des listes.

TEST 10

Algo: corrélation avec erreur + écriture des résultats
Machine: cdsxmatch
Rq: nouvel algo parallele, pas encore les rebords du losange healpix
pineau@cdsxmatch:~/Work/CDSXmatchService/scripts$ ant exec_correlate -DcatA_name="2MASS" -DcatB_name="USNOB1" -Dfiles_dir="/home/pineau/data2/catalogues" -Dnside="1" -Dhealpix_idx="7"
exec_correlate:
[java] NbrThreads = 16
[java] Memory (B) = 19088801792
[java] Memory (Gb) = 17.77783203125
[java] Index "USNOB1.astrom.nside0.idx" loaded in 7 ms.
[java] Index "USNOB1.astrom.nside0.idx" loaded in 0 ms.
[java] [16/06/10 10:17:59] TRACE: Sources of catalalogue B retieve in 226.073 s. Nside=1, HealpixIdx=7, Nbr of srcs=206261958
[java] [16/06/10 10:18:36] TRACE: - KDTree computed in 37.384 s.
[java] [16/06/10 10:18:36] TRACE: Ready to correlate!
[java] Index "2MASS.astrom.nside0.idx" loaded in 10 ms.
[java] Thread 0 ; Nbr correlated = 7120000 ; Nbr cp = 3822536
[java] Thread 1 ; Nbr correlated = 7290000 ; Nbr cp = 3886903
[java] Thread 2 ; Nbr correlated = 7340000 ; Nbr cp = 3920436
[java] Thread 3 ; Nbr correlated = 7210000 ; Nbr cp = 3901046
[java] Thread 4 ; Nbr correlated = 7090000 ; Nbr cp = 3821745
[java] Thread 5 ; Nbr correlated = 7240000 ; Nbr cp = 3872656
[java] Thread 6 ; Nbr correlated = 7110000 ; Nbr cp = 3808347
[java] Thread 7 ; Nbr correlated = 7210000 ; Nbr cp = 3884911
[java] Thread 8 ; Nbr correlated = 7280000 ; Nbr cp = 3917133
[java] Thread 9 ; Nbr correlated = 7196677 ; Nbr cp = 3842943
[java] Thread 10 ; Nbr correlated = 7150000 ; Nbr cp = 3841634
[java] Thread 11 ; Nbr correlated = 7220000 ; Nbr cp = 3890064
[java] Thread 12 ; Nbr correlated = 7290000 ; Nbr cp = 3917041
[java] Thread 13 ; Nbr correlated = 7300000 ; Nbr cp = 3927522
[java] Thread 14 ; Nbr correlated = 7310000 ; Nbr cp = 3930752
[java] Thread 15 ; Nbr correlated = 7200000 ; Nbr cp = 3848662
[java] [16/06/10 10:32:41] TRACE: Correlation of the 115556677 2MASS src + write result performed in 844.266 s
BUILD SUCCESSFUL
Total time: 18 minutes 29 seconds

TEST 11

Idem test10 mais all sky, toujours sans rebords avec erreur: 73 minutes 10 seconds
cds=# select count(*) from twomass_vs_usnob1;
count

---

302241346
(1 row)

cds=# select count(distinct oid_2mass) from twomass_vs_usnob1;
count

---

283559130
(1 row)

cds=# select count(distinct oid_usnob1) from twomass_vs_usnob1;
count

---

300221755
(1 row)

TEST 12

idem test 11 avec 2MASS-SDSS DR7 = 22min
idem avec rebord = 25min!

Remarques

Dans tous les test, j'étais le seul connecté sur la machine distante,
et aucun autre programme n'était lancé, ni aucun autre accès disque.

Les tests 1 et 2 sont identiques mais sur des machines différentes.
Les tests 3 et 4 sont identiques mais sur des machines différentes.
Les tests 7 et 8 sont identiques mais sur des machines différentes.
Les tests 5 et 6 sont identiques et sur la même machine.

La simple lecture des données sur HDD + mise en mémoire virtuelle utilise les 16 threads de cdsxmatch et 19 threads de blazar (visualisation via htop).

Machine cdsxmatch

uname -r: 2.6.32-21-server
cat /etc/issue: Ubuntu 10.04 LTS \n \l
cat /etc/fstab: => ext4
dmesg | grep CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz stepping 05 | L1 I cache: 32K, L1 D cache: 32K | CPU: L2 cache: 256K | CPU: L3 cache: 8192K
dmesg | grep Memory: 24722732k/26214400k available
dmesg | grep Hz: hpet0: 4 comparators, 64-bit 14.318180 MHz counter
sudo dmidecode --type 17: Data Width: 64 bits | Size: 2048 MB | Speed: 1333 MHz (0.8 ns)
on peut aussi utiliser 'lshw -short', 'lshw -short -C memory', ...
ou encore 'cat /proc/cpuinfo'

Machine blazar

uname -r: 2.6.18-164.2.1.el5
cat /etc/issue: Scientific Linux SL release 5.4 (Boron) Kernel \r on an \m
cat /etc/fstab: => ext3
dmesg | grep CPU: Six-Core AMD Opteron(tm) Processor 8431 stepping 00 | CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) | CPU: L2 Cache: 512K (64 bytes/line)
dmesg | grep Memory: 115400068k/119537660k available
dmesg | grep Hz: hpet0: 3 32-bit timers, 25000000 Hz | 2411.114 MHz processor
sudo dmidecode --type 17: Data Width: 64 bits | Size: 4096 MB | Speed: 533 MHz

Premières conclusions

cdsxmatch est plus rapide que blazar: lecture des données sur le disque et mise en mémoire 37% plus rapide!

La vitesse des création des KDTree consiste essentiellement en des accès mémoire. Etant donnnées la lenteur de la création des KDTree sur blazar, je pense que la différence de performance viens essentiellement de la mémoire vive (ou de l'OS: scientific linux pour blazar).

Ce diagnostique est confirmé par le fait que dans les tests 3 et 4 le programme passe plus de temps en calcul pure qu'en accès mémoire. La différence entre les test 3 et 4 et moins sensible qu'entre les test 1 et 2.

MISE A JOUR: conclusion confirmée: la vitesse de la ram est plus de 2 fois plus lente sur blazar!!

Perspectives/Evolution

Grace à l'outil htop, j'ai pu constaté que chaque thread passe un certain pourcentage de temps (environ 6%) en sy:, i.e. en appel système (http://www.clochix.net/post/2009/02/10/htop-et-compagnie).
Cela pourrait venir de la parallèlisation, et j'envisage d'amèliorer l'algo en faisant des groupe de sources soumis à chaque Threads.

Effectivement, voir test9, on note une amélioration de 20% sur le temps total d'execution!

Database creation

Code executé sur ma machine perso: cds-dev-fxp (Ram 4Go,HDD 2To, Double core)

2010-06-10 11:18:15 MSG: Drop table twomass ...
2010-06-10 11:18:18 MSG: Create table twomass ...
2010-06-10 11:18:18 MSG: Copy file /home/pineau/data2/catalogues/twomass.csv into tab: twomass ...
2010-06-10 01:09:21 MSG: Create pkey index tab: twomass col: oid ...
2010-06-10 02:18:40 MSG: Add uniq constraint tab: twomass col: Cntr ...
2010-06-10 03:05:23 MSG: Add uniq constraint tab: twomass col: name_2MASS ...
2010-06-10 05:07:52 MSG: Create index tab: twomass col: RAdeg ...
2010-06-10 06:00:43 MSG: Create index tab: twomass col: DEdeg ...
2010-06-10 06:51:49 MSG: Vacuum and Analyse tab: twomass ...
2010-06-10 07:40:44 MSG: BEGIN
Buildfile: build.xml
exec_hp_idx:
[java] [10/06/10 19:40:47] TRACE: Start Job...
[java] [10/06/10 19:40:47] DEBUG: Connexion at host:localhost/cds user:pineau pwd:******
[java] [10/06/10 20:38:54] TRACE: Job done in 3487176 ms.
BUILD SUCCESSFUL
Total time: 58 minutes 8 seconds
2010-06-10 08:38:54 MSG: Drop table twomass_healpix_idx ...
2010-06-10 08:38:56 MSG: Create table twomass_healpix_idx ...
2010-06-10 08:38:57 MSG: Copy /home/pineau/data2/catalogues/twomass_healpix_idx.csv into twomass_healpix_idx ...
2010-06-10 09:01:46 MSG: Create pkey index tab: twomass_healpix_idx col: oid ...
2010-06-10 09:29:10 MSG: Create index tab: twomass_healpix_idx col: idx_1024 ...
2010-06-10 09:58:44 MSG: Vaccum and Analyse twomass_healpix_idx
2010-06-10 10:14:53 MSG: Drop table twomass_light_healpix ...
2010-06-10 10:14:53 MSG: Create table twomass_light_healpix
2010-06-11 12:37:43 MSG: Create pkey index tab: twomass_light_healpix col: oid ...
2010-06-11 01:30:14 MSG: Create index tab: twomass_light_healpix col: idx_1024 ...
2010-06-11 01:59:33 MSG: Vaccum and Analyse twomass_light_healpix
2010-06-11 02:25:52 MSG: END
2010-06-11 02:25:52 MSG: Create 2MASS healpix density nside=1024
2010-06-11 02:25:52 MSG: - Drop table twomass_healpix_dens_1024 ...
2010-06-11 02:25:54 MSG: - Create table twomass_healpix_dens_1024 ...
2010-06-11 02:36:42 MSG: - Create pkey index tab: twomass_healpix_dens_1024 col: idx_1024 ...
2010-06-11 02:37:06 MSG: Create 2MASS healpix density nside==512
2010-06-11 02:37:06 MSG: - Drop table twomass_healpix_dens_512 ...
2010-06-11 02:37:06 MSG: - Create table twomass_healpix_dens_512 ...
2010-06-11 02:37:32 MSG: - Create pkey index tab: twomass_healpix_dens_512 col: idx_512 ...
2010-06-11 02:37:34 MSG: Create 2MASS healpix density nside==256
2010-06-11 02:37:34 MSG: - Drop table twomass_healpix_dens_256 ...
2010-06-11 02:37:34 MSG: - Create table twomass_healpix_dens_256 ...
2010-06-11 02:37:42 MSG: - Create pkey index tab: twomass_healpix_dens_256 col: idx_256 ...
2010-06-11 02:37:42 MSG: Create 2MASS healpix density nside==128
2010-06-11 02:37:42 MSG: - Drop table twomass_healpix_dens_128 ...
2010-06-11 02:37:42 MSG: - Create table twomass_healpix_dens_128 ...
2010-06-11 02:37:44 MSG: - Create pkey index tab: twomass_healpix_dens_128 col: idx_128 ...
2010-06-11 02:37:44 MSG: Create 2MASS healpix density nside==64
2010-06-11 02:37:44 MSG: - Drop table twomass_healpix_dens_64 ...
2010-06-11 02:37:45 MSG: - Create table twomass_healpix_dens_64 ...
2010-06-11 02:37:45 MSG: - Create pkey index tab: twomass_healpix_dens_64 col: idx_64 ...
2010-06-11 02:37:45 MSG: Create 2MASS healpix density nside==32
2010-06-11 02:37:45 MSG: - Drop table twomass_healpix_dens_32 ...
2010-06-11 02:37:45 MSG: - Create table twomass_healpix_dens_32 ...
2010-06-11 02:37:46 MSG: - Create pkey index tab: twomass_healpix_dens_32 col: idx_32 ...
2010-06-11 02:37:46 MSG: Create 2MASS healpix density nside==16
2010-06-11 02:37:46 MSG: - Drop table twomass_healpix_dens_16 ...
2010-06-11 02:37:46 MSG: - Create table twomass_healpix_dens_16 ...
2010-06-11 02:37:46 MSG: - Create pkey index tab: twomass_healpix_dens_16 col: idx_16 ...
2010-06-11 02:37:46 MSG: Create 2MASS healpix density nside==8
2010-06-11 02:37:46 MSG: - Drop table twomass_healpix_dens_8 ...
2010-06-11 02:37:46 MSG: - Create table twomass_healpix_dens_8 ...
2010-06-11 02:37:47 MSG: - Create pkey index tab: twomass_healpix_dens_8 col: idx_8 ...
2010-06-11 02:37:47 MSG: Create 2MASS healpix density nside==4
2010-06-11 02:37:47 MSG: - Drop table twomass_healpix_dens_4 ...
2010-06-11 02:37:47 MSG: - Create table twomass_healpix_dens_4 ...
2010-06-11 02:37:47 MSG: - Create pkey index tab: twomass_healpix_dens_4 col: idx_4 ...
2010-06-11 02:37:47 MSG: Create 2MASS healpix density nside==2
2010-06-11 02:37:47 MSG: - Drop table twomass_healpix_dens_2 ...
2010-06-11 02:37:47 MSG: - Create table twomass_healpix_dens_2 ...
2010-06-11 02:37:47 MSG: - Create pkey index tab: twomass_healpix_dens_2 col: idx_2 ...
2010-06-11 02:37:47 MSG: Create 2MASS healpix density nside==0
2010-06-11 02:37:47 MSG: - Drop table twomass_healpix_dens_0 ...
2010-06-11 02:37:48 MSG: - Create table twomass_healpix_dens_0 ...
2010-06-11 02:37:48 MSG: - Create pkey index tab: twomass_healpix_dens_0 col: idx_O ...
2010-06-11 02:37:48 MSG: Drop table usnob1 ...
2010-06-11 02:37:48 MSG: Create table usnob1 ...
2010-06-11 02:37:48 MSG: Copy file /home/pineau/data2/catalogues/usnob1.csv into tab: usnob1 ...
2010-06-11 05:41:18 MSG: Create pkey index tab: usnob1 col: oid ...
2010-06-11 07:21:22 MSG: Add uniq constraint tab: usnob1 col: USNOB1 ...
2010-06-11 10:31:22 MSG: Create index tab: usnob1 col: RA ...
2010-06-11 12:12:15 MSG: Create index tab: usnob1 col: DE ...
2010-06-11 01:34:57 MSG: Vacuum and Analyse tab: usnob1 ...
2010-06-11 02:37:25 MSG: BEGIN
Buildfile: build.xml
exec_hp_idx:
[java] [11/06/10 14:37:27] TRACE: Start Job...
[java] [11/06/10 14:37:27] DEBUG: Connexion at host:localhost/cds user:pineau pwd:******
[java] [11/06/10 16:24:57] TRACE: Job done in 6449798 ms.
BUILD SUCCESSFUL
Total time: 107 minutes 31 seconds
2010-06-11 04:24:58 MSG: Drop table usnob1_healpix_idx ...
2010-06-11 04:25:02 MSG: Create table usnob1_healpix_idx ...
2010-06-11 04:25:02 MSG: Copy /home/pineau/data2/catalogues/usnob1_healpix_idx.csv into usnob1_healpix_idx ...
2010-06-11 05:09:32 MSG: Create pkey index tab: usnob1_healpix_idx col: oid ...
2010-06-11 06:40:41 MSG: Create index tab: usnob1_healpix_idx col: idx_1024 ...
2010-06-11 07:46:45 MSG: Vaccum and Analyse usnob1_healpix_idx
2010-06-11 08:11:57 MSG: Drop table usnob1_light_healpix ...
2010-06-11 08:11:58 MSG: Create table usnob1_light_healpix
2010-06-12 01:56:20 MSG: Create pkey index tab: usnob1_light_healpix col: oid ...
2010-06-12 03:50:36 MSG: Create index tab: usnob1_light_healpix col: idx_1024 ...
2010-06-12 04:52:39 MSG: Vaccum and Analyse usnob1_light_healpix
2010-06-12 05:39:08 MSG: END
2010-06-12 05:39:08 MSG: Create USNOB1 healpix density nside=1024
2010-06-12 05:39:08 MSG: - Drop table usnob1_healpix_dens_1024 ...
2010-06-12 05:39:11 MSG: - Create table usnob1_healpix_dens_1024 ...
2010-06-12 05:57:06 MSG: - Create pkey index tab: usnob1_healpix_dens_1024 col: idx_1024 ...
2010-06-12 05:57:32 MSG: Create USNOB1 healpix density nside==512
2010-06-12 05:57:32 MSG: - Drop table usnob1_healpix_dens_512 ...
2010-06-12 05:57:33 MSG: - Create table usnob1_healpix_dens_512 ...
2010-06-12 05:57:59 MSG: - Create pkey index tab: usnob1_healpix_dens_512 col: idx_512 ...
2010-06-12 05:58:02 MSG: Create USNOB1 healpix density nside==256
2010-06-12 05:58:02 MSG: - Drop table usnob1_healpix_dens_256 ...
2010-06-12 05:58:02 MSG: - Create table usnob1_healpix_dens_256 ...
2010-06-12 05:58:09 MSG: - Create pkey index tab: usnob1_healpix_dens_256 col: idx_256 ...
2010-06-12 05:58:10 MSG: Create USNOB1 healpix density nside==128
2010-06-12 05:58:10 MSG: - Drop table usnob1_healpix_dens_128 ...
2010-06-12 05:58:10 MSG: - Create table usnob1_healpix_dens_128 ...
2010-06-12 05:58:11 MSG: - Create pkey index tab: usnob1_healpix_dens_128 col: idx_128 ...
2010-06-12 05:58:11 MSG: Create USNOB1 healpix density nside==64
2010-06-12 05:58:11 MSG: - Drop table usnob1_healpix_dens_64 ...
2010-06-12 05:58:11 MSG: - Create table usnob1_healpix_dens_64 ...
2010-06-12 05:58:12 MSG: - Create pkey index tab: usnob1_healpix_dens_64 col: idx_64 ...
2010-06-12 05:58:12 MSG: Create USNOB1 healpix density nside==32
2010-06-12 05:58:12 MSG: - Drop table usnob1_healpix_dens_32 ...
2010-06-12 05:58:12 MSG: - Create table usnob1_healpix_dens_32 ...
2010-06-12 05:58:12 MSG: - Create pkey index tab: usnob1_healpix_dens_32 col: idx_32 ...
2010-06-12 05:58:12 MSG: Create USNOB1 healpix density nside==16
2010-06-12 05:58:12 MSG: - Drop table usnob1_healpix_dens_16 ...
2010-06-12 05:58:12 MSG: - Create table usnob1_healpix_dens_16 ...
2010-06-12 05:58:13 MSG: - Create pkey index tab: usnob1_healpix_dens_16 col: idx_16 ...
2010-06-12 05:58:13 MSG: Create USNOB1 healpix density nside==8
2010-06-12 05:58:13 MSG: - Drop table usnob1_healpix_dens_8 ...
2010-06-12 05:58:13 MSG: - Create table usnob1_healpix_dens_8 ...
2010-06-12 05:58:13 MSG: - Create pkey index tab: usnob1_healpix_dens_8 col: idx_8 ...
2010-06-12 05:58:13 MSG: Create USNOB1 healpix density nside==4
2010-06-12 05:58:13 MSG: - Drop table usnob1_healpix_dens_4 ...
2010-06-12 05:58:13 MSG: - Create table usnob1_healpix_dens_4 ...
2010-06-12 05:58:13 MSG: - Create pkey index tab: usnob1_healpix_dens_4 col: idx_4 ...
2010-06-12 05:58:14 MSG: Create USNOB1 healpix density nside==2
2010-06-12 05:58:14 MSG: - Drop table usnob1_healpix_dens_2 ...
2010-06-12 05:58:14 MSG: - Create table usnob1_healpix_dens_2 ...
2010-06-12 05:58:14 MSG: - Create pkey index tab: usnob1_healpix_dens_2 col: idx_2 ...
2010-06-12 05:58:14 MSG: Create USNOB1 healpix density nside==0
2010-06-12 05:58:14 MSG: - Drop table usnob1_healpix_dens_0 ...
2010-06-12 05:58:14 MSG: - Create table usnob1_healpix_dens_0 ...
2010-06-12 05:58:14 MSG: - Create pkey index tab: usnob1_healpix_dens_0 col: idx_O ...
2010-06-12 05:58:14 MSG: Drop table sdss_dr7 ...
2010-06-12 05:58:15 MSG: Create table sdss_dr7 ...
2010-06-12 05:58:15 ERROR: Query failed! "COPY sdss_dr7 (mode,Quality_phot,cl,SDSS7_Jname,objID,specID,RA,DE,e_RA,e_Dec,ccPos,Obs_Date,Q,zsp,e_zsp,u,u_err,g,g_err,r,r_err,i,i_err,z,z_err) FROM '/home/pineau/data2/catalogues/sdss_dr7.csv' WITH NULL AS 'NULL' CSV HEADER;"
tail -2 /home/pineau/data2/catalogues/logs/default.log
ERROR: invalid input syntax for integer: "1 b 3 J003725.83-235333.4 587747072206635690 0 009.357633 -23.892635 0.209 0.324 -0.64 1994.9525 2 NULL NULL 24.011 1.999 22.940 0.336 21.779 0.154 22.425 0.460 21.418 0.648" CONTEXT: COPY sdss_dr7, line 2, column mode: "1 b 3 J003725.83-235333.4 587747072206635690 0 009.357633 -23.892635 0.209 0.324 -0.64 1994.9525 2 N..."
2010-06-14 04:43:22 MSG: Create the file /home/pineau/data2/catalogues/sdss_dr7.csv ...
2010-06-14 08:34:36 MSG: Drop table sdss_dr7 ...
2010-06-14 08:34:39 MSG: Create table sdss_dr7 ...
2010-06-14 08:34:39 MSG: Copy file /home/pineau/data2/catalogues/sdss_dr7.csv into tab: sdss_dr7 ...
2010-06-14 10:18:23 MSG: Create pkey index tab: sdss_dr7 col: oid ...
2010-06-14 11:14:29 MSG: Create index tab: sdss_dr7 col: objID ...
2010-06-14 11:51:09 MSG: Create index tab: sdss_dr7 col: SDSS7_jname ...
2010-06-15 01:27:37 MSG: Create index tab: sdss_dr7 col: RA ...
2010-06-15 02:08:01 MSG: Create index tab: sdss_dr7 col: DE ...
2010-06-15 02:45:40 MSG: Vacuum and Analyse tab: sdss_dr7 ...
2010-06-15 03:29:18 MSG: BEGIN
Buildfile: build.xml
exec_hp_idx:
[java] [15/06/10 03:29:21] TRACE: Start Job...
[java] [15/06/10 03:29:21] DEBUG: Connexion at host:localhost/cds user:pineau pwd:******
[java] [15/06/10 04:17:31] TRACE: Job done in 2889606 ms.
BUILD SUCCESSFUL
Total time: 48 minutes 10 seconds
2010-06-15 04:17:31 MSG: Drop table sdss_dr7_healpix_idx ...
2010-06-15 04:17:32 MSG: Create table sdss_dr7_healpix_idx ...
2010-06-15 04:17:32 MSG: Copy /home/pineau/data2/catalogues/sdss_dr7_healpix_idx.csv into sdss_dr7_healpix_idx ...
2010-06-15 04:34:00 MSG: Create pkey index tab: sdss_dr7_healpix_idx col: oid ...
2010-06-15 04:55:44 MSG: Create index tab: sdss_dr7_healpix_idx col: idx_1024 ...
2010-06-15 05:18:18 MSG: Vaccum and Analyse sdss_dr7_healpix_idx
2010-06-15 05:30:06 MSG: Drop table sdss_dr7_light_healpix ...
2010-06-15 05:30:07 MSG: Create table sdss_dr7_light_healpix
2010-06-15 07:15:01 MSG: Create pkey index tab: sdss_dr7_light_healpix col: oid ...
2010-06-15 07:53:29 MSG: Create index tab: sdss_dr7_light_healpix col: idx_1024 ...
2010-06-15 08:14:17 MSG: Vaccum and Analyse sdss_dr7_light_healpix
2010-06-15 08:33:01 MSG: END
2010-06-15 08:33:01 MSG: Create SDSS_DR7 healpix density nside=1024
2010-06-15 08:33:01 MSG: - Drop table sdss_dr7_healpix_dens_1024 ...
2010-06-15 08:33:03 MSG: - Create table sdss_dr7_healpix_dens_1024 ...
2010-06-15 08:40:57 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_1024 col: idx_1024 ...
2010-06-15 08:41:00 MSG: Create SDSS_DR7 healpix density nside==512
2010-06-15 08:41:00 MSG: - Drop table sdss_dr7_healpix_dens_512 ...
2010-06-15 08:41:00 MSG: - Create table sdss_dr7_healpix_dens_512 ...
2010-06-15 08:41:07 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_512 col: idx_512 ...
2010-06-15 08:41:08 MSG: Create SDSS_DR7 healpix density nside==256
2010-06-15 08:41:08 MSG: - Drop table sdss_dr7_healpix_dens_256 ...
2010-06-15 08:41:08 MSG: - Create table sdss_dr7_healpix_dens_256 ...
2010-06-15 08:41:09 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_256 col: idx_256 ...
2010-06-15 08:41:09 MSG: Create SDSS_DR7 healpix density nside==128
2010-06-15 08:41:09 MSG: - Drop table sdss_dr7_healpix_dens_128 ...
2010-06-15 08:41:09 MSG: - Create table sdss_dr7_healpix_dens_128 ...
2010-06-15 08:41:09 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_128 col: idx_128 ...
2010-06-15 08:41:10 MSG: Create SDSS_DR7 healpix density nside==64
2010-06-15 08:41:10 MSG: - Drop table sdss_dr7_healpix_dens_64 ...
2010-06-15 08:41:10 MSG: - Create table sdss_dr7_healpix_dens_64 ...
2010-06-15 08:41:10 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_64 col: idx_64 ...
2010-06-15 08:41:10 MSG: Create SDSS_DR7 healpix density nside==32
2010-06-15 08:41:10 MSG: - Drop table sdss_dr7_healpix_dens_32 ...
2010-06-15 08:41:10 MSG: - Create table sdss_dr7_healpix_dens_32 ...
2010-06-15 08:41:10 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_32 col: idx_32 ...
2010-06-15 08:41:11 MSG: Create SDSS_DR7 healpix density nside==16
2010-06-15 08:41:11 MSG: - Drop table sdss_dr7_healpix_dens_16 ...
2010-06-15 08:41:11 MSG: - Create table sdss_dr7_healpix_dens_16 ...
2010-06-15 08:41:11 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_16 col: idx_16 ...
2010-06-15 08:41:11 MSG: Create SDSS_DR7 healpix density nside==8
2010-06-15 08:41:11 MSG: - Drop table sdss_dr7_healpix_dens_8 ...
2010-06-15 08:41:11 MSG: - Create table sdss_dr7_healpix_dens_8 ...
2010-06-15 08:41:11 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_8 col: idx_8 ...
2010-06-15 08:41:11 MSG: Create SDSS_DR7 healpix density nside==4
2010-06-15 08:41:11 MSG: - Drop table sdss_dr7_healpix_dens_4 ...
2010-06-15 08:41:11 MSG: - Create table sdss_dr7_healpix_dens_4 ...
2010-06-15 08:41:12 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_4 col: idx_4 ...
2010-06-15 08:41:12 MSG: Create SDSS_DR7 healpix density nside==2
2010-06-15 08:41:12 MSG: - Drop table sdss_dr7_healpix_dens_2 ...
2010-06-15 08:41:12 MSG: - Create table sdss_dr7_healpix_dens_2 ...
2010-06-15 08:41:12 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_2 col: idx_2 ...
2010-06-15 08:41:12 MSG: Create SDSS_DR7 healpix density nside==0
2010-06-15 08:41:12 MSG: - Drop table sdss_dr7_healpix_dens_0 ...
2010-06-15 08:41:12 MSG: - Create table sdss_dr7_healpix_dens_0 ...
2010-06-15 08:41:12 MSG: - Create pkey index tab: sdss_dr7_healpix_dens_0 col: idx_O ...
./exec.next.bash
2010-06-15 09:47:38 MSG: Compute and create healpix ordered_bin_files for cat 2MASS...
Buildfile: build.xml
exec_healpixp_ordered_binfiles:
[java] Args: [-c, localhost, cds, pineau, , SELECT idx_1024, oid, radeg, dedeg, errMaj, errMin, ePA FROM twomass_light_healpix ORDER BY idx_1024 ASC, ELLIPSE_MAJ_MIN_ANG, /home/pineau/data2/catalogues, 2MASS]
[java] [15/06/10 09:47:39] TRACE: Start Job...
[java] [15/06/10 09:47:39] DEBUG: Connexion at host:localhost/cds user:pineau pwd:******
[java] [15/06/10 11:15:01] TRACE: Job done in 5241871 ms.
BUILD SUCCESSFUL
Total time: 87 minutes 25 seconds
2010-06-15 11:15:07 MSG: Done!
2010-06-15 11:15:07 MSG: Compute and create healpix ordered_bin_files for cat USNOB1...
Buildfile: build.xml
exec_healpixp_ordered_binfiles:
[java] Args: [-c, localhost, cds, pineau, , SELECT idx_1024, oid, ra, de, ra_err, de_err FROM usnob1_light_healpix ORDER BY idx_1024 ASC, ELLIPSE_RA_DEC, /home/pineau/data2/catalogues, USNOB1]
[java] [15/06/10 11:15:09] TRACE: Start Job...
[java] [15/06/10 11:15:10] DEBUG: Connexion at host:localhost/cds user:pineau pwd:******
[java] [15/06/10 13:25:02] TRACE: Job done in 7792398 ms.
BUILD SUCCESSFUL
Total time: 129 minutes 54 seconds
2010-06-15 01:25:06 MSG: Done!
2010-06-15 01:25:06 MSG: Compute and create healpix ordered_bin_files for cat SDSS_DR7...
Buildfile: build.xml
exec_healpixp_ordered_binfiles:
[java] Args: [-c, localhost, cds, pineau, , SELECT idx_1024, oid, ra, de, e_ra, e_dec, ccpos FROM sdss_dr7_light_healpix ORDER BY idx_1024 ASC, ELLIPSE_VAR_VAR_CORR, /home/pineau/data2/catalogues, SDSS_DR7]
[java] [15/06/10 13:25:08] TRACE: Start Job...
[java] [15/06/10 13:25:08] DEBUG: Connexion at host:localhost/cds user:pineau pwd:******
[java] [15/06/10 14:20:09] TRACE: Job done in 3300935 ms.
BUILD SUCCESSFUL
Total time: 55 minutes 2 seconds
2010-06-15 02:20:09 MSG: Done!

Healpix sort and other tools

Sorting of all 2MASS data from a 11Gb file (healpix_idx_1024,ra,de,maf,min,theta) done in 23min on cdsxmatch
Transforming the binary file in csv of 20Gb: 42min
Transforming the csv in binary file of 11Gb: 17min
(all tests made on cdsxmatch with buffers of 1Gb)

Performances can be increased by multithreading the code
Problem: to keep the order in input and in output.
Solution: use 2 arrays: buffer in --> array in --> multithread with index parameters and increment an int to know when the arry is full --> array out --> buffer out

--++ Test SDSS DR7x2MASS on cdsxmatch
Simple Xcorr at 1arcse done in... ~15min !!
2010-09-10 02:13:50 : BEGIN
NbrThreads = 16
Memory (B) = 19088801792
Memory (Gb) = 17.77783203125
Index "2MASS.nside1.idx" loaded in 10 ms.
Index "SDSS7.nside512.idx" loaded in 250 ms.
[10/09/10 14:13:52] TRACE: Number of source in each catalogue evaluated in 1011 ms. Nside=1, HealpixIdx=0, Nbr of srcs B=32934937 Nbr of srcs A (with borders)=19373704
[10/09/10 14:14:13] TRACE: Sources of catalalogue B retieve in 21.176 s. Nside=1, HealpixIdx=0, Nbr of srcs=32934937
[10/09/10 14:14:21] TRACE: - KDTree computed in 7.731 s. Err max = 1.0 arcsec
[10/09/10 14:14:21] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 1220000 ; Nbr cp = 163480
Thread 1 ; Nbr correlated = 1170000 ; Nbr cp = 158794
Thread 2 ; Nbr correlated = 1210000 ; Nbr cp = 167817
Thread 3 ; Nbr correlated = 1220000 ; Nbr cp = 166839
Thread 4 ; Nbr correlated = 1190000 ; Nbr cp = 159214
Thread 5 ; Nbr correlated = 1213704 ; Nbr cp = 167601
Thread 6 ; Nbr correlated = 1220000 ; Nbr cp = 165571
Thread 7 ; Nbr correlated = 1210000 ; Nbr cp = 163911
Thread 8 ; Nbr correlated = 1190000 ; Nbr cp = 165314
Thread 9 ; Nbr correlated = 1220000 ; Nbr cp = 165014
Thread 10 ; Nbr correlated = 1210000 ; Nbr cp = 164246
Thread 11 ; Nbr correlated = 1230000 ; Nbr cp = 170106
Thread 12 ; Nbr correlated = 1230000 ; Nbr cp = 165969
Thread 13 ; Nbr correlated = 1210000 ; Nbr cp = 163423
Thread 14 ; Nbr correlated = 1220000 ; Nbr cp = 164449
Thread 15 ; Nbr correlated = 1210000 ; Nbr cp = 165703
[10/09/10 14:14:59] TRACE: Correlation of the 19373704 SDSS7 src + write result performed in 38.741 s
[10/09/10 14:15:00] TRACE: Number of source in each catalogue evaluated in 903 ms. Nside=1, HealpixIdx=1, Nbr of srcs B=9748407 Nbr of srcs A (with borders)=67676281
[10/09/10 14:15:03] TRACE: Sources of catalalogue B retieve in 2.342 s. Nside=1, HealpixIdx=1, Nbr of srcs=9748407
[10/09/10 14:15:04] TRACE: - KDTree computed in 1.107 s. Err max = 1.0 arcsec
[10/09/10 14:15:04] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 4250000 ; Nbr cp = 333352
Thread 1 ; Nbr correlated = 4270000 ; Nbr cp = 333659
Thread 2 ; Nbr correlated = 4260000 ; Nbr cp = 334199
Thread 3 ; Nbr correlated = 4260000 ; Nbr cp = 339130
Thread 4 ; Nbr correlated = 4050000 ; Nbr cp = 318470
Thread 5 ; Nbr correlated = 4300000 ; Nbr cp = 336223
Thread 6 ; Nbr correlated = 4110000 ; Nbr cp = 325946
Thread 7 ; Nbr correlated = 4270000 ; Nbr cp = 336781
Thread 8 ; Nbr correlated = 4320000 ; Nbr cp = 335942
Thread 9 ; Nbr correlated = 4316281 ; Nbr cp = 337492
Thread 10 ; Nbr correlated = 4260000 ; Nbr cp = 335553
Thread 11 ; Nbr correlated = 4070000 ; Nbr cp = 322471
Thread 12 ; Nbr correlated = 4100000 ; Nbr cp = 321855
Thread 13 ; Nbr correlated = 4260000 ; Nbr cp = 335032
Thread 14 ; Nbr correlated = 4290000 ; Nbr cp = 333589
Thread 15 ; Nbr correlated = 4290000 ; Nbr cp = 338402
[10/09/10 14:16:41] TRACE: Correlation of the 67676281 SDSS7 src + write result performed in 97.179 s
[10/09/10 14:16:42] TRACE: Number of source in each catalogue evaluated in 880 ms. Nside=1, HealpixIdx=2, Nbr of srcs B=7064624 Nbr of srcs A (with borders)=80948878
[10/09/10 14:16:42] TRACE: Sources of catalalogue B retieve in 0.458 s. Nside=1, HealpixIdx=2, Nbr of srcs=7064624
[10/09/10 14:16:43] TRACE: - KDTree computed in 0.632 s. Err max = 1.0 arcsec
[10/09/10 14:16:43] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 5130000 ; Nbr cp = 329484
Thread 1 ; Nbr correlated = 5070000 ; Nbr cp = 327468
Thread 2 ; Nbr correlated = 5080000 ; Nbr cp = 327477
Thread 3 ; Nbr correlated = 5160000 ; Nbr cp = 332594
Thread 4 ; Nbr correlated = 5020000 ; Nbr cp = 322963
Thread 5 ; Nbr correlated = 5000000 ; Nbr cp = 321236
Thread 6 ; Nbr correlated = 5120000 ; Nbr cp = 330245
Thread 7 ; Nbr correlated = 5090000 ; Nbr cp = 331466
Thread 8 ; Nbr correlated = 4980000 ; Nbr cp = 320720
Thread 9 ; Nbr correlated = 5050000 ; Nbr cp = 325385
Thread 10 ; Nbr correlated = 5090000 ; Nbr cp = 329258
Thread 11 ; Nbr correlated = 4930000 ; Nbr cp = 319573
Thread 12 ; Nbr correlated = 5118878 ; Nbr cp = 330080
Thread 13 ; Nbr correlated = 5020000 ; Nbr cp = 326544
Thread 14 ; Nbr correlated = 5130000 ; Nbr cp = 332618
Thread 15 ; Nbr correlated = 4960000 ; Nbr cp = 320476
[10/09/10 14:18:37] TRACE: Correlation of the 80948878 SDSS7 src + write result performed in 113.826 s
[10/09/10 14:18:38] TRACE: Number of source in each catalogue evaluated in 857 ms. Nside=1, HealpixIdx=3, Nbr of srcs B=65466157 Nbr of srcs A (with borders)=40803588
[10/09/10 14:18:45] TRACE: Sources of catalalogue B retieve in 7.799 s. Nside=1, HealpixIdx=3, Nbr of srcs=65466157
[10/09/10 14:18:53] TRACE: - KDTree computed in 7.959 s. Err max = 1.0 arcsec
[10/09/10 14:18:53] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 2470000 ; Nbr cp = 461566
Thread 1 ; Nbr correlated = 2590000 ; Nbr cp = 479122
Thread 2 ; Nbr correlated = 2570000 ; Nbr cp = 473566
Thread 3 ; Nbr correlated = 2560000 ; Nbr cp = 477264
Thread 4 ; Nbr correlated = 2590000 ; Nbr cp = 476745
Thread 5 ; Nbr correlated = 2570000 ; Nbr cp = 477969
Thread 6 ; Nbr correlated = 2570000 ; Nbr cp = 478864
Thread 7 ; Nbr correlated = 2480000 ; Nbr cp = 458759
Thread 8 ; Nbr correlated = 2580000 ; Nbr cp = 475901
Thread 9 ; Nbr correlated = 2580000 ; Nbr cp = 486113
Thread 10 ; Nbr correlated = 2480000 ; Nbr cp = 459917
Thread 11 ; Nbr correlated = 2580000 ; Nbr cp = 482786
Thread 12 ; Nbr correlated = 2570000 ; Nbr cp = 471854
Thread 13 ; Nbr correlated = 2580000 ; Nbr cp = 479145
Thread 14 ; Nbr correlated = 2560000 ; Nbr cp = 479219
Thread 15 ; Nbr correlated = 2473588 ; Nbr cp = 465235
[10/09/10 14:21:08] TRACE: Correlation of the 40803588 SDSS7 src + write result performed in 135.157 s
[10/09/10 14:21:09] TRACE: Number of source in each catalogue evaluated in 868 ms. Nside=1, HealpixIdx=4, Nbr of srcs B=7115957 Nbr of srcs A (with borders)=25060327
[10/09/10 14:21:10] TRACE: Sources of catalalogue B retieve in 0.998 s. Nside=1, HealpixIdx=4, Nbr of srcs=7115957
[10/09/10 14:21:11] TRACE: - KDTree computed in 0.683 s. Err max = 1.0 arcsec
[10/09/10 14:21:11] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 1610000 ; Nbr cp = 127565
Thread 1 ; Nbr correlated = 1580000 ; Nbr cp = 127081
Thread 2 ; Nbr correlated = 1590000 ; Nbr cp = 125397
Thread 3 ; Nbr correlated = 1590000 ; Nbr cp = 127373
Thread 4 ; Nbr correlated = 1600000 ; Nbr cp = 127312
Thread 5 ; Nbr correlated = 1550000 ; Nbr cp = 124862
Thread 6 ; Nbr correlated = 1580000 ; Nbr cp = 127004
Thread 7 ; Nbr correlated = 1530327 ; Nbr cp = 123445
Thread 8 ; Nbr correlated = 1580000 ; Nbr cp = 126548
Thread 9 ; Nbr correlated = 1550000 ; Nbr cp = 124346
Thread 10 ; Nbr correlated = 1540000 ; Nbr cp = 123239
Thread 11 ; Nbr correlated = 1570000 ; Nbr cp = 124007
Thread 12 ; Nbr correlated = 1530000 ; Nbr cp = 121372
Thread 13 ; Nbr correlated = 1540000 ; Nbr cp = 125466
Thread 14 ; Nbr correlated = 1530000 ; Nbr cp = 122297
Thread 15 ; Nbr correlated = 1590000 ; Nbr cp = 128115
[10/09/10 14:21:50] TRACE: Correlation of the 25060327 SDSS7 src + write result performed in 38.619 s
[10/09/10 14:21:50] TRACE: Number of source in each catalogue evaluated in 858 ms. Nside=1, HealpixIdx=5, Nbr of srcs B=33375789 Nbr of srcs A (with borders)=30283944
[10/09/10 14:21:56] TRACE: Sources of catalalogue B retieve in 5.11 s. Nside=1, HealpixIdx=5, Nbr of srcs=33375789
[10/09/10 14:21:59] TRACE: - KDTree computed in 3.62 s. Err max = 1.0 arcsec
[10/09/10 14:21:59] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 1820000 ; Nbr cp = 328821
Thread 1 ; Nbr correlated = 1920000 ; Nbr cp = 344619
Thread 2 ; Nbr correlated = 1833944 ; Nbr cp = 328188
Thread 3 ; Nbr correlated = 1930000 ; Nbr cp = 350633
Thread 4 ; Nbr correlated = 1910000 ; Nbr cp = 345577
Thread 5 ; Nbr correlated = 1900000 ; Nbr cp = 345408
Thread 6 ; Nbr correlated = 1920000 ; Nbr cp = 348483
Thread 7 ; Nbr correlated = 1920000 ; Nbr cp = 348422
Thread 8 ; Nbr correlated = 1890000 ; Nbr cp = 339542
Thread 9 ; Nbr correlated = 1850000 ; Nbr cp = 330029
Thread 10 ; Nbr correlated = 1920000 ; Nbr cp = 346493
Thread 11 ; Nbr correlated = 1940000 ; Nbr cp = 350124
Thread 12 ; Nbr correlated = 1900000 ; Nbr cp = 344721
Thread 13 ; Nbr correlated = 1910000 ; Nbr cp = 344691
Thread 14 ; Nbr correlated = 1900000 ; Nbr cp = 345892
Thread 15 ; Nbr correlated = 1820000 ; Nbr cp = 329664
[10/09/10 14:22:51] TRACE: Correlation of the 30283944 SDSS7 src + write result performed in 51.635 s
[10/09/10 14:22:52] TRACE: Number of source in each catalogue evaluated in 853 ms. Nside=1, HealpixIdx=6, Nbr of srcs B=6734054 Nbr of srcs A (with borders)=55138113
[10/09/10 14:22:53] TRACE: Sources of catalalogue B retieve in 0.972 s. Nside=1, HealpixIdx=6, Nbr of srcs=6734054
[10/09/10 14:22:53] TRACE: - KDTree computed in 0.588 s. Err max = 1.0 arcsec
[10/09/10 14:22:53] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 3318113 ; Nbr cp = 186364
Thread 1 ; Nbr correlated = 3450000 ; Nbr cp = 191390
Thread 2 ; Nbr correlated = 3480000 ; Nbr cp = 193863
Thread 3 ; Nbr correlated = 3470000 ; Nbr cp = 192866
Thread 4 ; Nbr correlated = 3520000 ; Nbr cp = 198141
Thread 5 ; Nbr correlated = 3440000 ; Nbr cp = 193659
Thread 6 ; Nbr correlated = 3530000 ; Nbr cp = 197547
Thread 7 ; Nbr correlated = 3500000 ; Nbr cp = 195209
Thread 8 ; Nbr correlated = 3250000 ; Nbr cp = 181237
Thread 9 ; Nbr correlated = 3520000 ; Nbr cp = 194811
Thread 10 ; Nbr correlated = 3370000 ; Nbr cp = 188498
Thread 11 ; Nbr correlated = 3360000 ; Nbr cp = 188824
Thread 12 ; Nbr correlated = 3490000 ; Nbr cp = 193799
Thread 13 ; Nbr correlated = 3510000 ; Nbr cp = 196185
Thread 14 ; Nbr correlated = 3440000 ; Nbr cp = 192665
Thread 15 ; Nbr correlated = 3490000 ; Nbr cp = 196315
[10/09/10 14:24:08] TRACE: Correlation of the 55138113 SDSS7 src + write result performed in 74.985 s
[10/09/10 14:24:09] TRACE: Number of source in each catalogue evaluated in 870 ms. Nside=1, HealpixIdx=7, Nbr of srcs B=115559646 Nbr of srcs A (with borders)=32485946
[10/09/10 14:24:43] TRACE: Sources of catalalogue B retieve in 34.068 s. Nside=1, HealpixIdx=7, Nbr of srcs=115559646
[10/09/10 14:24:59] TRACE: - KDTree computed in 16.11 s. Err max = 1.0 arcsec
[10/09/10 14:24:59] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 1990000 ; Nbr cp = 306839
Thread 1 ; Nbr correlated = 2060000 ; Nbr cp = 323753
Thread 2 ; Nbr correlated = 2060000 ; Nbr cp = 325440
Thread 3 ; Nbr correlated = 2060000 ; Nbr cp = 324662
Thread 4 ; Nbr correlated = 2060000 ; Nbr cp = 321658
Thread 5 ; Nbr correlated = 2060000 ; Nbr cp = 319418
Thread 6 ; Nbr correlated = 2050000 ; Nbr cp = 315874
Thread 7 ; Nbr correlated = 2030000 ; Nbr cp = 316500
Thread 8 ; Nbr correlated = 1960000 ; Nbr cp = 301688
Thread 9 ; Nbr correlated = 2050000 ; Nbr cp = 316648
Thread 10 ; Nbr correlated = 1945946 ; Nbr cp = 304149
Thread 11 ; Nbr correlated = 1960000 ; Nbr cp = 309321
Thread 12 ; Nbr correlated = 2040000 ; Nbr cp = 323324
Thread 13 ; Nbr correlated = 2040000 ; Nbr cp = 312877
Thread 14 ; Nbr correlated = 2060000 ; Nbr cp = 324113
Thread 15 ; Nbr correlated = 2060000 ; Nbr cp = 321066
[10/09/10 14:26:08] TRACE: Correlation of the 32485946 SDSS7 src + write result performed in 68.453 s
[10/09/10 14:26:09] TRACE: Number of source in each catalogue evaluated in 866 ms. Nside=1, HealpixIdx=8, Nbr of srcs B=8432514 Nbr of srcs A (with borders)=2305243
[10/09/10 14:26:11] TRACE: Sources of catalalogue B retieve in 2.679 s. Nside=1, HealpixIdx=8, Nbr of srcs=8432514
[10/09/10 14:26:12] TRACE: - KDTree computed in 0.874 s. Err max = 1.0 arcsec
[10/09/10 14:26:12] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 150000 ; Nbr cp = 9466
Thread 1 ; Nbr correlated = 150000 ; Nbr cp = 9286
Thread 2 ; Nbr correlated = 150000 ; Nbr cp = 9576
Thread 3 ; Nbr correlated = 155243 ; Nbr cp = 9821
Thread 4 ; Nbr correlated = 150000 ; Nbr cp = 8903
Thread 5 ; Nbr correlated = 150000 ; Nbr cp = 9291
Thread 6 ; Nbr correlated = 140000 ; Nbr cp = 8814
Thread 7 ; Nbr correlated = 140000 ; Nbr cp = 8586
Thread 8 ; Nbr correlated = 130000 ; Nbr cp = 8630
Thread 9 ; Nbr correlated = 150000 ; Nbr cp = 10044
Thread 10 ; Nbr correlated = 130000 ; Nbr cp = 8817
Thread 11 ; Nbr correlated = 140000 ; Nbr cp = 9543
Thread 12 ; Nbr correlated = 150000 ; Nbr cp = 10019
Thread 13 ; Nbr correlated = 150000 ; Nbr cp = 10166
Thread 14 ; Nbr correlated = 130000 ; Nbr cp = 8679
Thread 15 ; Nbr correlated = 140000 ; Nbr cp = 9086
[10/09/10 14:26:17] TRACE: Correlation of the 2305243 SDSS7 src + write result performed in 5.006 s
[10/09/10 14:26:18] TRACE: Number of source in each catalogue evaluated in 868 ms. Nside=1, HealpixIdx=9, Nbr of srcs B=57941190 Nbr of srcs A (with borders)=302723
[10/09/10 14:26:38] TRACE: Sources of catalalogue B retieve in 19.554 s. Nside=1, HealpixIdx=9, Nbr of srcs=57941190
[10/09/10 14:26:45] TRACE: - KDTree computed in 7.478 s. Err max = 1.0 arcsec
[10/09/10 14:26:45] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 22723 ; Nbr cp = 1454
Thread 1 ; Nbr correlated = 20000 ; Nbr cp = 1147
Thread 2 ; Nbr correlated = 20000 ; Nbr cp = 1324
Thread 3 ; Nbr correlated = 20000 ; Nbr cp = 1360
Thread 4 ; Nbr correlated = 20000 ; Nbr cp = 1645
Thread 5 ; Nbr correlated = 20000 ; Nbr cp = 2136
Thread 6 ; Nbr correlated = 20000 ; Nbr cp = 2384
Thread 7 ; Nbr correlated = 20000 ; Nbr cp = 2517
Thread 8 ; Nbr correlated = 20000 ; Nbr cp = 2486
Thread 9 ; Nbr correlated = 20000 ; Nbr cp = 2237
Thread 10 ; Nbr correlated = 20000 ; Nbr cp = 2225
Thread 11 ; Nbr correlated = 10000 ; Nbr cp = 1196
Thread 12 ; Nbr correlated = 20000 ; Nbr cp = 2481
Thread 13 ; Nbr correlated = 20000 ; Nbr cp = 2492
Thread 14 ; Nbr correlated = 20000 ; Nbr cp = 2555
Thread 15 ; Nbr correlated = 10000 ; Nbr cp = 1271
[10/09/10 14:26:47] TRACE: Correlation of the 302723 SDSS7 src + write result performed in 1.935 s
[10/09/10 14:26:48] TRACE: Number of source in each catalogue evaluated in 882 ms. Nside=1, HealpixIdx=10, Nbr of srcs B=103721508 Nbr of srcs A (with borders)=1307620
[10/09/10 14:27:30] TRACE: Sources of catalalogue B retieve in 42.504 s. Nside=1, HealpixIdx=10, Nbr of srcs=103721508
[10/09/10 14:27:59] TRACE: - KDTree computed in 28.983 s. Err max = 1.0 arcsec
[10/09/10 14:27:59] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 90000 ; Nbr cp = 7484
Thread 1 ; Nbr correlated = 90000 ; Nbr cp = 7455
Thread 2 ; Nbr correlated = 80000 ; Nbr cp = 6213
Thread 3 ; Nbr correlated = 90000 ; Nbr cp = 7236
Thread 4 ; Nbr correlated = 90000 ; Nbr cp = 8407
Thread 5 ; Nbr correlated = 80000 ; Nbr cp = 7152
Thread 6 ; Nbr correlated = 80000 ; Nbr cp = 7470
Thread 7 ; Nbr correlated = 90000 ; Nbr cp = 7894
Thread 8 ; Nbr correlated = 80000 ; Nbr cp = 7534
Thread 9 ; Nbr correlated = 80000 ; Nbr cp = 7258
Thread 10 ; Nbr correlated = 70000 ; Nbr cp = 6484
Thread 11 ; Nbr correlated = 80000 ; Nbr cp = 7547
Thread 12 ; Nbr correlated = 87620 ; Nbr cp = 8245
Thread 13 ; Nbr correlated = 70000 ; Nbr cp = 6374
Thread 14 ; Nbr correlated = 80000 ; Nbr cp = 7469
Thread 15 ; Nbr correlated = 70000 ; Nbr cp = 6427
[10/09/10 14:28:23] TRACE: Correlation of the 1307620 SDSS7 src + write result performed in 23.268 s
[10/09/10 14:28:24] TRACE: Number of source in each catalogue evaluated in 858 ms. Nside=1, HealpixIdx=11, Nbr of srcs B=22898187 Nbr of srcs A (with borders)=2758604
[10/09/10 14:28:32] TRACE: Sources of catalalogue B retieve in 8.533 s. Nside=1, HealpixIdx=11, Nbr of srcs=22898187
[10/09/10 14:28:35] TRACE: - KDTree computed in 3.078 s. Err max = 1.0 arcsec
[10/09/10 14:28:35] TRACE: Ready to correlate!
Thread 0 ; Nbr correlated = 180000 ; Nbr cp = 20531
Thread 1 ; Nbr correlated = 180000 ; Nbr cp = 20242
Thread 2 ; Nbr correlated = 180000 ; Nbr cp = 19643
Thread 3 ; Nbr correlated = 170000 ; Nbr cp = 18314
Thread 4 ; Nbr correlated = 180000 ; Nbr cp = 19873
Thread 5 ; Nbr correlated = 170000 ; Nbr cp = 18701
Thread 6 ; Nbr correlated = 180000 ; Nbr cp = 19971
Thread 7 ; Nbr correlated = 170000 ; Nbr cp = 19483
Thread 8 ; Nbr correlated = 170000 ; Nbr cp = 20268
Thread 9 ; Nbr correlated = 170000 ; Nbr cp = 20308
Thread 10 ; Nbr correlated = 160000 ; Nbr cp = 18588
Thread 11 ; Nbr correlated = 170000 ; Nbr cp = 20048
Thread 12 ; Nbr correlated = 170000 ; Nbr cp = 20221
Thread 13 ; Nbr correlated = 170000 ; Nbr cp = 20385
Thread 14 ; Nbr correlated = 168604 ; Nbr cp = 19731
Thread 15 ; Nbr correlated = 170000 ; Nbr cp = 20418
[10/09/10 14:28:45] TRACE: Correlation of the 2758604 SDSS7 src + write result performed in 9.941 s
2010-09-10 02:28:46 : END

-- PineauFrancoisXavier - 01 Jun 2010