Project

General

Profile

Actions

Bug #887

open

cross-match with reference catalogue hangs forever

Added by Herve Bouy about 4 years ago. Updated about 4 years ago.

Status:
Feedback
Priority:
High
Start date:
10/01/2017
Due date:
% Done:

50%

Estimated time:
2.00 h
Spent time:

Description

In some case when scamping a rather long list of DECam catalogues (~700), the xmatching with the reference catalogue hangs forever.
It does not seem to be due to a corrupted catalogue, as splitting the list into small chunks works for all individual chunks.

The problem can be tested on dance09 in /raid/hbouy/scamp:

scamp @r_sdss.list

I attach a snapshot showing where it hangs (since over 36h now...)


Files

hanging.png (1.29 MB) hanging.png Herve Bouy, 10/01/2017 11:06 AM
still_hanging.png (520 KB) still_hanging.png Herve Bouy, 10/10/2017 07:44 AM
Actions #1

Updated by Emmanuel Bertin about 4 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 20
  • Estimated time set to 2.00 h

(/) This happens because, according to SCAMP, the 7th group in your list has an angular radius of 138°, which requires about 9 million reference sources. This is plain wrong of course. 361 exposures fit within a 6° radius, while the 3 others are at completely different positions in the sky.
(/) However these fields should not overlap. This has to be a bug, may be in the new "cleverer" grouping algorithm I introduced in V2.6 last June. Let me investigate.

Actions #2

Updated by Emmanuel Bertin about 4 years ago

  • % Done changed from 20 to 80

(/) I think I found the bug (in the field grouping algorithm). I will fix it later today.
(!) Note that this does not solve the matching inefficiency problem when dealing with long chains of fields (which could be something for Sébastien to tackle!), but at least it should take forever anymore.

Actions #3

Updated by Herve Bouy about 4 years ago

awesome!
and indeed a good task for Sebastien.

Actions #4

Updated by Emmanuel Bertin about 4 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 80 to 100

(/) Bug fixed in field grouping

Actions #5

Updated by Herve Bouy about 4 years ago

Re-opening: it has been hanging since yesterday afternoon...
There is one very large group of $\sim$700', could it be it?
I will try with 2MASS instead of Gaia, it's shallower so if it's the problem it should help

Actions #6

Updated by Emmanuel Bertin about 4 years ago

Are you sure you are running V2.6.2? With the same settings and catalogs it does not hang for me.

Actions #7

Updated by Herve Bouy about 4 years ago

  • Status changed from In Progress to Feedback

Indeed I confirm that the problem was with Gaia DR1!
I just switch to 2MASS, and it worked.
Are we having a memory issue for very large areas? this one was ~700'...
That will be an issue at some point

Actions #8

Updated by Emmanuel Bertin about 4 years ago

With Gaia it should work the same. Please kill your job and let me check.

Actions #9

Updated by Emmanuel Bertin about 4 years ago

No problem with Group 6 and Gaia here; however I get bad MATCHing over all Group 7.

Actions #10

Updated by Herve Bouy about 4 years ago

Ok, so it must be a GAIA-DR1 thing...
With 2MASS group 7 works fine

Actions

Also available in: Atom PDF