Project

General

Profile

Actions

Bug #1006

closed

MATCHing in SCAMP is less effective when many exposures are processed together

Added by Emmanuel Bertin almost 4 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Urgent
Start date:
02/13/2018
Due date:
02/20/2018
% Done:

0%

Estimated time:
6.00 h
Spent time:

Files

refcat.jpg (110 KB) refcat.jpg Herve Bouy, 02/20/2018 12:59 PM
contrast.png (442 KB) contrast.png DECam fields: the XY_Contrast is shown as color scale. Problematic fields appear in red. Herve Bouy, 10/22/2019 09:09 AM
Actions #1

Updated by Emmanuel Bertin almost 4 years ago

  • % Done changed from 0 to 50

(/) I was able to pin down the issue using very small catalogs. The problem comes from the coordinate offset (and not the change of WCS) in the new_fieldset() routine. I will work on a fix in the coming days.

Actions #2

Updated by Emmanuel Bertin almost 4 years ago

  • Status changed from New to In Progress
  • % Done changed from 50 to 90

(/) Found the problem: this was due to star pairs being missed along diagonals due to the pair distance cut-off being too aggressive.
(!) There are still weird MATCHing results for the last field of a list.

Actions #3

Updated by Emmanuel Bertin almost 4 years ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Emmanuel Bertin to Herve Bouy
  • % Done changed from 90 to 100

(/) The (apparently) strange behavior mentioned earlier was due to the new field grouping algorithm that would change the ordering of fields in each group (and therefore on the display) depending on the distribution of fields on the celestial sphere. The field grouping function now restore the original ordering of fields.
(/) Fixed a few valgrind warnings (nothingh critical)
(/) Pushed version number to 2.6.4
(!) I had to modify test.py to accommodate the slight changes induced by the fix in MATCHing.
(/) The new version is now available on the bleeding channel repository.

Further testing welcome :)

Actions #4

Updated by Herve Bouy almost 4 years ago

  • % Done changed from 100 to 50

I tried on some DECam images that were having the problem, and it still fails...
I am doing some more tests and preparing a minimal sample of files for debugging.

Actions #5

Updated by Herve Bouy almost 4 years ago

I think I found a hint of where the problem lies.
I think it is in the CDS query.
While preparing a subset of data for the debugging, I did the following:
  1. ran scamp on a set of 30 catalogues. Identified a subset of 6 fields in one group that pass/fail the xmatch. I then decided to save the reference catalogue to speed up further tests
  2. ran scamp on this subset using the saved reference catalogue to confirm that some catalogs fail as well when scamping only this subset. Some catalogues indeed failed.
  3. by chance I ran scamp again on the subset,but this time retrieving the catalogue from CDS again instead of using the one previously saved. It worked.
    So I thought it had to be related to the query.

I checked the two reference catalogues and saw the surprising result attached:
You can see the problem when scamping the 30 fields. Is it a limitation of the CDS??? This is a very dense region (low galactic latitude)

Actions #6

Updated by Emmanuel Bertin almost 4 years ago

Thanks for this info! Just noticed that I had set up a hardwired 10,000,000 limit in the number of reference sources retrieved for each group in the CDS query. Are you hitting that limit? If yes, then the fix will be easy! :).

Actions #7

Updated by Herve Bouy almost 4 years ago

YES!
this is it!
10,028,965 :-)

Actions #8

Updated by Emmanuel Bertin almost 4 years ago

(/) OK there is a new SCAMP v2.6.5 RPM package just waiting for you. Just do a yum clean all followed by a yum update scamp and you are good to go! I have pushed the limit to 100,000,000 reference sources, which should have us covered for some time.

Actions #9

Updated by Herve Bouy almost 4 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 50 to 100

That solves my issue, thanks!!!

Actions #10

Updated by Emmanuel Bertin almost 4 years ago

  • Status changed from Resolved to Closed

... well at least it also helped fix a less obvious bug...

Actions #11

Updated by Herve Bouy about 2 years ago

  • File contrast.png contrast.png added
  • Status changed from Closed to In Progress
  • Priority changed from High to Urgent
  • % Done changed from 100 to 0

I am re-opening this issue because it happens again.
While scamping a bunch of DECam catalogues, some xmatch with the reference catalogue (which is a combination of Gaia DR2 + 2MASS) fail (XY_CONTRAST<2) while they work (XY_CONTRAST>~20) when scamped individually... See attached figure.
I'll make a minimum test sample so you can work on it.

DECam fields: the XY_Contrast is shown as color scale. Problematic fields appear in red.
Figure: DECam fields plotted on top of the reference catalogue: the XY_Contrast is shown in color. Problematic fields appear in red.

Actions #12

Updated by Herve Bouy about 2 years ago

A sample of 20 DECam catalogues and associated .ahead displaying the problem is in morpho in /raid/hbouy/debug_scamp/1006/
The scamp.conf and local external catalogue (called gaiadr2_2mass.refcat) used are also present.
The catalogue ct4m20140507t043830-V3.1.1-ooi.cat fails the external xmatch when scamped with all the other catalogues, but works when scamped alone.

Actions #13

Updated by Herve Bouy about 2 years ago

  • Status changed from In Progress to Closed

sorry... i seems it is related to my local Gaia DR2 + 2MASS catalogue!
If I use Gaia-DR2 only it works...

Actions

Also available in: Atom PDF