8000 Fix interpolated flat suffix in MOS/FS case by melanieclarke · Pull Request #9209 · spacetelescope/jwst · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Fix interpolated flat suffix in MOS/FS case #9209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 26, 2025
Merged

Fix interpolated flat suffix in MOS/FS case #9209

merged 3 commits into from
Feb 26, 2025

Conversation

melanieclarke
Copy link
Collaborator
@melanieclarke melanieclarke commented Feb 21, 2025 •
edited
Loading

Closes #9208

In the NIRSpec MOS/FS mode, the flat_field step is run twice: once for MOS slits, then for any planned FS slits. Intermediate flat-corrected files for the second run are saved with an additional _fs suffix, to avoid overwriting the intermediate files from the first run. The interpolated flat intermediate output did not receive this additional suffix, so FS flats were overwriting the MOS flats. This PR fixes the oversight.

I added the interpolatedflat and interpolatedflat_fs outputs to the MOS/FS spec2 regression test. The new truth files are from a local run on this branch, so I expect no diffs in the regtest run.

Tasks

  • request a review from someone specific, to avoid making the maintainers review every PR
  • add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
  • Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
    • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
    • update or add relevant tests
    • update relevant docstrings and / or docs/ page
    • start a regression test and include a link to the running job (click here for instructions)
      • Do truth files need to be updated ("okified")?
        • after the reviewer has approved these changes, run okify_regtests to update the truth files
  • if a JIRA ticket exists, make sure it is resolved properly
news fragment change types...
  • changes/<PR#>.general.rst: infrastructure or miscellaneous change
  • changes/<PR#>.docs.rst
  • changes/<PR#>.stpipe.rst
  • changes/<PR#>.datamodels.rst
  • changes/<PR#>.scripts.rst
  • changes/<PR#>.set_telescope_pointing.rst
  • changes/<PR#>.pipeline.rst

stage 1

  • changes/<PR#>.group_scale.rst
  • changes/<PR#>.dq_init.rst
  • changes/<PR#>.emicorr.rst
  • changes/<PR#>.saturation.rst
  • changes/<PR#>.ipc.rst
  • changes/<PR#>.firstframe.rst
  • changes/<PR#>.lastframe.rst
  • changes/<PR#>.reset.rst
  • changes/<PR#>.superbias.rst
  • changes/<PR#>.refpix.rst
  • changes/<PR#>.linearity.rst
  • changes/<PR#>.rscd.rst
  • changes/<PR#>.persistence.rst
  • changes/<PR#>.dark_current.rst
  • changes/<PR#>.charge_migration.rst
  • changes/<PR#>.jump.rst
  • changes/<PR#>.clean_flicker_noise.rst
  • changes/<PR#>.ramp_fitting.rst
  • changes/<PR#>.gain_scale.rst

stage 2

  • changes/<PR#>.assign_wcs.rst
  • changes/<PR#>.badpix_selfcal.rst
  • changes/<PR#>.msaflagopen.rst
  • changes/<PR#>.nsclean.rst
  • changes/<PR#>.imprint.rst
  • changes/<PR#>.background.rst
  • changes/<PR#>.extract_2d.rst
  • changes/<PR#>.master_background.rst
  • changes/<PR#>.wavecorr.rst
  • changes/<PR#>.srctype.rst
  • changes/<PR#>.straylight.rst
  • changes/<PR#>.wfss_contam.rst
  • changes/<PR#>.flatfield.rst
  • changes/<PR#>.fringe.rst
  • changes/<PR#>.pathloss.rst
  • changes/<PR#>.barshadow.rst
  • changes/<PR#>.photom.rst
  • changes/<PR#>.pixel_replace.rst
  • changes/<PR#>.resample_spec.rst
  • changes/<PR#>.residual_fringe.rst
  • changes/<PR#>.cube_build.rst
  • changes/<PR#>.extract_1d.rst
  • changes/<PR#>.resample.rst

stage 3

  • changes/<PR#>.assign_mtwcs.rst
  • changes/<PR#>.mrs_imatch.rst
  • changes/<PR#>.tweakreg.rst
  • changes/<PR#>.skymatch.rst
  • changes/<PR#>.exp_to_source.rst
  • changes/<PR#>.outlier_detection.rst
  • changes/<PR#>.tso_photometry.rst
  • changes/<PR#>.stack_refs.rst
  • changes/<PR#>.align_refs.rst
  • changes/<PR#>.klip.rst
  • changes/<PR#>.spectral_leak.rst
  • changes/<PR#>.source_catalog.rst
  • changes/<PR#>.combine_1d.rst
  • changes/<PR#>.ami.rst

other

  • changes/<PR#>.wfs_combine.rst
  • changes/<PR#>.white_light.rst
  • changes/<PR#>.cube_skymatch.rst
  • changes/<PR#>.engdb_tools.rst
  • changes/<PR#>.guider_cds.rst

@melanieclarke melanieclarke added this to the Build 11.3 milestone Feb 21, 2025
@melanieclarke
Copy link
Collaborator Author
melanieclarke commented Feb 21, 2025 •
edited
Loading

Regression tests here:
https://github.com/spacetelescope/RegressionTests/actions/runs/13460764584

No failures, as expected.

@codecov Codecov
Copy link
codecov bot commented Feb 21, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 73.65%. Comparing base (c4d6277) to head (108092d).
Report is 722 commits behind head on main.

Files with missing lines Patch % Lines
jwst/pipeline/calwebb_spec2.py 0.00% 6 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #9209      +/-   ##
==========================================
- Coverage   73.66%   73.65%   -0.02%     
==========================================
  Files         368      368              
  Lines       36436    36371      -65     
==========================================
- Hits        26842    26790      -52     
+ Misses       9594     9581      -13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jhennawi
Copy link

Just a quick question here, under this fix, will the final _interpolatedflat.fits file be aligned with the MultiSlit data models in the other files, i.e. _cal.fits. My understanding is that the _cal.fits file contains calibration for both the MOS slits and the FS slit, all in the same MultiSlit datamodel.

@jhennawi
Copy link

Sorry, nevermind my question. I see now from the commits that the FS flat data is being appended to the MOS flat data.

@melanieclarke
Copy link
Collaborator Author

@jhennawi - The FS flat data is not appended to the MOS flat data. There will be two separate interpolated flat output files: the 'interpolatedflat' file will contain the flat corrections for the MOS data and the 'interpolatedflat_fs' file will contain the corrections for the FS data.

This is aligned with the way we're handling all intermediate files for the fixed slits processed alongside MOS slits: they are saved to separate files with a '_fs' suffix. Only the final 'cal', 's2d', and 'x1d' products are reassembled into a single product.

It should be fairly straightforward to assemble an interpolated flat from the two saved products that contains both MOS and FS corrections, for your purposes.

@jhennawi
Copy link

Thanks @melanieclarke. Could you please send me a code snippet to read in the interpolated flat outputs from FS and MOS and append them into a single fits file so that they are aligned with the _cal.fits file.

From my perspective this design choice of separate files is not ideal. As a discerning user who is scrutinizing the calibrations carefully, I would prefer all the calibration images (pathloss, barshadow, wave, flat) to be in one MultiSlit data model. Already the division of these into a _cal file and an _interpolatedflat is rather cumbersome, and it's not obvious why that was necessary. Now that latter is being split into two separate files, and needs to be reassembled into a single MultiSlit model.

@melanieclarke
Copy link
Collaborator Author

Thanks @melanieclarke. Could you please send me a code snippet to read in the interpolated flat outputs from FS and MOS and append them into a single fits file so that they are aligned with the _cal.fits file.

This should do it:

from jwst import datamodels
d1 = datamodels.open('jw01219006001_04101_00001_nrs1_interpolatedflat.fits')
d2 = datamodels.open('jw01219006001_04101_00001_nrs1_interpolatedflat_fs.fits')
for slit in d2.slits:
     d1.slits.append(slit)
d1.save('jw01219006001_04101_00001_nrs1_interpolatedflat_assembled.fits')

From my perspective this design choice of separate files is not ideal. As a discerning user who is scrutinizing the calibrations carefully, I would prefer all the calibration images (pathloss, barshadow, wave, flat) to be in one MultiSlit data model. Already the division of these into a _cal file and an _interpolatedflat is rather cumbersome, and it's not obvious why that was necessary. Now that latter is being split into two separate files, and needs to be reassembled into a single MultiSlit model.

Understood. Since the intermediate products are not required for automated processing, this reassembly hasn't risen to the top of our development priorities, but I can pass on your feedback. In the meantime, this fix should at least get you access to the data you need for your manual reprocessing.

@melanieclarke melanieclarke marked this pull request as ready for review February 21, 2025 18:20
@melanieclarke melanieclarke requested review from a team as code owners February 21, 2025 18:20
@jhennawi
Copy link

Many thanks!

tapastro
tapastro approved these changes Feb 26, 2025
View reviewed changes
Copy link
Contributor
@tapastro tapastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@melanieclarke melanieclarke merged commit 62f112c into spacetelescope:main Feb 26, 2025
29 of 30 checks passed
@melanieclarke melanieclarke deleted the msa_fs_interp_flat branch February 26, 2025 20:59
@stscijgbot-jp stscijgbot-jp mentioned this pull request Mar 12, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tapastro tapastro tapastro approved these changes

Assignees
No one assigned
Labels
NIRSPEC pipeline regression_testing spec2 pipeline
Projects
None yet
Milestone
Build 11.3
Development

Successfully merging this pull request may close these issues.

bug found in handling of NIRSPEC MSA data plus fixed slit
3 participants
@melanieclarke @jhennawi @tapastro
0