Implement correction parameter API for FlatFieldStep #5242
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hbushouse
approved these changes
Aug 13, 2020
|
About the "where are the correction parameters stored": It is in the step attribute "correction_pars". This is set when a Step is run. If a Step is then re-run or has the Step.correction_pars explicitly set, and the attribute Step.use_correction_pars is set, then it is used instead of recalculation based on reference files, etc. Currently there is no option to save as part of the Step. But that is something to be discussed. |
|
So the correction arrays (or a datamodel containing them) are somehow stored in memory somewhere within the Step class instance? |
stscieisenhamer
added a commit
to stscieisenhamer/jwst
that referenced
this pull request
Aug 26, 2020
…5242) Changes: * Fill out spec2 to take advantage of inverse and correction pars * Setup the correction parameters attributes for Step * Implement correction_pars * Fix FlatFIeldStep issue with dropping operation completely * Fix pep8 issue * Remove redundant instrument check
stscieisenhamer
added a commit
to stscieisenhamer/jwst
that referenced
this pull request
Aug 28, 2020
…5242) Changes: * Fill out spec2 to take advantage of inverse and correction pars * Setup the correction parameters attributes for Step * Implement correction_pars * Fix FlatFIeldStep issue with dropping operation completely * Fix pep8 issue * Remove redundant instrument check
stscieisenhamer
added a commit
to stscieisenhamer/jwst
that referenced
this pull request
Sep 2, 2020
…5242) Changes: * Fill out spec2 to take advantage of inverse and correction pars * Setup the correction parameters attributes for Step * Implement correction_pars * Fix FlatFIeldStep issue with dropping operation completely * Fix pep8 issue * Remove redundant instrument check
stscieisenhamer
added a commit
that referenced
this pull request
Sep 3, 2020
…ec2 (#5302) Changes: * Move all the step decisions to _step_verification * Bug fixes from refactor * Remove the use of `input` as a variable name * Sketch out how master background would work. * Clean up the steps needed and updated commentary * MSB NIRSpec flow agreed upon after 20200730 meeting * Base structure to incorporate master background for MOS * Official add master background but disable by default (#5213) * JP-1630: Implement calspec2 function to create MOS master bkg (#5220) * A few minor updates to create_background_from_multislit (#5221) * A few minor updates to create_background_from_multislit * fix a flake8 issue * JP-1631: implement master bkg mapping to MOS slits (#5222) * JP-1631: implement master bkg mapping to MOS slits * Fix function reference * JP-1632: implement apply_master_background function in calwebb_spec2 (#5224) * Spec2 step refactor (#5214) Changes: * Start spec2 refactor. Move bkg_subtraction decision tree * Move step checking for imprint and msa_flagging * Move step decisions for straylight, fring, pathloss, and barshadow * Move all the step decisions to _step_verification * Bug fixes from refactor * Break of processing into individual methods specific to data types * Fix PEP8 and add changelog * Updates from reviewer comments and typo fixes * Move common step settings up front. * Put initial input in a context manager. * Fix minor formatting issue. * Rebase on master and cleanup * Update `flat_field` to allow pre-computed flat for NRS IFU * Fix bug in saving interpolated flat field option * Enable flat_field.nirspec_ifu to take original flat * Test for using predefined flat for nirspec ifu * Provide for a user-supplied flat from the FlatFieldStep parameters * Setup test for FlatFieldStep using a supplied flat field * Setup nrs_brightobj for user-supplied flats * Enabled user-supplied flat for nirspec msa * Setup test for nirspec mos user-supplied flat field * Update changelog * Add pytest.mark.bigdata to the appropriate tests * Fix big data tests and pep8 issues * JP-1587 Implement inverse of the FlatFieldStep (#5236) Changes: * Implement inversion for nirspec msa * Add inversion for nirspec bright object * Implement inversion for nirspec ifu * Completed inversion for all other modes * Record user-supplied flat as R_FLAT * Remove use of datamodels.open in a test * Fix sphinx build issue * Use the already opened datamodel to retrieve filename from * Remove unused import * More docstring fixes but not sure if these are the problems * Implement correction parameter API for FlatFieldStep (#5242) Changes: * Fill out spec2 to take advantage of inverse and correction pars * Setup the correction parameters attributes for Step * Implement correction_pars * Fix FlatFIeldStep issue with dropping operation completely * Fix pep8 issue * Remove redundant instrument check * Jp 1588 Implement inverse of the PathLossStep (#5247) Changes * Compartmentalize the different pathloss algorithms. * Typo fixes * Implement the saving and using correction_pars for MOS * Add testing for MOS PathLossStep correction parameters * Implement correction_pars usage for Fixed-slit PathLossStep * Test for nirspec fixed-slit pathloss correction pars * Add PathLossStep correction pars to nirspec ifu mode * Implement PathLossStep correction_pars usage for IFU * Warning message for NIS_SOSS mode * Implement inverse PathLossStep for nirspec mos * Implement PathLossInversion for nirspec fixed-slit * Implement PathLossStep inversion for nirspec ifu * Implement force of source type for nirspec mos * Implement source type forcing for IFU and fixed-slit * Setup Spec2Pipeline to run with just pathloss and flat_field * Remove redundant check on correction_pars * Fix unit tests for correction_pars * Fix regressions and pep8 * Update changelog * Jp 1589 Implement inverse of the BarShadowStep (#5248) Changes: * Setup BarShadowStep to handle inversion, source type coercion, and correction_pars * Implemented BarShadowStep correction_pars and inverse * Implement forcing source type for BarShadowStep * Implement testing for all new feature * Include barshadow in spec2 master background subtraction * Jp 1590 Implement inverse for PhotomStep and complete MBS for NIRSpec MOS in Spec2Pipeline (#5268) Changes: * Initial implementation using correction_pars * Ensure everything is pre-set in the DataSet initialization * Retrieval of spectral order using `get` * Setup test for PhotomStep inversion * Add inverse and source_type to `DataSet` initialization * Fixed inversion for MOS data * Complete implementation of source type override * Update commentary and fix source type setting bugs * Set units for photom inverse to be "DN/s" * Update changelog * Repair pep8 issues * Update commentary * Update nrsmsb branch for assign_wcs fix and log messages (#5272) * Add ability to change artifactory environment (#5278) Changes: * Update to match master and restrict to one test again * Update to versions in master * Add back setting of artifactory env * Remove single test * Update slow flag for some tests (#5281) * Fix rebase errors * Fix for rebase losing the flat field fix * Fix changelog to move everything into 0.17.1 release * Set S_MSBSUB when performing NIRSpec MOS master background * Set S_MSBSUB to the correct value of COMPLETE * Do not perform master background if background subtraction has occurred * Fix test scope * Allow jobs to not specify the artifactory environment. * Check for null environment specification * Get groovy syntax right * Remove single test test Co-authored-by: Howard Bushouse <bushouse@stsci.edu>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement the "easy/lazy" method of passing correction parameters around for FlatFieldStep.
Side note: The regressions discovered a dangling-if-without-else condition. See line 293 in flat_field.py. Basically, there was the possibility that NIRSPec-like model could be passed in, but have no flat field applied silently. Do not know if the condition could have happened in real life. However, because of changes where the reference files are no longer always required, the condition became possible. A runtime exception has been added just to ensure that the condition does not go unnoticed.