Fixed a bug in preprocessing in crop_to_nonzero. #2796 #2824
Conversation
|
Hi there, thanks for the comprehensive writeup. I believe however that there are some thins you did not interpret quite right, leading to the perception there is a bug or unintended behavior. I have already written a short response outlining the need for the nonzero mask in the issue you link to, here is a rundown for why none of the suggestions are quite fitting:
Since you seem to have gone quite deep into the cropping and normalization rabbit hole may I ask whether you have encountered any issues arising from it? I would be very interested in that! PS: also remember that the -1 mask is basically never used unless cropping to nonzero region resulted in a substantial reduction in image size in which case it is used to apply normalization in a masked way |
|
Thank you very much for your answer!
I then dug into the details of avcl_utils, where the logic for how the mask is constrained by the bounding box works differently. I built it using that approach and got these results. This is a yellow rectangle spanning the entire volume - its contours are only visible a bit at the top (i.e., the overall image size remains the same, and this did not affect the final result). If you look at the mask itself, you can see that binary_fill_holes still does introduce points inside the body, although this only affects the first and last axial slices.
If we talk about how to isolate the body using some heuristics, then it’s probably not worth using I tried to segment the body mask using heuristics. I had hypotheses that, by using certain intensity statistics of the image, I would be able to delineate the body mask cleanly, but those approaches did not work. The best option turned out to be choosing a specific HU threshold and segmenting based on that [I settled on –600 HU, although I don’t really know why - it was empirical]. Overall, ended up with three versions that work reliably:
I tried other approaches based on image-intensity statistics or more generic table-based methods, but the results were unclear, so I’m not including them here. I understand that this example does not cover every possible case, but the conclusions were drawn based on a review of the flow of studies from different hospitals and hospitals.. There can be significant problems when a patient has their head positioned such that the mouth and nose are included, causing the largest connected component of the background to encompass the airways and lungs - making the last two methods less stable and preventing the lungs from being included as part of the body. Also understand that this may not work well if the organs (for example, the lungs) have already been pre-cut before training nnUNetv2, I have not tested these methods on such cases. Additionally, CT equipment settings can vary greatly, and the patient may not be separable from the table on which they lie. However, I believe that if we apply your method for obtaining the bounding box, everything will work much more reliably. Maybe this will help you somehow. Best, |
Hi!
#2796
I’ve identified the root cause of the issue. In
preprocessing.preprocessors.default_preprocessor, the functioncrop_to_nonzerois called to generate anonzero_maskof the image and crop to the region of interest—but that region is defined as every non-zero pixel in the image. Clearly, that’s not correct: any voxel inside the patient can be non-zero, and manufacturer-specific differences in CT scanner background intensity mean the background is almost never exactly zero.I see several possible solutions:
Swap the preprocessing steps: run
normalizefirst, thencrop_to_nonzero. However, this has drawbacks: if users have already cropped tightly around an organ or body region, the mask may over-crop and break training. Moreover, the CT normalization used doesn’t zero out voxels outside the body, which could lead to label errors (for example, mask labels becoming –1).Remove this heuristic entirely and let users define the ROI before preprocessing. This would increase training time, but I believe avoiding hard-to-detect errors is worth it. I suggest this option.
Add a toggle parameter to enable or disable cropping. But this approach will always perform worse than other options and require extra tuning, since background intensities vary so widely. A better idea is to leverage your body-localization model [Foreground-and-Anonymization-Area-Segmentation](https://github.com/MIC-DKFZ/Foreground-and-Anonymization-Area-Segmentation). The downsides are needing to download additional weights and perform an extra inference step, plus a small chance of failure on unusual anatomies. In my opinion, though, it’s far more robust than any simple algorithmic ROI heuristic.
I spent considerable time developing heuristics to separate patient body from background, but they invariably fail when applied to diverse data from different hospitals and scanners (on the order of 1,000–2,000 varied samples). In my experience, there’s no reliable algorithmic solution for this - it simply didn’t work for me.
I understand that at the moment this doesn’t cause major training errors—since the training pipeline applies an augmentation that replaces label values of –1 with 0—but it could potentially lead to errors if mask values, rather than background, get replaced.