Add gen #800
Add gen #800
Conversation
|
|
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #800 +/- ##
==========================================
- Coverage 96.73% 96.69% -0.04%
==========================================
Files 65 65
Lines 5077 5084 +7
Branches 880 882 +2
==========================================
+ Hits 4911 4916 +5
- Misses 85 86 +1
- Partials 81 82 +1
☔ View full report in Codecov by Sentry. |
cleanlab/outlier.py
Outdated
| ) | ||
| probs = softmax(pred_probs, axis=1) | ||
| probs_sorted = np.sort(probs, axis=1)[:,-M:] | ||
| ood_predictions_scores = -np.sum(probs_sorted**gamma * (1 - probs_sorted)**(gamma), axis=1) |
There was a problem hiding this comment.
please confirm:
You are transforming the GEN scores (posthoc) to ensure they lie in 0-1 and smaller values == datapoints that are more severe outliers
There was a problem hiding this comment.
To transform the GEN scores to lie in 0-1, I use 1-ori_gen_score/M as the output ood prediction scores. Please let me know if you have other ideas.
There was a problem hiding this comment.
seems fine to me. Please add:
- a comment in the code clarifying that you're doing this
- a unit test that ensures all values are in 0-1 range, and smallest values are the outliers in a toy dataset
There was a problem hiding this comment.
Yes I have already added the unit test to ensure all values are in 0-1 range. And for the smallest values things, I saw your another comment. Yes the ood score will be 1 for both entropy and least_confidence methods because the prediction confidence will be 1 for that datapoint. I tried using the mean of three means of data distribution and it works well for me.
There was a problem hiding this comment.
Thanks for your great contribution @coding-famer!
We really appreciate all of your benchmarking efforts.
Add GEN as new pred_probs based out-of-distribution detection method. For #728.