Create Cross-Validation example #317
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Done! |
|
The documentation is not available anymore as the PR was closed or merged. |
sgugger
approved these changes
Apr 19, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Cross-Validation Example + Tests
What does this add?
This adds an example to
examples/by_featurefor performing Cross Validation withAccelerate.Along with this, since we cannot run Cross-Validation well on 4 examples (our test sample dataset), I have elected to move the bits over from
transformersso we can haveslowtests. The CV examples run under slow, and take quite a bit (even on GPU, a minute). Ideally I should come up with a better solution for this, but I think for now if it is the only one it is not the end of the world.This also implements the "skip_example" feature mentioned in my previous PR for making sure all aspects of a test are implemented in
complete_*_example.py, since cross-validation is too niche for that.Who is it for?
Users of Accelerate, especially Kagglers, who want to perform Cross Validation with Accelerate and still have clean code
Why is it needed?
Cross-Validation has a variety of setups that need to happen, and its not always straightforward how to do so with Accelerate in those instances (such as running
Accelerator.clean_memory()at the end of a fold. This example shows how to do thisWhat parts of the API does this impact?
User-facing:
A new
cross_validation.pyscript, with a single added argument ofnum_foldsBasic Usage Example(s):
python examples/cross_validation.py --num_folds 2
Anticipated maintence burden? (What will happen in say, 3 months if something changes)
Most likely a better way of having "longer" example datasets to test will be used, and this test will need a refactor as a result