Expose iterator shuffling through Trainer config #1557
Conversation
|
hmm, are we shuffling the validation set by default? that's kind of weird? it's ok if you don't want to test this. if I really wanted to test this, I would probably (in the test file) create a |
| @@ -166,6 +166,7 @@ def __init__(self, | |||
| patience: Optional[int] = None, | |||
| validation_metric: str = "-loss", | |||
| validation_iterator: DataIterator = None, | |||
| shuffle: bool = True, | |||
There was a problem hiding this comment.
Trainer has a custom from_params, so you'll need to add it there too
There was a problem hiding this comment.
good catch, thanks!
yeah, I noticed this too. I'd be in favor of turning off shuffling at validation, since I presume that's what we do at test time? thoughts? |
|
it looks like we are shuffling at test time too. 😬 anyway, it appears that we've always been shuffling the validation set, so I wouldn't casually change that behavior. (in theory it shouldn't make a difference, addition is commutative, but who knows) |
|
Can someone remind me why this isn't a constructor arg for the |
|
I assumed it was so the same iterator could behave differently on training / dev / test, although that's not possible right now |
* Expose iterator shuffling through Trainer config * Add shuffle to Trainer.from_params
Not really sure how to test this, any suggestions?