Description
While running Priorband, the PriorbandSampler sometimes resamples configs. This would be fine as long as the configs don't already exist in all of the rungs of all the brackets (which is extremely unlikely). The issue here is that the same config is resampled and a lower rung (hence, fidelity value) is assigned to it. This would cause an issue when calculating continuations in the following manner: config: (f1, f2-f1, f3-f2-f1), where f3>f2>f1.
Answer the following questions as and when they're figured out:
-
Which
sampling_policyinside thePriorbandSamplercauses this issue?
Incumbent-basedsampling -
Are the sources of both the repeated samples the same? (which
sampling_policyis used for each?)- The config is sampled for the very first time by the
randomsampling policy when theincumbent sampleris not yet active. - The config is resampled during
incumbent-basedsampling policy for a separateHyperbandbracket where the incumbent is not mutated at all and the same configuration is returned. - This can happen for either of 2 reasons:
a. Themutate_config()function takes in theincumbent_configwith the fidelity parameter and selects HPs for perturbation based on all of those. So, at times, the fidelity parameter is chosen but since the fidelity is not there in themutation_priorwhich is generated from thisincumbent_config, it's not really selected and the same configuration is passed back.
b. Sometimes, none of the HPs in the config might be selected if all the values randomly generated here is <mutation_rate. In that case the same config is returned.
- The config is sampled for the very first time by the
-
Why is the resampled
mutated_configassigned a lower rung than before? -
How to assign a configuration sampled by the
incumbent-basedsampling policy to a rung? And thereafter, how to handle resamples? -
Bookkeeping for continuations in Neps?
Metadata
Metadata
Assignees
Labels
Type
Projects
Status