8000 Resampled configs from the Priorband Sampler · Issue #217 · automl/neps · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Resampled configs from the Priorband Sampler #217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Sohambasu07 opened this issue May 7, 2025 · 0 comments
Open

Resampled configs from the Priorband Sampler #217

Sohambasu07 opened this issue May 7, 2025 · 0 comments
Assignees
@Sohambasu07

Comments

@Sohambasu07
Copy link
Collaborator
Sohambasu07 commented May 7, 2025 •
edited
Loading

While running Priorband, the PriorbandSampler sometimes resamples configs. This would be fine as long as the configs don't already exist in all of the rungs of all the brackets (which is extremely unlikely). The issue here is that the same config is resampled and a lower rung (hence, fidelity value) is assigned to it. This would cause an issue when calculating continuations in the following manner: config: (f1, f2-f1, f3-f2-f1), where f3>f2>f1.

Answer the following questions as and when they're figured out:

  1. Which sampling_policy inside the PriorbandSampler causes this issue?
    Incumbent-based sampling

  2. Are the sources of both the repeated samples the same? (which sampling_policy is used for each?)

    • The config is sampled for the very first time by the random sampling policy when the incumbent sampler is not yet active.
    • The config is resampled during incumbent-based sampling policy for a separate Hyperband bracket where the incumbent is not mutated at all and the same configuration is returned.
    • This can happen for either of 2 reasons:
      a. The mutate_config() function takes in the incumbent_config with the fidelity parameter and selects HPs for perturbation based on all of those. So, at times, the fidelity parameter is chosen but since the fidelity is not there in the mutation_prior which is generated from this incumbent_config, it's not really selected and the same configuration is passed back.
      b. Sometimes, none of the HPs in the config might be selected if all the values randomly generated here is < mutation_rate. In that case the same config is returned.

  3. Why is the resampled mutated_config assigned a lower rung than before?

  4. How to assign a configuration sampled by the incumbent-based sampling policy to a rung? And thereafter, how to handle resamples?

  5. Bookkeeping for continuations in Neps?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Sohambasu07 Sohambasu07

Labels
None yet
Projects
NePS Project Board
Status: No status
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Sohambasu07
0