8000 post-processing multiple algorithms asserts · Issue #625 · numbbo/coco · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

post-processing multiple algorithms asserts #625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
TGlas opened this issue Feb 11, 2016 · 7 comments
Closed

post-processing multiple algorithms asserts #625

TGlas opened this issue Feb 11, 2016 · 7 comments
Assignees
@dtusar
Labels
bug Code-Postprocessing Priority-High

Comments

@TGlas
Copy link
TGlas commented Feb 11, 2016

I have run two algorithms, both in 5 dimensions. I can run post-processing for both individually with the current development branch (11.2.2016). However, if I try post-processing with the results of both algorithms in the exdata folder then I get an assertion. The error message is as follows:

$ python2 -m bbob_pproc exdata
Post-processing: will generate output data in folder ppdata
  this might take several minutes.
Post-processing (1): will generate output data in folder ppdata/exdata
  this might take several minutes.
  Data consistent according to test in consistency_check() in pproc.DataSet
/Users/tobi/.local/lib/python2.7/site-packages/bbob_pproc-0.0.0-py2.7.egg/bbob_pproc/rungeneric1.py:318: UserWarning: Data with multiple algId [('mytest', '% mytest'), ('MOCMA', '% Shark Default Implementation')] will be processed together.
  'will be processed together.')
Scaling figures...
Traceback (most recent call last):
  File "/Users/tobi/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/tobi/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/tobi/.local/lib/python2.7/site-packages/bbob_pproc-0.0.0-py2.7.egg/bbob_pproc/__main__.py", line 287, in <module>
    rungeneric.main(args)
  File "/Users/tobi/.local/lib/python2.7/site-packages/bbob_pproc-0.0.0-py2.7.egg/bbob_pproc/rungeneric.py", line 282, in main
    + ["-o", outputdir, alg])
  File "/Users/tobi/.local/lib/python2.7/site-packages/bbob_pproc-0.0.0-py2.7.egg/bbob_pproc/rungeneric1.py", line 349, in main
    outputdir, genericsettings.verbose)
  File "/Users/tobi/.local/lib/python2.7/site-packages/bbob_pproc-0.0.0-py2.7.egg/bbob_pproc/ppfigdim.py", line 531, in main
    plot(dictFunc[func], _valuesOfInterest, styles=styles)  # styles might have changed via config
  File "/Users/tobi/.local/lib/python2.7/site-packages/bbob_pproc-0.0.0-py2.7.egg/bbob_pproc/ppfigdim.py", line 359, in plot
    assert len(dictFunc[func][dim]) == 1
AssertionError
@dtusar
Copy link
Contributor
dtusar commented Feb 11, 2016

Usually you have to specify separately the path to each algorithm, e.g. python2 -m bbob_pproc exdata/alg1 exdata/alg2. I think that we don't support the case when you specify just one path for multiple algorithms.

@nikohansen
Copy link
Contributor

Agreed. That means the raised error should not be an assertion error, because it is a case to be expected to appear on non-compliant user input.

@nikohansen nikohansen added the bug label Feb 11, 2016
@TGlas
Copy link
Author
TGlas commented Feb 12, 2016

Thanks for the clarification. Specifying the algorithms individually works. However, this feels inconsistent, because for a single algorithm I can specify the exdata directory, for multiple algorithms the same fails. I'd like to propose to simply scan the directory to simulate the effect of python2 -m bbob_pproc exdata/* .

There is another small bug, in case of processing multiple algorithms the links to the single algorithm data in index.html are broken: the link text is empty (therefore not clickable), and the href starts with a slash (absolute path), therefore the link would not work, even if it was clickable.
EDIT: This is not reproducible, so please ignore for now.
EDIT 2: It happened again, so it seems to be a systematic bug. The links read: <H3>&nbsp;<a href="/templateBBOBarticle.html"></a></H3>

@nikohansen
Copy link
Contributor

I'd like to propose to simply scan the directory to simulate the effect of python2 -m bbob_pproc exdata/* .

The problem with this is the following: the user is allowed to distribute data from the same algorithm into several subfolders (and even subsubfolders). This interface would be broken if we would scan the folder for subfolders and feed each as if it came from a different algorithm. We could make an exception on the folder name exdata. I am not sure if this is too much of a hack though. Also it means that we would by default always post-process all data which have been simulated into the exdata folder over time. This is probably not what we want to have.

Yet, indeed, the raised error should clearly indicate how to (probably) proceed in this case.

@nikohansen
Copy link
Contributor

FTR, I added the bug label because the

raised error should not be an assertion error

@TGlas
Copy link
Author
TGlas commented Feb 12, 2016

Agreed, a constructive diagnostic message would do the job.

@dtusar dtusar self-assigned this Feb 17, 2016
@dtusar
Copy link
Contributor
dtusar commented Apr 8, 2016

An error with the explanation is now raised.

@dtusar dtusar closed this as completed Apr 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dtusar dtusar

Labels
bug Code-Postprocessing Priority-High
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@TGlas @nikohansen @brockho @dtusar
0