Closed
Description
What's the problem this feature will solve?
When the requirements of a project is torch, pip-tools is not able to get the latest available cuda version(10.2).
Instead pip-compile resolves to version 9.2.
The command I use is:
pip-compile --find-links=https://download.pytorch.org/whl/torch_stable.html --generate-hashes --upgrade --output-file=requirements-lock.txt
Describe the solution you'd like
pip-compile shouldn't fail when using tripple = and version 1.5.1. For any reason this strategy works for version 1.5.0.
Pip-compile should be able to get latest Pytorch with latest CUDA as default when plain torch requirement is selected and --find-links is correctly provided.
A real world example is the following project.
Alternative Solutions
The repository I am testing with it is this.
If I set in setup.py torch version with tripple = and torch version just before the latest, it is able to resolve to latest cuda version.
torch===1.5.0But if I change to torch===1.5.1, pip-compile fails with the following message:
Traceback (most recent call last):
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\dani_\Workspaces\Python\time_series_predictor\.env\Scripts\pip-compile.exe\__main__.py", line 9, in <module>
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\piptools\scripts\compile.py", line 444, in cli
results = resolver.resolve(max_rounds=max_rounds)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\piptools\resolver.py", line 169, in resolve
has_changed, best_matches = self._resolve_one_round()
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\piptools\resolver.py", line 274, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\piptools\resolver.py", line 380, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\piptools\repositories\pypi.py", line 229, in get_dependencies
download_dir, ireq, wheel_cache
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\piptools\repositories\pypi.py", line 181, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 362, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 314, in _get_abstract_dist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\operations\prepare.py", line 469, in prepare_linked_requirement
hashes=hashes,
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\operations\prepare.py", line 264, in unpack_url
unpack_file(file.path, location, file.content_type)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\utils\unpacking.py", line 252, in unpack_file
flatten=not filename.endswith('.whl')
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\utils\unpacking.py", line 114, in unzip_file
zip = zipfile.ZipFile(zipfp, allowZip64=True)
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\zipfile.py", line 1258, in __init__
self._RealGetContents()
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\zipfile.py", line 1325, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Unfortunately using the partial correct CUDA solution with tripple = in setup.py and version 1.5.0, uploading to pypi fails with the follwing output:
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: aUserName
Enter your password:
Uploading time_series_predictor-1.3.0-py3-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 108k/108k [00:01<00:00, 77.0kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Invalid value for requires_dist. Error: Invalid requirement: 'torch (===1.5.0)'.
Additional context
I tried to create a setup.py command too, in order to create a cron update job in CI.
But running it in vscode and windows results in the following error:
running upgrade
['pip-compile', '--find-links=https://download.pytorch.org/whl/torch_stable.html --generate-hashes --upgrade --output-file=requirements-lock.txt', 'C:\\Users\\dani_\\Workspaces\\Python\\time_series_predictor']
Running command: ['pip-compile', '--find-links=https://download.pytorch.org/whl/torch_stable.html --generate-hashes --upgrade --output-file=requirements-lock.txt', 'C:\\Users\\dani_\\Workspaces\\Python\\time_series_predictor']
Traceback (most recent call last):
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\req\req_file.py", line 573, in get_file_content
with open(url, 'rb') as f:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\dani_\\Workspaces\\Python\\time_series_predictor'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\dani_\Workspaces\Python\time_series_predictor\.env\Scripts\pip-compile.exe\__main__.py", line 9, in <module>
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\piptools\scripts\compile.py", line 405, in cli
options=repository.options,
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\piptools\_compat\pip_compat.py", line 27, in parse_requirements
filename, session, finder=finder, options=options, constraint=constraint
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\req\req_file.py", line 151, in parse_requirements
for parsed_line in parser.parse(filename, constraint):
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\req\req_file.py", line 330, in parse
for line in self._parse_and_recurse(filename, constraint):
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\req\req_file.py", line 335, in _parse_and_recurse
for line in self._parse_file(filename, constraint):
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\req\req_file.py", line 369, in _parse_file
filename, self._session, comes_from=self._comes_from
File "c:\users\dani_\workspaces\python\time_series_predictor\.env\lib\site-packages\pip\_internal\req\req_file.py", line 577, in get_file_content
'Could not open requirements file: {}'.format(exc)
pip._internal.exceptions.InstallationError: Could not open requirements file: [Errno 13] Permission denied: 'C:\\Users\\dani_\\Workspaces\\Python\\time_series_predictor'
Traceback (most recent call last):
File "setup.py", line 126, in <module>
'lxml'
File "C:\Users\dani_\Workspaces\Python\time_series_predictor\.env\lib\site-packages\setuptools\__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "setup.py", line 45, in run
subprocess.check_call(command)
File "C:\Users\dani_\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip-compile', '--find-links=https://download.pytorch.org/whl/torch_stable.html --generate-hashes --upgrade --output-file=requirements-lock.txt', 'C:\\Users\\dani_\\Workspaces\\Python\\time_series_predictor']' returned non-zero exit status 1.