8000 pip install with BUILD_S3=TRUE broken · Issue #1451 · pytorch/data · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

pip install with BUILD_S3=TRUE broken #1451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
h-vetinari opened this issue Feb 24, 2025 · 2 comments · May be fixed by #1460
Open

pip install with BUILD_S3=TRUE broken #1451

h-vetinari opened this issue Feb 24, 2025 · 2 comments · May be fixed by #1460
Assignees
@ramanishsingh

Comments

@h-vetinari
Copy link

🐛 Describe the bug

In conda-forge, we're building torchdata using python -m pip install (after setting some environment variables like USE_SYSTEM_LIBS). However, c64801f (CC @andrewkho) removed the root CMakeLists.txt, which now trivially fails the cmake build for the python extension

data/tools/setup_helpers/extension.py

Lines 125 to 126 in d155220

subprocess.check_call(["cmake", str(_ROOT_DIR)] + cmake_args, cwd=self.build_temp)
subprocess.check_call(["cmake", "--build", "."] + build_args, cwd=self.build_temp)

when using

data/tools/setup_helpers/extension.py

Lines 57 to 58 in d155220

if _BUILD_S3:
return [Pybind11Extension(name="torchdata._torchdata", sources=[])]

Given that that commit comprehensively removed torchdata/csrc etc. the fact that the install breaks with BUILD_S3=TRUE is probably an oversight.

In fact, probably a lot of that helper module can be cleaned up, since (aside from the CMake call to nowhere) none of the following variables are being used anywhere anymore

data/tools/setup_helpers/extension.py

Lines 41 to 44 in d155220

_BUILD_S3 = _get_build("BUILD_S3", False)
_USE_SYSTEM_AWS_SDK_CPP = _get_build("USE_SYSTEM_AWS_SDK_CPP", False)
_USE_SYSTEM_PYBIND11 = _get_build("USE_SYSTEM_PYBIND11", False)
_USE_SYSTEM_LIBS = _get_build("USE_SYSTEM_LIBS", False)

Versions

Starting from 0.10.1
@divyanshk
Copy link
Contributor

Thanks for raising this. You are right! Since we removed datapipes from v0.10 onwards we probably should clean up that file.

@ramanishsingh ramanishsingh self-assigned this Feb 26, 2025
@ramanishsingh ramanishsingh linked a pull request Mar 1, 2025 that will close this issue
Open
@ramanishsingh
Copy link
Contributor

Hi @h-vetinari can you check if #1460 solves this or let me know how to build so that I can reproduce the error and check. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ramanishsingh ramanishsingh

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Clean extension.py pytorch/data
3 participants
@divyanshk @h-vetinari @ramanishsingh
0