[RFC] PyTorch next wheel build platform: manylinux-2.28 #123649
Comments
|
@AlekseiNikiforovIBM recently adds the docker build image for s390x with the RedHat 9 base image from https://catalog.redhat.com/software/containers/ubi9/ubi/615bcf606feffc5384e8452e?architecture=amd64&image=65e093e60a21b531a96f93ca. But going with the official manylinux suggestion sounds like a good choice too as long as they have all the different flavors that we need and compatible with RedHat. |
|
Torch-TRT used to use the manylinux (Centos 7 based) containers for build and tests. Currently, TensorRT 10 only supports RHEL 9 and above (with glibc 2.28+ https://pypi.org/project/tensorrt/10.0.0b6/#files). It would be great if Pytorch supports these latest almalinux based releases soon as they would be compatible. So this feature would be of high interest for us and it would unblock our workflow. |
|
I don't think we should base on RHEL considering getting access to RHEL images is based on if you have a subscription or not (or at least that's been my experience, happy to be wrong here). I'd say we should go with whatever the official upstream |
|
manylinux-2.28 is the next standard we should be moving to. And perhaps it's also time to kill PRE_CXX11 ABI (which were necessary for all manylinux standards prior to 2.28) |
I'd agree here, no need to keep dinosaurs around if we're moving to the future! |
|
TensorRT is moving back to manylinux_2_17 after some push back from customers. Although this shouldn't create issues for projects like PyTorch consuming TRT using manylinux_2_28 due to glibc backward compatibility. The CXX ABI change is also being discussed (slaying dinosaurs) |
Are there any estimate timelines on when this can happen? IIUC most of current builds are still targeting CXX03 ABI (from looking at https://download.pytorch.org/whl/torch/) |
|
I have the PRs for aarch64-linux platform CD migration to manylinux 2_28. but I see few OS distros (for example Amazon Linux2) is with glibc2_26 and switching to manylinux2_28 will break compatibility with it. I would suggest we first make the announcement about the timeline for this switch to avoid surprises and disappointments for customers. |
|
@snadampal @malfet @seemethere As per: https://aws.amazon.com/amazon-linux-2/faqs/ |
|
Our CI are still on Amazon Linux 2, so I guess we need to migrate it to use Amazon Linux 2023 before the the new wheel could even pass validation. |
|
@snadampal You are correct wheel generated with manylinux2_28 is not compatible with Amazon Linux 2. Here is the test: |
|
Hi @atalman, we're working on XPU nightly manylinux wheel build enabling, and for XPU we also need to use manylinux_2_28 environment. But I found that the official quay.io/pypa/manylinux_2_28_x86_64 docker image don't provide shared library libpython*.so for each python version under /opt/python (refer pypa/manylinux#255), so we can't use it to build pytorch python wheel directly, we may still need to rebuild the python by ourselves. How do you think about it? |
|
For Intel GPU, we will submit a PR to support ABI=0 if PT2.5 cannot switch to ABI=1 mode. Due to the ABI change for Intel GPU software breaking the backward compatibility, we will refine the cmake a little bit - #130110 FYI |
|
@atalman @malfet @albanD @seemethere Actually, do we need to support the _GLIBCXX_USE_CXX11_ABI=0 case for PyTorch wheels, or can we sunset that requirement? |
|
@jithunnair-amd that depends on who your customers. If you believe that this is fine for ROCm, please do not hesitate to propose a patch |
Okay, I think we can start with setting |
…CUDA11.8, CUDA12.4 (pytorch#141565) For release 2.6 we will be using only CUDA 12.6 binaries on Manylinux 2.28. Issue: pytorch#123649 Pull Request resolved: pytorch#141565 Approved by: https://github.com/Skylion007, https://github.com/huydhn, https://github.com/malfet
Follow #123649 Works for #114850 Pull Request resolved: #142210 Approved by: https://github.com/EikanWang, https://github.com/atalman, https://github.com/malfet
Follow #123649 Works for #114850 Pull Request resolved: #142210 Approved by: https://github.com/EikanWang, https://github.com/atalman, https://github.com/malfet
Fixes pytorch#123649 Use Manylinux 2_28 Docker builds for PyTorch Nightly builds This moves the wheels to a Docker image that uses : ``quay.io/pypa/manylinux_2_28_x86_64`` as a base rather then ``centos:7`` which is EOL on June 30, 2024. Information: https://github.com/pypa/manylinux#manylinux_2_28-almalinux-8-based manylinux_2_28 (AlmaLinux 8 based) Toolchain: GCC 13 Built wheels are also expected to be compatible with other distros using glibc 2.28 or later, including: Debian 10+ Ubuntu 18.10+ Fedora 29+ CentOS/RHEL 8+ This migration should enable us to migrate to latest CUDNN version, and land this PR: pytorch#137978 Pull Request resolved: pytorch#138732 Approved by: https://github.com/Skylion007, https://github.com/malfet, https://github.com/huydhn
|
Switching PyTorch nightlies to Manylinux 2.28 builds and cxx11-abi by default: #143423 |
) Related to: #123649 Removing pre-cxx11 abi builds. As per announcement : https://dev-discuss.pytorch.org/t/pytorch-linux-wheels-switching-to-new-wheel-build-platform-manylinux-2-28-on-november-12-2024/2581 Pull Request resolved: #146200 Approved by: https://github.com/kit1980, https://github.com/huydhn
…rch#146200) Related to: pytorch#123649 Removing pre-cxx11 abi builds. As per announcement : https://dev-discuss.pytorch.org/t/pytorch-linux-wheels-switching-to-new-wheel-build-platform-manylinux-2-28-on-november-12-2024/2581 Pull Request resolved: pytorch#146200 Approved by: https://github.com/kit1980, https://github.com/huydhn
We're dropping regular old manylinux so let's drop it here too Relates to #123649 Signed-off-by: Eli Uriegas <eliuriegas@meta.com> Pull Request resolved: #148129 Approved by: https://github.com/Camyll, https://github.com/huydhn, https://github.com/malfet, https://github.com/atalman ghstack dependencies: #148126
1. Migrate all the workflows to liux_job_v2 since its uses Manylinux 2.28 : RFC: pytorch/pytorch#123649 2. Remove stronghold workflow, since not used 3. Use get_stable_cuda_version to validate repacked binary size whls
We're dropping regular old manylinux so let's drop it here too Relates to pytorch#123649 Signed-off-by: Eli Uriegas <eliuriegas@meta.com> Pull Request resolved: pytorch#148129 Approved by: https://github.com/Camyll, https://github.com/huydhn, https://github.com/malfet, https://github.com/atalman ghstack dependencies: pytorch#148126
|
Removed workaround created in test-infra repo to support Manylinux 2014 workers: pytorch/test-infra#6491 |
|
Closing this as completed |
Related to Manylinux 2.28 migration: #123649 Cleanup old Docker files and `manylinuxaarch64-builder:cpu-aarch64` image which has been replaced by `manylinux2_28_aarch64-builder:cpu-aarch64` Pull Request resolved: #152428 Approved by: https://github.com/Skylion007, https://github.com/malfet
Uh oh!
There was an error while loading. Please reload this page.
🚀 The feature, motivation and pitch
CentOS EOL is scheduled for the summer: https://www.redhat.com/en/blog/fastest-road-centos-linux-red-hat-enterprise-linux
With official manylinux has moved to almalinux: https://github.com/pypa/manylinux/blob/main/docker/Dockerfile#L2
manylinux_2_28 (AlmaLinux 8 based)
Toolchain: GCC 12
x86_64 image: quay.io/pypa/manylinux_2_28_x86_64
aarch64 image: quay.io/pypa/manylinux_2_28_aarch64
ppc64le image: quay.io/pypa/manylinux_2_28_ppc64le
s390x image: quay.io/pypa/manylinux_2_28_s390x
Almalinux:
https://almalinux.org/
Opening this issue to gather feedback, decide on the timeline for migration of the wheel build images
Work Completed:
RFC: Discussion issue #126551
cc @seemethere @malfet @osalpekar @pytorch/pytorch-dev-infra @ptrblck
The text was updated successfully, but these errors were encountered: