Add Azure Files support to persistent storage documentation #54055
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add Azure File Shares to the list of supported shared filesystem options - Update section header to include Azure File Shares - Add commented example showing Azure File Share mount path - Update description text to mention Azure File Shares alongside other options This change documents that Azure File Shares with NFS protocol can be used as persistent storage for Ray Train, similar to AWS EFS and Google Filestore. Closes #54054 Signed-off-by: masoud@anyscale.com <masoud@anyscale.com>
|
LGTM |
Signed-off-by: elizabethhu13 <elizabeth@anyscale.com>
jjyao
reviewed
Jun 25, 2025
| @@ -29,7 +29,7 @@ Here are some capabilities that persistent storage enables: | |||
| and artifacts to share them with others or use them in downstream tasks. | |||
|
|
|||
|
|
|||
| Cloud storage (AWS S3, Google Cloud Storage) | |||
| Cloud storage (AWS S3, Google Cloud Storage, Azure Blob) | |||
There was a problem hiding this comment.
Have we tested this? I'm actually not sure whether azure blob is supported or not here.
There was a problem hiding this comment.
@jjyao yes Azure blob has been tested and presigned url was fixed awhile ago (confirmed with Janet)
There was a problem hiding this comment.
Suggested change
| Cloud storage (AWS S3, Google Cloud Storage, Azure Blob) | |
| Cloud storage (AWS S3, Google Cloud Storage, Azure Blob Storage) |
dstrodtman
approved these changes
Jun 25, 2025
| @@ -74,11 +74,13 @@ Use by specifying the shared storage path as the :class:`RunConfig(storage_path) | |||
| storage_path="/mnt/cluster_storage", | |||
| # HDFS example: | |||
| # storage_path=f"hdfs://{hostname}:{port}/subpath", | |||
| # Azure File Shares example: | |||
| # storage_path="/mnt/azure-fileshare", | |||
There was a problem hiding this comment.
Just wondering: how is this getting mounted? Sorry if this is an ignorant question, I just don't see any new docs or x-refs that demonstrate this for Azure.
| @@ -57,8 +57,8 @@ Ensure that all nodes in the Ray cluster have access to cloud storage, so output | |||
| In this example, all files are uploaded to shared storage at ``s3://bucket-name/sub-path/experiment_name`` for further processing. | |||
|
|
|||
|
|
|||
| Shared filesystem (NFS, HDFS) | |||
| ----------------------------- | |||
| Shared filesystem (NFS, HDFS, Azure File Shares) | |||
There was a problem hiding this comment.
Just noting: NFS and HDFS are general, while the Azure File Shares is cloud-specific. Is it also subsumed under NFS technically or is it a different filesystem?
There was a problem hiding this comment.
good point @masoudcharkhabi I am fine here to say either (NFS, HDFS) OR (AWS EFS, Google Filestore, Azure File Shares, HDFS). Azure File Shares (when configured with NFS) is subsumed by NFS
| @@ -15,7 +15,7 @@ A Ray Train run produces :ref:`checkpoints <train-checkpointing>` that can be sa | |||
|
|
|||
| **Ray Train expects all workers to be able to write files to the same persistent storage location.** | |||
| Therefore, Ray Train requires some form of external persistent storage such as | |||
| cloud storage (e.g., S3, GCS) or a shared filesystem (e.g., AWS EFS, Google Filestore, HDFS) | |||
| cloud storage (e.g., S3, GCS, Azure Blob) or a shared filesystem (e.g., AWS EFS, Google Filestore, Azure File Shares, HDFS) | |||
There was a problem hiding this comment.
Suggested change
| cloud storage (e.g., S3, GCS, Azure Blob) or a shared filesystem (e.g., AWS EFS, Google Filestore, Azure File Shares, HDFS) | |
| cloud storage (e.g., S3, GCS, Azure Blob Storage) or a shared filesystem (e.g., AWS EFS, Google Filestore, Azure File Shares, HDFS) |
This is the official name. Just wondering: do we require ADLS Gen2, or do we support vanilla blob storage?
| @@ -29,7 +29,7 @@ Here are some capabilities that persistent storage enables: | |||
| and artifacts to share them with others or use them in downstream tasks. | |||
|
|
|||
|
|
|||
| Cloud storage (AWS S3, Google Cloud Storage) | |||
| Cloud storage (AWS S3, Google Cloud Storage, Azure Blob) | |||
There was a problem hiding this comment.
Suggested change
| Cloud storage (AWS S3, Google Cloud Storage, Azure Blob) | |
| Cloud storage (AWS S3, Google Cloud Storage, Azure Blob Storage) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
The Ray Train persistent storage documentation currently lists AWS EFS and Google Filestore as supported shared filesystem options, 8000 but does not mention Azure File Shares. Azure File Shares support NFS protocol (NFS 4.1) and are commonly used in Azure environments, making them a natural addition to the documentation alongside other cloud filesystem options.
This change documents that Azure File Shares can be used as persistent storage for Ray Train, similar to AWS EFS and Google Filestore.
Related issue number
Closes #54054
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.