feat(runtime_env): add Azure Blob Storage support #53135
Conversation
99aee2c to
7b67c55
Compare
Add support for Azure Blob Storage URIs in Ray's runtime_env, allowing users to specify Azure as a location for their dependencies. Implement two authentication methods: connection string and managed identity. Closes ray-project#38316 Signed-off-by: Walid El Bouchikhi <walid.elbouchikhi@datadoghq.com>
7b67c55 to
ffb0d1a
Compare
|
cc @edoakes to find someone to review? |
| azure_storage_connection_string = os.getenv( | ||
| "AZURE_STORAGE_CONNECTION_STRING" | ||
| ) | ||
| azure_storage_account_name = os.getenv("AZURE_STORAGE_ACCOUNT") | ||
|
|
||
| # Connection string authentication | ||
| if azure_storage_connection_string: | ||
| tp = { | ||
| "client": BlobServiceClient.from_connection_string( | ||
| azure_storage_connection_string | ||
| ) | ||
| } |
There was a problem hiding this comment.
The gs and aws clients don't require you to pass in a connection string. Is there a version of the Azure Blob Storage Client that can handle this transparently? e.g. option number 1 described here
There was a problem hiding this comment.
Yes, the second option with ManagedIdentity can help on that
I added the connection string as this was the first option described in smart_open library https://github.com/piskvorky/smart_open?tab=readme-ov-file#azure-credentials
I'll remove it and add the DefaultAzureCredential that will eventually fallback to ManagedIdentity
| To use packages via ``Azure`` URIs, you must have the ``smart_open``, ``azure-storage-blob``, and ``azure-identity`` libraries (you can install them using ``pip install smart_open[azure] azure-storage-blob azure-identity``). | ||
| Ray supports two authentication methods for Azure Blob Storage: | ||
|
|
||
| 1. Connection string: Set the environment variable ``AZURE_STORAGE_CONNECTION_STRING`` with your Azure storage connection string. |
There was a problem hiding this comment.
This doesn't seem secure. I don't think we should support this for production.
There was a problem hiding this comment.
@walkoss can this be removed from the docs since we're only supporting Managed Identity now?
…Credential one Signed-off-by: Walid El Bouchikhi <walid.elbouchikhi@datadoghq.com>
Signed-off-by: Walid El Bouchikhi <walid.elbouchikhi@datadoghq.com>
Signed-off-by: Walid <walkoss@pm.me>
4a037be to
63c60f3
Compare
Signed-off-by: Walid El Bouchikhi <walid.elbouchikhi@datadoghq.com>
63c60f3 to
26cd5f4
Compare
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
Add support for Azure Blob Storage URIs in Ray's runtime_env, allowing users to specify Azure as a location for their dependencies.
Closes #38316
Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.