8000 `exists` with `key` causes `PermissionError` · Issue #965 · fsspec/s3fs · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

exists with key causes PermissionError #965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
PythonFZ opened this issue May 22, 2025 · 1 comment
Closed

exists with key causes PermissionError #965

PythonFZ opened this issue May 22, 2025 · 1 comment

Comments

@PythonFZ
Copy link

I am running a s3-compatible minio storage and encounter the following issue

import s3fs

s3 = s3fs.S3FileSystem(anon=True, endpoint_url='https://minio.icp.uni-stuttgart.de')
print(s3.exists("public")) # works 
print(s3.exists('public/dvc/zntrack-examples/files/md5/f7/b8a247465f1aac988675ea086fd76a')) # also works
# print(s3.exists('public/dvc/zntrack')) # raises PermissionError: Forbidden
print(s3.exists('public/dvc/zntrack-examples')) # raises PermissionError: Forbidden

The error comes from

s3fs/s3fs/core.py

Lines 1445 to 1452 in e960296

out = await self._call_s3(
"head_object",
self.kwargs,
Bucket=bucket,
Key=key,
**version_id_kw(version_id),
**self.req_kw,
)

and can be circumvented by replacing

s3fs/s3fs/core.py

Line 1463 in e960296

except FileNotFoundError:

with the following although I assume this would cause major issues if there is a PermissionIssue due to something else.

except (FileNotFoundError, PermissionError) as e:

Reading through minio/minio#4434 it seems that calling head_object is not supported for prefixes.
I.e. this will only succeed if an actual object exists at the key dvc/zntrack-examples. If it’s just a prefix (i.e., a “folder”), head_object will raise an error.

This I can replicate using boto3

import boto3
from botocore import UNSIGNED
from botocore.config import Config
from botocore.exceptions import ClientError

# Create an anonymous S3 client
s3 = boto3.client(
    's3',
    endpoint_url='https://minio.icp.uni-stuttgart.de',
    config=Config(signature_version=UNSIGNED)
)

bucket = 'public'
key = 'dvc/zntrack-examples/files/md5/f7/b8a247465f1aac988675ea086fd76a' # works
key = 'dvc/zntrack-examples' # returns An error occurred (403) when calling the HeadObject operation: Forbidden

try:
    response = s3.head_object(Bucket=bucket, Key=key)
    print(response)
except ClientError as e:
    print(e)

I am not certain if this is only with minio handling things differently. Either way it would be great if there is an alternative or a fallback option, e.g. to list_objects_v2 instead.
@PythonFZ
Copy link
Author
PythonFZ commented May 22, 2025 •
edited
Loading

Issue was caused by some caching issue in the nginx setup
The issue remains without any setup in between using the docker container directly

s3 = boto3.client(
    's3',
    endpoint_url='https://129.69.120.126:5005',
    config=Config(signature_version=UNSIGNED),
    verify=False
)

Using a different minio instance I can not reproduce this issue. I am therefore closing this now.

@PythonFZ PythonFZ reopened this May 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@PythonFZ
0