Closed
Description
Hi!
Thanks for building this awesome framework!
We are running Luigi with Python 3.6 and when we tried updating our Luigi from 2.7.5 to 2.8.0 we encountered the following issue:
Traceback (most recent call last):
File "/luigienv/lib/python3.6/site-packages/luigi/worker.py", line 199, in run
new_deps = self._run_get_new_deps()
File "/luigienv/lib/python3.6/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
task_gen = self.task.run()
File "/app/plumbing/luigi_warehouse/gsheet_to_redshift.py", line 167, in run
client.put(local_path, s3_path)
File "/luigienv/lib/python3.6/site-packages/luigi/contrib/s3.py", line 266, in put
self.put_multipart(local_path, destination_s3_path, **kwargs)
File "/luigienv/lib/python3.6/site-packages/luigi/contrib/s3.py", line 300, in put_multipart
self.s3.meta.client.upload_fileobj(
File "/luigienv/lib/python3.6/site-packages/luigi/contrib/s3.py", line 165, in s3
logger.error(e.message)
AttributeError: 'TypeError' object has no attribute 'message'
The issue is that the file luigi/contrib/s3.py uses a call to TypeError member e.message which is only available in Python 2 as opposed to e.args which is available in both Python 2 and Python 3.
Here is the code snippet in luigi/contrib/s3.py from Luigi release 2.8.0 causing the issue (in fact, it seems it's the same code for all releases >= 2.7.6):
# At this stage, if no credentials provided, boto3 would handle their resolution for us
# For finding out about the order in which it tries to find these credentials
# please see here details
# http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials
if not (aws_access_key_id and aws_secret_access_key):
logger.debug('no credentials provided, delegating credentials resolution to boto3')
try:
self._s3 = boto3.resource('s3',
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
aws_session_token=aws_session_token,
**options)
except TypeError as e:
logger.error(e.message)
if 'got an unexpected keyword argument' in e.message:
raise DeprecatedBotoClientException(
"Now using boto3. Check that you're passing the correct arguments")
raiseTo fix this issue, I propose to use e.args[0] instead of e.message like so:
except TypeError as e:
logger.error(e.args[0])
if 'got an unexpected keyword argument' in e.args[0]:
raise DeprecatedBotoClientException(
"Now using boto3. Check that you're passing the correct arguments")
raiseAfter applying this patch locally and re-running, we see the exception was thrown as expected:
> "Now using boto3. Check that you're passing the correct arguments")
E luigi.contrib.s3.DeprecatedBotoClientException: Now using boto3. Check that you're passing the correct arguments
/luigienv/lib/python3.6/site-packages/luigi/contrib/s3.py:168: DeprecatedBotoClientException
Metadata
Metadata
Assignees
Labels
No labels