8000 Incompatibility Airflow 3.0.0 · Issue #2601 · dlt-hub/dlt · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Incompatibility Airflow 3.0.0 #2601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
darleo opened this issue May 5, 2025 · 2 comments
Open

Incompatibility Airflow 3.0.0 #2601

darleo opened this issue May 5, 2025 · 2 comments
Labels
community This issu 8000 e came from slack community workspace dependency update

Comments

@darleo
Copy link
darleo commented May 5, 2025

dlt version

1.10

Describe the problem

There seems to be an incompatibility between the freshly released airflow==3.0.0 and dlt=1.10:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/dlt/helpers/airflow_helper.py", line 21, in <module>
    from airflow.operators.dummy import DummyOperator
ModuleNotFoundError: No module named 'airflow.operators.dummy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/airflow/dags/01_load_staging.py", line 7, in <module>
    from dlt.helpers.airflow_helper import PipelineTasksGroup
  File "/home/airflow/.local/lib/python3.12/site-packages/dlt/helpers/airflow_helper.py", line 24, in <module>
    raise MissingDependencyException("Airflow", ["apache-airflow>=2.5"])
dlt.common.exceptions.MissingDependencyException: 
You must install additional dependencies to run Airflow. If you use pip you may do the following:

pip install "apache-airflow>=2.5"

Expected behavior

If this is indeed a problem caused by changes of the recent airflow release, I expect the pyproject.toml to specify an upper bound for apache-airflow.

Steps to reproduce

Install airflow=3.0.0 and dlt=1.10 and create a pipeline using the airflow_helper.

Operating system

macOS

Runtime environment

Airflow

Python version

3.12

dlt data source

No response

dlt destination

No response

Other deployment details

No response

Additional information

No response
@anuunchin anuunchin added community This issue came from slack community workspace dependency update labels May 5, 2025
@sh-rp
Copy link
Collaborator
sh-rp commented May 6, 2025

It looks like we are using an outdated import (which has been marked deprecated since airflow 2.4): https://stackoverflow.com/questions/74345802/getting-error-in-visual-code-studio-importerror-cannot-import-name-dummyopera

Possibly we can just update this and it will work with airflow 2.5+. We should verify that it works with 3.0.0 too and maybe set a dependency of airflow<4.0.0 if it does.

@rudolfix
Copy link
Collaborator

airflow 3.0 is very different from 2.x so this is not a dependency upgrade. what we'll do right away is to remove deprecated imports so all versions until 2.10.x work

@rudolfix rudolfix mentioned this issue May 12, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community This issue came from slack community workspace dependency update
Projects
dlt core library
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sh-rp @rudolfix @darleo @anuunchin
0