You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a rest API source that I funnel to a fabric lakehouse files section. I run it from a fabric python notebook. It extracts data successfully, However when I kill the session and run it again it can't find the init file. The init file is created but is empty, hence the error. It works if I re-run it in the same session.
Pipeline fortnox load step completed in 0.69 seconds
1 load package(s) were loaded to destination filesystem and into dataset customers_data
The filesystem destination used file:///synfs/lakehouse/default/Files/dlthub/fortnox location to store data
Load package 1746775954.3512297 is LOADED and contains no failed jobs
When running it a second time:
PipelineStepFailed: Pipeline execution failed at stage load when processing package 1746776041.8751009 with exception:
<class 'FileNotFoundError'>
[Errno 2] No such file or directory: '/synfs/lakehouse/default/Files/dlthub/fortnox/customers_data/_dlt_loads/init'
dlt version
1.10.0
Describe the problem
I have a rest API source that I funnel to a fabric lakehouse files section. I run it from a fabric python notebook. It extracts data successfully, However when I kill the session and run it again it can't find the init file. The init file is created but is empty, hence the error. It works if I re-run it in the same session.
Pipeline fortnox load step completed in 0.69 seconds
1 load package(s) were loaded to destination filesystem and into dataset customers_data
The filesystem destination used file:///synfs/lakehouse/default/Files/dlthub/fortnox location to store data
Load package 1746775954.3512297 is LOADED and contains no failed jobs
When running it a second time:
PipelineStepFailed: Pipeline execution failed at stage load when processing package 1746776041.8751009 with exception:
<class 'FileNotFoundError'>
[Errno 2] No such file or directory: '/synfs/lakehouse/default/Files/dlthub/fortnox/customers_data/_dlt_loads/init'
`
%pip install dlt
%pip install "dlt[az]"
%pip install "dlt[filesystem]"
import dlt
from dlt.sources.rest_api import rest_api_source
dlt.secrets["fortnox_api_token"] = access_token
source = rest_api_source({
"client": {
"base_url": base_url,#"https://api.fortnox.se/3",
"auth": {
"token": dlt.secrets["fortnox_api_token"],
},
"headers": {
"Content-Type": "application/json"
},
},
"resources": [
# Resource for fetching customer data
{
"name": resource_name,#"customers", # Resource name
"endpoint": {
"path": endpoint #"customers", # Dynamic path
},
}
})
from dlt.destinations import filesystem
bucket_url = "/lakehouse/default/Files/dlthub/fortnox/"
Define the pipeline
pipeline = dlt.pipeline(
pipeline_name="fortnox", # Pipeline name
destination=filesystem(
bucket_url= bucket_url #"/lakehouse/default/Files/fortnox/tmp"
),
dataset_name=f"{resource_name}_data", # Dataset name
dev_mode=False
# progress="enlighten", # Progress display method
)
Run the pipeline
load_info = pipeline.run(
source,
loader_file_format="parquet"
)
print(load_info)
print(dir(load_info))
print(load_info.load_packages)`
Expected behavior
Expected behaviour is that the init file should be populated.
Steps to reproduce
Run a fabric lakehouse
Operating system
Other
Runtime environment
Other
Python version
3.11
dlt data source
rest API, fortknox
dlt destination
Filesystem & buckets
Other deployment details
Fabric notebook
Additional information
No response
The text was updated successfully, but these errors were encountered: