8000 Importers overhaul by BielStela · Pull Request #1228 · Vizzuality/landgriffon · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Importers overhaul #1228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 57 commits into
base: dev
Choose a base branch
from
Open

Importers overhaul #1228

wants to merge 57 commits into from

Conversation

BielStela
Copy link
Contributor

General description

Please write a description. If the PR is hard to understand, provide a quick explanation of the code.

Designs

Link to the related design prototypes (if applicable)

Testing instructions

Provide minimal instructions on how to test this PR.

  • Apart from the added feature / bug fix, check overall performance, styling...

Checklist before merging

  • Branch name / PR includes the related Jira ticket Id.
  • Tests to check core implementation / bug fix added.
  • All checks in Continuous Integration workflow pass.
  • Feature functionally tested by reviewer(s).
  • Code reviewed by reviewer(s).
  • Documentation updated (README, CHANGELOG...) (if required)

@vercel Vercel
Copy link
vercel bot commented Feb 17, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

3 Skipped Deployments
Name Status Preview Comments Updated (UTC)
landgriffon-client ⬜️ Ignored (Inspect) Visit Preview Jun 13, 2025 4:34pm
landgriffon-cookie-traceability ⬜️ Ignored (Inspect) Visit Preview Jun 13, 2025 4:34pm
landgriffon-marketing ⬜️ Ignored (Inspect) Visit Preview Jun 13, 2025 4:34pm

@BielStela BielStela marked this pull request as draft February 17, 2025 09:42
alexeh
alexeh reviewed Feb 18, 2025
View reviewed changes
data/settings.py
indicator_coefficient_config: str


settings = Settings()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing this would break the current set up, can we leave it for now? (In case we aim to have this PoC merged until we can fully switch to a new, shiny, performant, amazing data-importing approach?

@BielStela BielStela force-pushed the importers-overhaul branch from 00d0b72 to faffc82 Compare April 23, 2025 09:01
BielStela added 26 commits May 12, 2025 11:29
@BielStela
Initial draft for new importer
ee267db 
Materials and indicators importer.
- Removes the dependency of hardcoded csv files in the source code
- Adds schemas for such jsons
- Allows different material and indicator sources
@BielStela
Add base commands
7d5ac18
@BielStela
Loader and tidyup
c98a470
@BielStela
Fix psycopg package, format and improve README.md
20a552a
@BielStela
add pre commit dev dep
f61c63f
@BielStela
add mypy
f15c7dd
@BielStela
Fix banner
cb1de61
@BielStela
Updates linter
7aee968
@BielStela
add main pipe
c43b7f9
@BielStela
adds preprocessing kedro module
755f7fc
@BielStela
Cleanup and GADM pipeline
124ec7e
@BielStela
(wip) gadm to h3
cc51cdf
@BielStela
Fix intermediate generation
d1a5fa3
@BielStela
Split to 2 tables
502da9f
@BielStela
Closer to the final shape of admin and geo region tables
ce4be8b
@BielStela
Switch to gadm partitioned by level dataset
10000
c3da5df
@BielStela
More gadm mess: join all levels
6203c06
@BielStela
Add uuid column
1eddabc
@BielStela
lasy meaningful name and gadm id for each level
98d7cff
@BielStela
Reorganize configuration
ca78830
@BielStela
organize better
ee48ea7
@BielStela
Replace custom function by build in coalesce
54da1f4
@BielStela
Add parent_id column logic
45cb9dc
@BielStela
fixes tests and normalizes column names to snake_case
943aec4
@BielStela
Add mpath calculation
571e59c
@BielStela
Update docstring
e9fbf76
@BielStela BielStela force-pushed the importers-overhaul branch from 1f10b9e to b0f7196 Compare May 12, 2025 09:29
@BielStela BielStela self-assigned this May 23, 2025
@BielStela BielStela marked this pull request as ready for review May 23, 2025 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexeh alexeh alexeh left review comments

Assignees

@BielStela BielStela

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@BielStela @alexeh
0