[LANDGRIF-1262] Refactor (removes) tiff_to_h3_table.py to something more modular
#899
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ |
2ee2ae1 to
b76e531
Compare
2fff34a to
8a3eb61
Compare
5fe2fce to
29388e4
Compare
tiff_to_h3_table.py to something more modulartiff_to_h3_table.py to something more modular
alexeh
approved these changes
Mar 7, 2023
e21e078 to
10e68ed
Compare
…porter a bit. Renames file Remove unused water footprint ingestion. Also it is not found in the current URL Clean unintended splited string Fix docker compose call in data.sh to new version and add host network to compose remove commented code Removes unnecessary branch and always raise when check fails Fix typos Remove coment Fix query string injection and documentation of the script Add some defensive raises in case the ingestion is not going at expected Replace all call to tif_folder to new script Also adds some necessary shushes to all the commands, the output was getting crazy. Add some logging, remove unused connection pool (not needed) and adds h3 dependency Makes raster to h3 parallel Uses multiprocessing for the conversion to h3 dataframes. Removes the usages of opened readers with ExitStack because are not serializable into pickle thus not allowing multiprocessing on a list of DatasetReaders. Stills slow in the CONCAT or JOIN and the DB ingestion EMBARASSING commit: I was deleting the table in another function 😭😭😭 WIP More refactor and try find out why table is not created Change conenction manager to ConnectinPool, minor fixes here and there to make it WORK 🥳 Finish raster_to_h3 script 🙌💃. (It will need fixes everywhere) Updates rasterio and h3ronpy deps WIP: Add new raster to h3 file Delete unneded files
10e68ed to
cc09e6c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
General description
Rewrite of the raster to h3 ingestion. Highlights:
psycopg3 apiTesting instructions
Known issues & a bit of Discussion
The process has 3 bottlenecks currently [seconds running SPAM ingestion]:
DataFrame.join(list[DataFrame])[57s]So for the SPAM case, which has 51 rasters to ingest (the worst case), the old script takes ~90s and the new approach takes 116s.
For datasets with a single raster it is faster.
All this has to be put into perspective that the longest time in the ingestion is taken by the deforestation (hansen + ghg) part which is 99.9% downloading and preprocessing with gdal and the resulting raster ingested is as small as a single SPAM one.
Thus, the next big step should be move the postprocessed results of deforestation to the cloud.
And why you may ask? I find it a little bit less cluttered and easy to follow to apply modifications safely.
PD.: I almost forget, this PR also removes unused files in the data folder which are quite annoying to have around
Checklist before merging