feat(config): ipfs add and Import options for controling UnixFS DAG Width
#10774
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2563b31 to
a0628f2
Compare
This introduces support for setting DAG-shaping options that control the maximum number of links in that a DAG node can have. It is supported in two ways: first, via `Import` configuration options; second, via `--max-links` and `--max-hamt-fanout` options `ipfs add`. The resulting DAGs will respect MaxLinks (for files and basic directories) and MaxHAMTFanout for HAMT directories. When options are unset, the previous Kubo defaults will apply.
21cc49d to
b015c71
Compare
|
Based on ipfs/boxo#906. It needs some sharness tests I guess. I have tested manually at it seems to work, and I have ensured that not setting the new options does not break anything in terms of previous behaviour. |
lidel
reviewed
Apr 8, 2025
ipfs add and Import options for controling UnixFS DAG Width
moved `Internal.UnixFSShardingSizeThreshold` to `Import.UnixFSHAMTDirectorySizeThreshold` and included it in profiles
placeholder name based on year and kubo release
|
Looking good, noting remaining work for me next week:
|
basic smoke test confirming UnixFSFileMaxLinks sets to 'ipfs add' defaults and informs the DAG shape
keeping old name to minimize noise, just document pre-existing defaults
lets make the fanout match the max links from files and rename profile to `-wide` this will make it easier to discuss in ipfs/specs#499
thr 8000 eshold exists only as a global flag, so it was already set, but we were missing fanout. this applies both, for global consistency
d01209d to
104ede9
Compare
if cfg.Internal.UnixFSShardingSizeThreshold is set and cfg.Import.UnixFSHAMTDirectorySizeThreshold is not, assign old value to new field and print ERROR urging user to migrate this way we wont break any existing CI etc
lidel
approved these changes
Apr 15, 2025
There was a problem hiding this comment.
Thank you @hsanjuan
- added tests to
./test/cli/add_test.go, looking good - moved preexisting
Internal.UnixFSShardingSizeThresholdtoImport.UnixFSHAMTDirectorySizeThreshold
Merging to test it over Easter, if no issues we will ship this in Kubo 0.35.0-rc1
I'll document these in ipfs/specs#499
lidel
added a commit
to mishmosh/specs-1
that referenced
this pull request
Apr 15, 2025
Import.* config params for controlling DAG width were added in: ipfs/kubo#10774
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces support for setting DAG-shaping options that control the maximum number
of links in that a DAG node can have.
It is supported in two ways: first, via
Importconfiguration options; second, via--max-linksand--max-hamt-fanoutoptionsipfs add.The resulting DAGs will respect MaxLinks (for files and basic directories) and
MaxHAMTFanout for HAMT directories. When options are unset, the previous Kubo
defaults will apply.