rust: configurable compression level and threads #1394
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chriso
commented
Jun 12, 2025
| // Enable multithreaded encoding on non-WASM targets. | ||
| #[cfg(not(target_arch = "wasm32"))] | ||
| enc.set_parameter(zraw::CParameter::NbWorkers(num_cpus::get_physical() as u32))?; | ||
| enc.set_parameter(zraw::CParameter::NbWorkers(compression_threads))?; |
There was a problem hiding this comment.
A value of 1 means one background thread. Zero disables extra threads entirely. See https://github.com/gyscos/zstd-rs/blob/229054099aa73f7e861762f687d7e07cac1d9b3b/zstd-safe/src/lib.rs#L2228
| lz4::EncoderBuilder::new() | ||
| .level(compression_level) |
There was a problem hiding this comment.
With both lz4 and zstd, level=0 means use the default compression level. References:
|
Thanks for the quick review @bennetthardwick! Out of interest, how often is the crate updated? I'm happy to submit a PR to bump the |
|
That'd be great - I should have asked you to chuck it in this one. We don't have any specific release cycles, we'll just put out a new release as things accumulate. |
|
Thanks @bennetthardwick. See #1395 |
bennetthardwick
pushed a commit
that referenced
this pull request
Jun 12, 2025
### Changelog This PR bumps the Rust crate version from `0.18.0` to `0.19.0` ### Docs None ### Description Changes since 0.18.0: * #1394 * #1391 For context, see #1394 (comment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog
This PR adds two new write options:
compression_level: u32– compression level to use for zstd/lz4, or zero to use the default compression level.compression_threads: u32- number of threads to use for compression, or zero to disable multithreaded compression.Docs
None? Let me know if I need to update docs somewhere.
Description
Prior to this PR, it wasn't possible to set the compression level; the default lz4 and zstd compression levels were used. Now, it's possible to set the lz4 and zstd compression levels.
Prior to this PR, non-wasm targets were forced to use multithreaded zstd compression (reference). Now, it's possible to disable multithreaded compression, or to use a different number of compression threads.
lz4 supports multithreaded compression as of v1.10.0 (see [1] and [2]), but this feature doesn't seem to be accessible from the lz4 / lz4-sys crates. For now, the
compression_leveloption is ignored when using lz4 compression. Let me know if we should print a warning or fail.I made these write options rather than parameterizing the
Compressionenum variants, because these particular options are orthogonal to the compression algorithm, and because they're only applicable to writers.