utxocache: replace mapslice with concurrent-swiss-map #2346
Conversation
Pull Request Test Coverage Report for Build 14052973796Details
💛 - Coveralls |
|
Mostly curious: How does the concurrent-swiss-map in the library you used differ from the native Swiss Tables that are now implemented in Go 1.24? |
What about allocs/op? |
| m := ms.makeNewMap(totalEntryMemory) | ||
| m[op] = entry | ||
| func (ms *csMap) size() int { | ||
| return calculateRoughMapSize(ms.length(), bucketSize) |
There was a problem hiding this comment.
I think we also need to modify calculateRoughMapSize, as it makes assumptions w.r.t the current runtime implementation. With the introduction of Swiss maps in Go 1.24, this is likely out of date anyway (results in incorrect estimates).
There was a problem hiding this comment.
This right?
https://go.dev/blog/swisstable
Maybe the map calculation might be easier now based on how the blog described it. We'll see...
It is basically same because both project based by dolthub-swissmap. https://www.reddit.com/r/golang/comments/15jl2x7/a_highperformance_threadsafe_generic_concurrent/ |
|
Concurrent reads/writes are not much of a problem for performance since the block validation I think getting rid of mapslice is good since it's ugly. But I don't think the external dep makes Since the builtin map has multiple swiss tables, maybe we can just replace the entire mapslice |
|
@kcalvinalvin To use swissmap, it need to update go.mod version to 1.24. Also, using concurrentSwissMap is a better option than default swissmap since it avoids global locks. If needed, I can also include benchmarks for the 1.24 builtin swissmap. |
Fair point but I think avoiding an external dependency is better than avoiding global locks. |
Change Description
Currently,
utxocacheusesmapsliceinstead of a single largemap. This seems to be necessary because Go's built-inmapcan consume a significant amount of memory when scaled.However, I suggest using ConcurrentSwissMap as an alternative, which offers simple implementation and better performance in concurrent reads.
Benchmark
Environment
GOOS: linuxGOARCH: amd64CPU: AMD EPYC 7571PKG: github.com/btcsuite/btcd/blockchainCsmap outperforms the lock-based mapslice by over 6x in concurrent reads.
You can verify the performance using the mapslice and concurrentSwissMap benchmarks.
Steps to Test
Steps for reviewers to follow to test the change.
Pull Request Checklist
Testing
Code Style and Documentation
📝 Please see our Contribution Guidelines for further guidance.