Tags: ordo-one/package-benchmark
Tags
fix(patch): Rename 'Shared' target to 'BenchmarkShared' (#324) 'Shared' is too much of a popular word for a target name. Apps that already have a target named 'Shared' won't be able to use that target in combination with this package. This PR unblocks us to update our Package-Benchmark without having to rename our 'Shared' target. This PR also adds '.git' to repo URLs since that makes sure we avoid a Xcode-x-SwiftPM bug. After we reported this a bunch of months ago, NIO team have also been updating their packages with this. Can't find a link to that discussion but you can at least see that packages like SwiftNIO are using '.git' ([link](https://github.com/apple/swift-nio/blob/ea8703616e4cbc50fb6a41038cbdb5a53d2fec89/Package.swift#L597)).
feat(minor): Enable Swift 6 mode and support specifying units per met… …ric (#309) ## Description Swift 6 benchmark targets should now be possible with: ```swift let benchmarks: @sendable () -> Void = { ... } ``` Make it possible to specify output units for the text output for a given metric by specifying it in the configuration. E.g. ```swift Benchmark.defaultConfiguration.units = [.peakMemoryResident: .mega, .peakMemoryVirtual: .giga] ``` Also add the ability to override the time units from the command line using `--time-units`. E.g. ```bash swift package benchmark --time-units microseconds ``` This update also displays overflowing numeric values in scientific notation in the text output: ``` ╒═══════════════════════════════════════════════════════════════════════════╤═══════════╤═══════════╤═══════════╤═══════════╤═══════════╤═══════════╤═══════════╤═══════════╕ │ Test │ p0 │ p25 │ p50 │ p75 │ p90 │ p99 │ p100 │ Samples │ ╞═══════════════════════════════════════════════════════════════════════════╪═══════════╪═══════════╪═══════════╪═══════════╪═══════════╪═══════════╪═══════════╪═══════════╡ │ Samples:All metrics, full concurrency, async (ns) * │ 8.08e+07 │ 8.34e+07 │ 8.38e+07 │ 8.40e+07 │ 8.41e+07 │ 8.46e+07 │ 8.49e+07 │ 183 │ ├───────────────────────────────────────────────────────────────────────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┤ │ Samples:Counter, custom metric thresholds (ns) * │ 2375 │ 2417 │ 2417 │ 2459 │ 2501 │ 2709 │ 4334 │ 4753 │ ├───────────────────────────────────────────────────────────────────────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┤ │ Samples:Extended + custom metrics (ns) * │ 1875 │ 2000 │ 2000 │ 2041 │ 2167 │ 2667 │ 8750 │ 707 │ ├───────────────────────────────────────────────────────────────────────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┤ │ Samples:Extended metrics (ns) * │ 1750 │ 1875 │ 1875 │ 1917 │ 1958 │ 2209 │ 3250 │ 705 │ ╘═══════════════════════════════════════════════════════════════════════════╧═══════════╧═══════════╧═══════════╧═══════════╧═══════════╧═══════════╧═══════════╧═══════════╛ ``` Also updates documentation. Addresses: #306 and #237 #293 #277 #258 --------- Co-authored-by: Axel Andersson <axel@ordo.one> Co-authored-by: dimlio <122263440+dimlio@users.noreply.github.com>
fix(patch): Use `String(reflecting: error)` for printing errors (#290) ## Description A few (mostly server side) packages like PostgresNIO have a generic `description` in their errors, while providing the full error info in `debugDescription`. Using `String(reflecting: error)` we can make sure the `debugDescription` is preferred when transforming the error into an `String`. Sample error: ``` Benchmark MyBenchmark failed with PostgresDecodingError – Generic description to prevent accidental leakage of sensitive data. For debugging details, use String(reflecting: error). ``` ## How Has This Been Tested? Please describe the tests that you ran to verify your changes. ## Minimal checklist: - [x] I have performed a self-review of my own code - [ ] I have added `DocC` code-level documentation for any public interfaces exported by the package - [ ] I have added unit and/or integration tests that prove my fix is effective or that my feature works Co-authored-by: Joakim Hassila <jocke@ordo.one>
feat: Add explicit support for static thresholds (#284) ## Description Adding support for new thresholds subcommands - used for storing/reading/checking against static saved thresholds - by default output into `Thresholds` in the current directory, but location can be specified with `--path` too. To clarify, the existing thresholds in the benchmark code configuration is really _threshold tolerances_ - unfortunately difficult to rename now, but basically the new `thresholds` commands operate on static thresholds, while the code defined _tolerances_ continue to be applied to both dynamic and static benchmark checking as outlined in the documentation. # New thresholds support ## Update static thresholds (in the Thresholds/ directory) ```bash swift package --allow-writing-to-package-directory benchmark thresholds update ``` ```bash Build of product 'BenchmarkTool' complete! (2.44s) Build complete! Building BenchmarkTool in release mode... Building benchmark targets in release mode for benchmark run... Building URLBenchmarks Building StringBenchmarks Building PredicateBenchmarks Building JSONBenchmarks Building InternationalizationBenchmarks Building FormattingBenchmarks Building EssentialsBenchmarks Building DataIOBenchmarks Building Base64Benchmarks Building AttributedStringBenchmarks ================== Running Benchmarks ================== 100% [------------------------------------------------------------] ETA: 00:00:00 | AttributedStringBenchmarks:createLongString-swift 100% [------------------------------------------------------------] ETA: 00:00:00 | AttributedStringBenchmarks:createManyAttributesString-swift 100% [------------------------------------------------------------] ETA: 00:00:00 | AttributedStringBenchmarks:enumerateAttributes-swift 100% [------------------------------------------------------------] ETA: 00:00:00 | AttributedStringBenchmarks:enumerateAttributesSlice-swift .... 100% [------------------------------------------------------------] ETA: 00:00:00 | URLBenchmarks:URLComponents-SetEncodableComponents 100% [------------------------------------------------------------] ETA: 00:00:00 | URLBenchmarks:URLComponents-SetEncodableQueryItems 100% [------------------------------------------------------------] ETA: 00:00:00 | URLBenchmarks:URLComponents-SetQueryItems ================== Writing Thresholds ================== Writing to ./Thresholds/PredicateBenchmarks.predicateKeypathComputedPropertyCondition.p90.json Writing to ./Thresholds/PredicateBenchmarks.predicateTrivialCondition.p90.json Writing to ./Thresholds/StringBenchmarks.read-utf8.p90.json ... ``` # Reading thresholds ```bash package benchmark thresholds read ``` ```bash ... Build complete! Building BenchmarkTool in release mode... Building benchmark targets in release mode for benchmark run... Building URLBenchmarks Building StringBenchmarks Building PredicateBenchmarks Building JSONBenchmarks Building InternationalizationBenchmarks Building FormattingBenchmarks Building EssentialsBenchmarks Building DataIOBenchmarks Building Base64Benchmarks Building AttributedStringBenchmarks Reading thresholds from "Thresholds" utf16-decode ╒══════════════════════════════════════════╤══════════════════════╤══════════════════════╤══════════════════════╕ │ Metric │ Threshold .p90 │ Allowed % │ Allowed Δ │ ╞══════════════════════════════════════════╪══════════════════════╪══════════════════════╪══════════════════════╡ │ Time (total CPU) │ 3208473041 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Time (wall clock) │ 3211281125 │ 0.0 │ 0 │ ╘══════════════════════════════════════════╧══════════════════════╧══════════════════════╧══════════════════════╛ Canada-encodeToJSON ╒══════════════════════════════════════════╤══════════════════════╤══════════════════════╤══════════════════════╕ │ Metric │ Threshold .p90 │ Allowed % │ Allowed Δ │ ╞══════════════════════════════════════════╪══════════════════════╪══════════════════════╪══════════════════════╡ │ Time (total CPU) │ 62029823 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Throughput (# / s) │ 17 │ 0.0 │ 0 │ ╘══════════════════════════════════════════╧══════════════════════╧══════════════════════╧══════════════════════╛ utf32-decode ╒══════════════════════════════════════════╤══════════════════════╤══════════════════════╤══════════════════════╕ │ Metric │ Threshold .p90 │ Allowed % │ Allowed Δ │ ╞══════════════════════════════════════════╪══════════════════════╪══════════════════════╪══════════════════════╡ │ Time (wall clock) │ 3341120667 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Time (total CPU) │ 3341006500 │ 0.0 │ 0 │ ╘══════════════════════════════════════════╧══════════════════════╧══════════════════════╧══════════════════════╛ allocationsForAutoupdatingCurrentCalendar ╒══════════════════════════════════════════╤══════════════════════╤══════════════════════╤══════════════════════╕ │ Metric │ Threshold .p90 │ Allowed % │ Allowed Δ │ ╞══════════════════════════════════════════╪══════════════════════╪══════════════════════╪══════════════════════╡ │ Malloc (total) │ 4000000 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Time (total CPU) │ 1360681292 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Memory (resident peak) │ 12009472 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Throughput (# / s) │ 1 │ 0.0 │ 0 │ ╘══════════════════════════════════════════╧══════════════════════╧══════════════════════╧══════════════════════╛ getAttributeSubrange-swift ╒══════════════════════════════════════════╤══════════════════════╤══════════════════════╤══════════════════════╕ │ Metric │ Threshold .p90 │ Allowed % │ Allowed Δ │ ╞══════════════════════════════════════════╪══════════════════════╪══════════════════════╪══════════════════════╡ │ Throughput (# / s) │ 1479 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Time (wall clock) │ 730111 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Time (total CPU) │ 728575 │ 0.0 │ 0 │ ╘══════════════════════════════════════════╧══════════════════════╧══════════════════════╧══════════════════════╛ write-regularFile ╒══════════════════════════════════════════╤══════════════════════╤══════════════════════╤══════════════════════╕ │ Metric │ Threshold .p90 │ Allowed % │ Allowed Δ │ ╞══════════════════════════════════════════╪══════════════════════╪══════════════════════╪══════════════════════╡ │ Time (total CPU) │ 2656255 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Syscalls (total) │ 5 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Malloc (total) │ 10 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Throughput (# / s) │ 291 │ 0.0 │ 0 │ ├──────────────────────────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤ │ Time (wall clock) │ 3737599 │ 0.0 │ 0 │ ╘══════════════════════════════════════════╧══════════════════════╧══════════════════════╧══════════════════════╛ ``` # Check static thresholds ```bash swift package benchmark thresholds check --filter base64-decode-1MB-fromString-noOptions ``` ```bash ... Build of product 'BenchmarkTool' complete! (2.45s) Build complete! Building BenchmarkTool in release mode... Building benchmark targets in release mode for benchmark run... Building URLBenchmarks Building StringBenchmarks Building PredicateBenchmarks Building JSONBenchmarks Building InternationalizationBenchmarks Building FormattingBenchmarks Building EssentialsBenchmarks Building DataIOBenchmarks Building Base64Benchmarks Building AttributedStringBenchmarks ================== Running Benchmarks ================== 100% [------------------------------------------------------------] ETA: 00:00:00 | Base64Benchmarks:base64-decode-1MB-fromString-noOptions Reading thresholds from "Thresholds" Checking ["Base64Benchmarks:base64-decode-1MB-fromString-noOptions"] =========================================================================================== Deviations worse than threshold for Base64Benchmarks:base64-decode-1MB-fromString-noOptions =========================================================================================== ╒══════════════════════════════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╕ │ Throughput (# / s) (#, Δ) │ p90 threshold │ Current_run │ Difference Δ │ Threshold Δ │ ╞══════════════════════════════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╡ │ p90 │ 332 │ 322 │ 10 │ 0 │ ╘══════════════════════════════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╛ ╒══════════════════════════════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╕ │ Throughput (# / s) (#, %) │ p90 threshold │ Current_run │ Difference % │ Threshold % │ ╞══════════════════════════════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╡ │ p90 │ 332 │ 322 │ 3 │ 0 │ ╘══════════════════════════════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╛ ============================================================================================ Deviations better than threshold for Base64Benchmarks:base64-decode-1MB-fromString-noOptions ============================================================================================ ╒══════════════════════════════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╕ │ Time (total CPU) (μs, Δ) │ p90 threshold │ Current_run │ Difference Δ │ Threshold Δ │ ╞══════════════════════════════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╡ │ p90 │ 3217 │ 3111 │ -106 │ 0 │ ╘══════════════════════════════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╛ ╒══════════════════════════════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╕ │ Time (total CPU) (μs, %) │ p90 threshold │ Current_run │ Difference % │ Threshold % │ ╞══════════════════════════════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╡ │ p90 │ 3217 │ 3111 │ -3 │ 0 │ ╘══════════════════════════════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╛ The baseline 'Current_run' is WORSE than the defined thresholds. error: benchmarkThresholdRegression ```
PreviousNext