hsbench is an S3 compatable benchmark originally based on wasabi-tech/s3-benchmark. While there are already several S3 compatable benchmark suites available, each has various tradeoffs. What seemed to be missing was a simple yet extremely fast benchmark that could easily be scripted into orchestration frameworks to run tests at scale. hsbench tries to fill that niche. The current release of hsbench is considered to be alpha level software and may contain bugs.
hsbench tries to improve on the original Wasabi s3-benchmark in the following ways:
- Threads can distribute IOS across an arbitrary number of buckets.
- Tests can be run individually and externally coordinated across multiple clients.
- Intermediate results are logged periodically at user-defined intervals.
- Min/avg/max/percentile latency results are included.
- Test length can be limited either by duration or maximum number of objects.
- Object prefixes can be set to test large object names (12 bytes reserved for uniqueness)
- Bucket/Object prefixes can be used to allow multiple clients to target the same buckets
- You can use hsbench to delete buckets in parallel:
./hsbench -a ... -s ... -u http://... -m cx -t 32 -bl "bucket1 bucket2 ..."(-t is the number of threads)
- hsbench does not currently support multiple AWS endpoints
- hsbench has no built-in provisions for making graphs
- hsbench does not yet support mixed IO workloads
- hsbench is still in alpha and options/output may change at any moment
To leverage this tool, the following prerequisites apply:
- Git development environment
- Go version 1.7 or newer
- Access to an S3 compatable storage service (AWS, Ceph, etc)
The easiest way to install hsbench is to use go's built in github support:
go get github.com/vitalif/hsbench
Then just use ~/go/bin/hsbench.
If you want to patch and rebuild it, run go build or go install in the hsbench src directory:
$ cd ~/go/src/github.com/vitalif/hsbench
$ go build
$ ./hsbench --help
USAGE: ./hsbench [OPTIONS]
OPTIONS:
-a string
Access key
-b int
Number of buckets to distribute IOs across (default 1)
-bp string
Prefix for buckets (default "hotsauce_bench")
-d int
Maximum test duration in seconds <-1 for unlimited> (default 60)
-j string
Write JSON output to this file
-l int
Number of times to repeat test (default 1)
-m string
Run modes in order. See NOTES for more info (default "cxiplgdcx")
-mk int
Maximum number of keys to retreive at once for bucket listings (default 1000)
-n int
Maximum number of objects <-1 for unlimited> (default -1)
-o string
Write CSV output to this file
-op string
Prefix for objects
-r string
Region for testing (default "us-east-1")
-ri float
Number of seconds between report intervals (default 1)
-s string
Secret key
-t int
Number of threads to run (default 1)
-u string
URL for host with method prefix
-z string
Size of objects in bytes with postfix K, M, and G (default "1M")
NOTES:
- Valid mode types for the -m mode string are:
c: clear all existing objects from buckets (requires lookups)
x: delete buckets
i: initialize buckets
p: put objects in buckets
l: list objects in buckets
g: get objects from buckets
d: delete objects from buckets
These modes are processed in-order and can be repeated, ie "ippgd" will
initialize the buckets, put the objects, reput the objects, get the
objects, and then delete the objects. The repeat flag will repeat this
whole process the specified number of times.
- When performing bucket listings, many S3 storage systems limit the
maximum number of keys returned to 1000 even if MaxKeys is set higher.
hsbench will attempt to set MaxKeys to whatever value is passed via the
"mk" flag, but it's likely that any values above 1000 will be ignored.
Below is an example run of the benchmark using a 10s test duration, 10 threads, 10 buckets, and a 4K object size against a Ceph RadosGW backed by a single Ceph OSD running on an Intel P3700 NVMe device.
$ ./hsbench -a 3JZ0SVK94Z55OZU5J1N0 -s OdzEPyDDZ0ls1haDUu1NVWkJDcnG74Lb7XylfXRM -u http://127.0.0.1:7480 -z 4K -d 10 -t 10 -b 10
2019/08/19 11:21:06 Hotsauce S3 Benchmark Version 0.1
2019/08/19 11:21:06 Parameters:
2019/08/19 11:21:06 url=http://127.0.0.1:7480
2019/08/19 11:21:06 object_prefix=
2019/08/19 11:21:06 bucket_prefix=hotsauce_bench
2019/08/19 11:21:06 region=us-east-1
2019/08/19 11:21:06 modes=cxipgdx
2019/08/19 11:21:06 output=
2019/08/19 11:21:06 json_output=
2019/08/19 11:21:06 object_count=-1
2019/08/19 11:21:06 bucket_count=10
2019/08/19 11:21:06 duration=10
2019/08/19 11:21:06 threads=10
2019/08/19 11:21:06 loops=1
2019/08/19 11:21:06 size=4K
2019/08/19 11:21:06 interval=1.000000
2019/08/19 11:21:06 Running Loop 0 BUCKET CLEAR TEST
2019/08/19 11:21:06 Loop: 0, Int: TOTAL, Dur(s): 0.0, Mode: BCLR, Ops: 0, MB/s: 0.00, IO/s: 0, Lat(ms): [ min: 0.0, avg: 0.0, 99%: 0.0, max: 0.0 ], Slowdowns: 0
2019/08/19 11:21:06 Running Loop 0 BUCKET DELETE TEST
2019/08/19 11:21:06 Loop: 0, Int: TOTAL, Dur(s): 0.0, Mode: BDEL, Ops: 0, MB/s: 0.00, IO/s: 0, Lat(ms): [ min: 0.0, avg: 0.0, 99%: 0.0, max: 0.0 ], Slowdowns: 0
2019/08/19 11:21:06 Running Loop 0 BUCKET INIT TEST
2019/08/19 11:21:06 Loop: 0, Int: TOTAL, Dur(s): 0.0, Mode: BINIT, Ops: 10, MB/s: 0.00, IO/s: 944, Lat(ms): [ min: 7.9, avg: 9.3, 99%: 10.4, max: 10.4 ], Slowdowns: 0
2019/08/19 11:21:06 Running Loop 0 OBJECT PUT TEST
2019/08/19 11:21:07 Loop: 0, Int: 0, Dur(s): 1.0, Mode: PUT, Ops: 5209, MB/s: 20.35, IO/s: 5209, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.5, max: 8.4 ], Slowdowns: 0
2019/08/19 11:21:08 Loop: 0, Int: 1, Dur(s): 1.0, Mode: PUT, Ops: 5076, MB/s: 19.83, IO/s: 5076, Lat(ms): [ min: 1.2, avg: 2.0, 99%: 3.1, max: 42.3 ], Slowdowns: 0
2019/08/19 11:21:09 Loop: 0, Int: 2, Dur(s): 1.0, Mode: PUT, Ops: 4319, MB/s: 16.87, IO/s: 4319, Lat(ms): [ min: 1.3, avg: 2.3, 99%: 4.4, max: 58.0 ], Slowdowns: 0
2019/08/19 11:21:10 Loop: 0, Int: 3, Dur(s): 1.0, Mode: PUT, Ops: 4288, MB/s: 16.75, IO/s: 4288, Lat(ms): [ min: 1.3, avg: 2.3, 99%: 3.5, max: 63.1 ], Slowdowns: 0
2019/08/19 11:21:11 Loop: 0, Int: 4, Dur(s): 1.0, Mode: PUT, Ops: 4549, MB/s: 17.77, IO/s: 4549, Lat(ms): [ min: 1.3, avg: 2.2, 99%: 6.8, max: 57.9 ], Slowdowns: 0
2019/08/19 11:21:12 Loop: 0, Int: 5, Dur(s): 1.0, Mode: PUT, Ops: 4447, MB/s: 17.37, IO/s: 4447, Lat(ms): [ min: 1.3, avg: 2.2, 99%: 3.5, max: 58.8 ], Slowdowns: 0
2019/08/19 11:21:13 Loop: 0, Int: 6, Dur(s): 1.0, Mode: PUT, Ops: 4260, MB/s: 16.64, IO/s: 4260, Lat(ms): [ min: 1.3, avg: 2.4, 99%: 5.8, max: 58.4 ], Slowdowns: 0
2019/08/19 11:21:14 Loop: 0, Int: 7, Dur(s): 1.0, Mode: PUT, Ops: 5202, MB/s: 20.32, IO/s: 5202, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.1, max: 10.3 ], Slowdowns: 0
2019/08/19 11:21:15 Loop: 0, Int: 8, Dur(s): 1.0, Mode: PUT, Ops: 5213, MB/s: 20.36, IO/s: 5213, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.0, max: 13.3 ], Slowdowns: 0
2019/08/19 11:21:16 Loop: 0, Int: 9, Dur(s): 1.0, Mode: PUT, Ops: 5210, MB/s: 20.35, IO/s: 5210, Lat(ms): [ min: 1.2, avg: 1.9, 99%: 3.0, max: 9.6 ], Slowdowns: 0
2019/08/19 11:21:16 Loop: 0, Int: TOTAL, Dur(s): 10.0, Mode: PUT, Ops: 47783, MB/s: 18.66, IO/s: 4777, Lat(ms): [ min: 1.2, avg: 2.1, 99%: 3.4, max: 63.1 ], Slowdowns: 0
2019/08/19 11:21:16 Running Loop 0 OBJECT GET TEST
2019/08/19 11:21:17 Loop: 0, Int: 0, Dur(s): 1.0, Mode: GET, Ops: 1211, MB/s: 4.73, IO/s: 1211, Lat(ms): [ min: 0.6, avg: 0.9, 99%: 1.5, max: 2.0 ], Slowdowns: 0
2019/08/19 11:21:18 Loop: 0, Int: 1, Dur(s): 1.0, Mode: GET, Ops: 1182, MB/s: 4.62, IO/s: 1182, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.3, max: 2.1 ], Slowdowns: 0
2019/08/19 11:21:19 Loop: 0, Int: 2, Dur(s): 1.0, Mode: GET, Ops: 1110, MB/s: 4.34, IO/s: 1110, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 1.5 ], Slowdowns: 0
2019/08/19 11:21:20 Loop: 0, Int: 3, Dur(s): 1.0, Mode: GET, Ops: 1072, MB/s: 4.19, IO/s: 1072, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.3 ], Slowdowns: 0
2019/08/19 11:21:21 Loop: 0, Int: 4, Dur(s): 1.0, Mode: GET, Ops: 1098, MB/s: 4.29, IO/s: 1098, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 3.9 ], Slowdowns: 0
2019/08/19 11:21:22 Loop: 0, Int: 5, Dur(s): 1.0, Mode: GET, Ops: 1115, MB/s: 4.36, IO/s: 1115, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.4 ], Slowdowns: 0
2019/08/19 11:21:23 Loop: 0, Int: 6, Dur(s): 1.0, Mode: GET, Ops: 1110, MB/s: 4.34, IO/s: 1110, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.5 ], Slowdowns: 0
2019/08/19 11:21:24 Loop: 0, Int: 7, Dur(s): 1.0, Mode: GET, Ops: 1079, MB/s: 4.21, IO/s: 1079, Lat(ms): [ min: 0.6, avg: 0.7, 99%: 1.1, max: 1.7 ], Slowdowns: 0
2019/08/19 11:21:25 Loop: 0, Int: 8, Dur(s): 1.0, Mode: GET, Ops: 1089, MB/s: 4.25, IO/s: 1089, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 1.6 ], Slowdowns: 0
2019/08/19 11:21:26 Loop: 0, Int: 9, Dur(s): 1.0, Mode: GET, Ops: 1156, MB/s: 4.52, IO/s: 1156, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 5.4 ], Slowdowns: 0
2019/08/19 11:21:26 Loop: 0, Int: TOTAL, Dur(s): 10.0, Mode: GET, Ops: 11222, MB/s: 4.37, IO/s: 1118, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 5.4 ], Slowdowns: 0
2019/08/19 11:21:26 Running Loop 0 OBJECT DELETE TEST
2019/08/19 11:21:27 Loop: 0, Int: 0, Dur(s): 1.0, Mode: DEL, Ops: 5673, MB/s: 22.16, IO/s: 5673, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.7, max: 10.0 ], Slowdowns: 0
2019/08/19 11:21:28 Loop: 0, Int: 1, Dur(s): 1.0, Mode: DEL, Ops: 5597, MB/s: 21.86, IO/s: 5597, Lat(ms): [ min: 1.1, avg: 1.8, 99%: 2.9, max: 10.5 ], Slowdowns: 0
2019/08/19 11:21:29 Loop: 0, Int: 2, Dur(s): 1.0, Mode: DEL, Ops: 5123, MB/s: 20.01, IO/s: 5123, Lat(ms): [ min: 1.1, avg: 1.9, 99%: 3.2, max: 67.1 ], Slowdowns: 0
2019/08/19 11:21:30 Loop: 0, Int: 3, Dur(s): 1.0, Mode: DEL, Ops: 5547, MB/s: 21.67, IO/s: 5547, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.7, max: 15.2 ], Slowdowns: 0
2019/08/19 11:21:31 Loop: 0, Int: 4, Dur(s): 1.0, Mode: DEL, Ops: 5604, MB/s: 21.89, IO/s: 5604, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.7, max: 11.6 ], Slowdowns: 0
2019/08/19 11:21:32 Loop: 0, Int: 5, Dur(s): 1.0, Mode: DEL, Ops: 5610, MB/s: 21.91, IO/s: 5610, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.8, max: 8.4 ], Slowdowns: 0
2019/08/19 11:21:33 Loop: 0, Int: 6, Dur(s): 1.0, Mode: DEL, Ops: 5526, MB/s: 21.59, IO/s: 5526, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.8, max: 11.6 ], Slowdowns: 0
2019/08/19 11:21:34 Loop: 0, Int: 7, Dur(s): 1.0, Mode: DEL, Ops: 5538, MB/s: 21.63, IO/s: 5538, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.8, max: 8.7 ], Slowdowns: 0
2019/08/19 11:21:35 Loop: 0, Int: TOTAL, Dur(s): 8.6, Mode: DEL, Ops: 47783, MB/s: 21.61, IO/s: 5532, Lat(ms): [ min: 1.1, avg: 1.8, 99%: 2.8, max: 67.1 ], Slowdowns: 0
2019/08/19 11:21:35 Running Loop 0 BUCKET DELETE TEST
2019/08/19 11:21:35 Loop: 0, Int: TOTAL, Dur(s): 0.0, Mode: BDEL, Ops: 10, MB/s: 0.00, IO/s: 250, Lat(ms): [ min: 35.1, avg: 37.1, 99%: 39.4, max: 39.4 ], Slowdowns: 0
One notable point is that like the s3-benchmark program it is based on, hsbench has relatively low CPU overhead compared to some other S3 benchmarks. During the 4K PUT phase of the above test:
22628 root 20 0 6700416 5.308g 28072 S 413.3 8.4 676:28.86 ceph-osd
23329 root 20 0 1658700 253068 21320 S 333.3 0.4 530:06.76 radosgw
9017 perf 20 0 2184580 33612 6568 S 173.3 0.1 0:05.25 hsbench