Varnish Custom Counters (VCC) allows aggregation of custom log entries extracted from Varnish shared memory log (VSL).
VCC is a simple PoC Python script (Python >= 2.6 & < 3 required) showing how to access to VSL data (only Varnish 3.x is supported at the moment) using libvarnishapi.so and ctypes. Fetched items are locally aggregated using some user selected operator (currently count, hll, min, max, avg, last & first are supported) and periodically rendered in an ugly curses UI.
Remember this is just a PoC. Don't expect updates or fixes here. Anyway, comments, improvements and contributions are welcome!
VCC is sponsored by Allenta Consulting, the Varnish Software integration partner for Spain and Portugal.
Using the following VCL:
sub vcl_deliver {
std.log("vcc:Sample counter #1 (COUNT):count:");
std.log("vcc:Sample counter #2 (HLL):hll,5,234:" + req.http.X-Whatever);
std.log("vcc:Sample counter #3 (MIN):min:" + req.http.X-Whatever);
std.log("vcc:Sample counter #4 (MAX):max:" + req.http.X-Whatever);
std.log("vcc:Sample counter #5 (AVG):avg:" + req.http.X-Whatever);
std.log("vcc:Sample counter #6 (FIRST):first:" + req.http.X-Whatever);
std.log("vcc:Sample counter #7 (LAST):last:" + req.http.X-Whatever);
}
After processing some requests, you will get something like:
Install VCC and all its dependencies:
~$ sudo pip install VCC
Add some logging statement to your VCL. Format of logged messages is:
vcc:<counter name>:<aggregation function + configuration>:<value>
Only the
hllaggregation function requires configuration parameters:k(integer value in the range [2, 16]) andseed(integer value). Please, refer to https://github.com/ascv/HyperLogLog for extra information.Execute VCC while Varnish is processing requests (use
--helpfor extra options):~$ vcc --version 3 --nwindows=10 --wsize=60