Description
Hello Collectd community,
I've noticed a memory leak within Collectd, only when i have the libvirt HostnameMetadataXpath configured. I'm using collectd itself to monitor the RSS of the collectd process.
These charts were created from the following metrics in Prometheus. collectd_processes_ps_rss(processes="collectd"}
I've titled each chart with the version of collectd it is monitoring and whether the metadataXpath parameter is set. I've tried to display enough examples of versions and configuration that i only see this behavior when the metadataXpath is being used.
I believe the change that has causing the memory leak is here;
#2807
Here is my virt and processes plugin configuration.
LoadPlugin processes
<Plugin processes>
<Process "collectd">
CollectFileDescriptor true
CollectContextSwitch true
</Process>
</Plugin>
LoadPlugin virt
<Plugin "virt">
Connection "qemu:///system"
HostnameFormat name metadata hostname
HostnameMetadataXPath "/instance/owner/project/@uuid"
PluginInstanceFormat uuid
ExtraStats "disk disk_err pcpu perf vcpupin cpu_util"
</Plugin>
Tested on versions
-
Collectd-5.8.0-10 (no MetadataXpath available in this version) - No memory leak present with libvirt plugin enabled.
-
Collect 5.8.1.592.gd3eae6b-9 ( first build including the aformentioned PR ) - memory leak present with MetadataXPath is enabled, memory leak not present when disabled
-
Collect 5.9.0-9 - Latest stable release, memory leak present with MetadataXPath is enabled, memory leak not present when disabled
-
Operating system / distribution: RHEL 7.5 -
-
Kernel version (if applicable): kernel-3.10.0-862.el7.x86_64
Expected behavior
I'd expect collectd memory usage to remain stable.
Actual behavior
With the libvirt MetadataXpath parameter defined, collectd grows to consume approx 1.5GB per day. Although not shown in the grafana screenshot, i've seen this number grow to in excess of 10GB, but for obvious reasons, at this point we had to restart the process.
Steps to reproduce
- Install any collectd build that contains the PR for the libvirt metadataXpath commit.
- In my examples i installed it on hypervisors that are part of a Openstack cloud. Over a short period of time, as vms are provisioned and terminated, observe collectd memory RES.
- I have noticed the memory consumption gets worse on our hypervisors that are used more frequently.
- When installed on a hypervisor that is hardly being used at all, 1 or 2 vms, its very difficult to notice the behaviour. That is why i suspect it probably was not noticed in a testing environment.
Has anyone else seen this behaviour?