Expose more cgroup memory metrics #37328

bonnefoa · 2025-05-23T14:37:06Z

What does this PR do?

This PR adds additional metrics reported by cgroup:

shmem: Amount of memory used by tmpfs or shm_open
lruvec stats - active/inactive anon/file + unevictable
cache filesystem mapped/dirty/writeback
refault anon/file
page table size (cgroupv2 only)

Motivation

We currently only extract high level metrics from cgroup memory metrics like RSS and cache usage. This makes it difficult to understand to understand some OOM as shared memory and anonymous mmap are not covered.
Plus, we don't have a lot of information regarding the state of the page cache. Those metrics will give a better insight on what's going on: how much active memory is used and whether the page cache is dirty or being flushed.

Describe how you validated your changes

Unit tests were updated to add the new metrics. This was also manually tested that cgroupv1, cgroupv2 and containerd report the new metrics.

Possible Drawbacks / Trade-offs

Additional Notes

agent-platform-auto-pr · 2025-05-23T15:10:31Z

Uncompressed package size comparison

Comparison with ancestor 95894ac05dd78eb180313c4ca46d54219f73d916

Diff per package

package	diff	status	size	ancestor	threshold
datadog-agent-amd64-deb	0.02MB	⚠️	773.99MB	773.97MB	0.50MB
datadog-agent-x86_64-rpm	0.02MB	⚠️	782.94MB	782.92MB	0.50MB
datadog-agent-x86_64-suse	0.02MB	⚠️	782.94MB	782.92MB	0.50MB
datadog-agent-arm64-deb	0.02MB	⚠️	760.09MB	760.07MB	0.50MB
datadog-agent-aarch64-rpm	0.02MB	⚠️	769.02MB	769.00MB	0.50MB
datadog-heroku-agent-amd64-deb	0.01MB	⚠️	380.66MB	380.65MB	0.50MB
datadog-iot-agent-arm64-deb	0.01MB	⚠️	59.38MB	59.37MB	0.50MB
datadog-iot-agent-aarch64-rpm	0.01MB	⚠️	59.46MB	59.45MB	0.50MB
datadog-dogstatsd-arm64-deb	0.00MB	✅	30.48MB	30.47MB	0.50MB
datadog-iot-agent-amd64-deb	0.00MB	✅	62.80MB	62.80MB	0.50MB
datadog-iot-agent-x86_64-rpm	0.00MB	✅	62.89MB	62.88MB	0.50MB
datadog-iot-agent-x86_64-suse	0.00MB	✅	62.89MB	62.88MB	0.50MB
datadog-dogstatsd-amd64-deb	0.00MB	✅	31.99MB	31.99MB	0.50MB
datadog-dogstatsd-x86_64-rpm	0.00MB	✅	32.07MB	32.07MB	0.50MB
datadog-dogstatsd-x86_64-suse	0.00MB	✅	32.07MB	32.07MB	0.50MB

Decision

⚠️ Warning

cit-pr-commenter · 2025-05-23T16:12:56Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 42d58efb-c43e-4206-8ec5-b79fca1d3454

Baseline: d080da0
Comparison: 93c5687
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+3.70	[+0.66, +6.73]	1	Logs
➖	quality_gate_logs	% cpu utilization	+2.93	[+0.14, +5.72]	1	Logs bounds checks dashboard
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+2.83	[+1.96, +3.70]	1	Logs
➖	docker_containers_memory	memory utilization	+1.36	[+1.28, +1.44]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	+0.75	[+0.69, +0.81]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.67	[+0.53, +0.81]	1	Logs bounds checks dashboard
➖	file_tree	memory utilization	+0.52	[+0.35, +0.68]	1	Logs
➖	quality_gate_idle	memory utilization	+0.32	[+0.25, +0.39]	1	Logs bounds checks dashboard
➖	otlp_ingest_metrics	memory utilization	+0.28	[+0.12, +0.44]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	+0.11	[-0.51, +0.73]	1	Logs
➖	file_to_blackhole_0ms_latency_http2	egress throughput	+0.09	[-0.51, +0.68]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	+0.09	[-0.47, +0.64]	1	Logs
➖	file_to_blackhole_0ms_latency_http1	egress throughput	+0.03	[-0.52, +0.59]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.02	[-0.05, +0.09]	1	Logs
➖	file_to_blackhole_1000ms_latency_linear_load	egress throughput	+0.01	[-0.22, +0.24]	1	Logs
➖	file_to_blackhole_300ms_latency	egress throughput	+0.01	[-0.61, +0.63]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.02, +0.02]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.28, +0.26]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.01	[-0.61, +0.58]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	-0.08	[-0.67, +0.50]	1	Logs
➖	ddot_logs	memory utilization	-0.19	[-0.29, -0.10]	1	Logs
➖	ddot_metrics	memory utilization	-0.44	[-0.55, -0.32]	1	Logs
➖	otlp_ingest_logs	memory utilization	-0.55	[-0.68, -0.43]	1	Logs

Bounds Checks: ❌ Failed

perf	experiment	bounds_check_name	replicates_passed	links
❌	quality_gate_idle_all_features	memory_usage	9/10	bounds checks dashboard
✅	docker_containers_cpu	simple_check_run	10/10
✅	docker_containers_memory	memory_usage	10/10
✅	docker_containers_memory	simple_check_run	10/10
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http1	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http1	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http2	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http2	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency_linear_load	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	lost_bytes	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

❌ Failed. Some Quality Gates were violated.

quality_gate_idle_all_features, bounds check memory_usage: 9/10 replicas passed. Failed 1 which is > 0. Gate FAILED.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.

iadjivon

All set from docs

agent-platform-auto-pr · 2025-05-23T17:08:42Z

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor d080da0

Successful checks

Info

	Quality gate	Delta	On disk size (MiB)	Delta	On wire size (MiB)
✅	agent_deb_amd64	$${+0.02}$$	$${697.07}$$ < $${697.37}$$	$${+26.76}$$	$${176.1}$$ < $${177.03}$$
✅	agent_deb_amd64_fips	$${+0.01}$$	$${695.36}$$ < $${695.59}$$	$${+25.57}$$	$${175.57}$$ < $${176.51}$$
✅	agent_heroku_amd64	$${+0.01}$$	$${358.7}$$ < $${359.67}$$	$${+11.33}$$	$${96.53}$$ < $${97.47}$$
✅	agent_msi	$${+0}$$	$${958.9}$$ < $${959.86}$$	$${-0.01}$$	$${146.3}$$ < $${147.27}$$
✅	agent_rpm_amd64	$${+0.02}$$	$${697.06}$$ < $${697.36}$$	$${+25.9}$$	$${177.7}$$ < $${178.56}$$
✅	agent_rpm_amd64_fips	$${+0.01}$$	$${695.35}$$ < $${695.58}$$	$${+24.95}$$	$${177.57}$$ < $${178.43}$$
✅	agent_rpm_arm64	$${+0.01}$$	$${687.08}$$ < $${687.37}$$	$${+22.79}$$	$${161.1}$$ < $${161.99}$$
✅	agent_rpm_arm64_fips	$${+0.02}$$	$${685.49}$$ < $${685.72}$$	$${+21.48}$$	$${160.28}$$ < $${161.11}$$
✅	agent_suse_amd64	$${+0.02}$$	$${697.06}$$ < $${697.36}$$	$${+25.9}$$	$${177.7}$$ < $${178.56}$$
✅	agent_suse_amd64_fips	$${+0.01}$$	$${695.35}$$ < $${695.58}$$	$${+24.95}$$	$${177.57}$$ < $${178.43}$$
✅	agent_suse_arm64	$${+0.01}$$	$${687.08}$$ < $${687.37}$$	$${+22.79}$$	$${161.1}$$ < $${161.99}$$
✅	agent_suse_arm64_fips	$${+0.02}$$	$${685.49}$$ < $${685.72}$$	$${+21.48}$$	$${160.28}$$ < $${161.11}$$
✅	docker_agent_amd64	$${+0.02}$$	$${780.87}$$ < $${781.16}$$	$${+0.01}$$	$${268.82}$$ < $${269.63}$$
✅	docker_agent_arm64	$${+0.01}$$	$${794.34}$$ < $${794.62}$$	$${+0}$$	$${256.16}$$ < $${257.0}$$
✅	docker_agent_jmx_amd64	$${+0.02}$$	$${972.07}$$ < $${972.35}$$	$${-0.01}$$	$${337.78}$$ < $${338.6}$$
✅	docker_agent_jmx_arm64	$${+0.01}$$	$${974.13}$$ < $${974.41}$$	$${+0}$$	$${321.12}$$ < $${321.97}$$
✅	docker_agent_windows1809	$${+0}$$	$${1180.54}$$ < $${1488.0}$$	$${+0.01}$$	$${416.19}$$ < $${488.95}$$
✅	docker_agent_windows1809_core	$${+0}$$	$${5910.5}$$ < $${6218.0}$$	$${0}$$	$${2048.0}$$ < $${2049.0}$$
✅	docker_agent_windows1809_core_jmx	$${-0.13}$$	$${6032.07}$$ < $${6361.0}$$	$${0}$$	$${2048.0}$$ < $${2049.0}$$
✅	docker_agent_windows1809_jmx	$${+0}$$	$${1302.17}$$ < $${1609.5}$$	$${+0.15}$$	$${458.63}$$ < $${531.32}$$
✅	docker_agent_windows2022	$${-0.06}$$	$${1199.64}$$ < $${1507.0}$$	$${+0.04}$$	$${428.95}$$ < $${501.7}$$
✅	docker_agent_windows2022_core	$${+0.17}$$	$${5883.56}$$ < $${6311.0}$$	$${0}$$	$${2048.0}$$ < $${2049.0}$$
✅	docker_agent_windows2022_core_jmx	$${-0.46}$$	$${6005.01}$$ < $${6313.0}$$	$${0}$$	$${2048.0}$$ < $${2049.0}$$
✅	docker_agent_windows2022_jmx	$${-0.28}$$	$${1321.43}$$ < $${1628.16}$$	$${+0.02}$$	$${471.21}$$ < $${543.98}$$
✅	docker_cluster_agent_amd64	$${+0}$$	$${212.86}$$ < $${213.79}$$	$${+0}$$	$${72.39}$$ < $${73.33}$$
✅	docker_cluster_agent_arm64	$${-0.06}$$	$${228.69}$$ < $${229.64}$$	$${-0.02}$$	$${68.65}$$ < $${69.6}$$
✅	docker_cws_instrumentation_amd64	$${-0}$$	$${7.08}$$ < $${7.12}$$	$${-0}$$	$${2.95}$$ < $${3.29}$$
✅	docker_cws_instrumentation_arm64	$${-0}$$	$${6.69}$$ < $${6.92}$$	$${-0}$$	$${2.7}$$ < $${3.07}$$
✅	docker_dogstatsd_amd64	$${+0}$$	$${39.22}$$ < $${39.57}$$	$${-0}$$	$${15.12}$$ < $${15.76}$$
✅	docker_dogstatsd_arm64	$${0}$$	$${37.88}$$ < $${38.2}$$	$${+0.01}$$	$${14.54}$$ < $${14.83}$$
✅	dogstatsd_deb_amd64	$${+0}$$	$${30.46}$$ < $${31.4}$$	$${+0.08}$$	$${8.0}$$ < $${8.95}$$
✅	dogstatsd_deb_arm64	$${+0}$$	$${29.03}$$ < $${29.97}$$	$${+0.09}$$	$${6.94}$$ < $${7.89}$$
✅	dogstatsd_rpm_amd64	$${+0}$$	$${30.46}$$ < $${31.4}$$	$${+0.08}$$	$${8.01}$$ < $${8.96}$$
✅	dogstatsd_suse_amd64	$${+0}$$	$${30.46}$$ < $${31.4}$$	$${+0.08}$$	$${8.01}$$ < $${8.96}$$
✅	iot_agent_deb_amd64	$${+0.01}$$	$${50.45}$$ < $${51.38}$$	$${+0.25}$$	$${12.85}$$ < $${13.79}$$
✅	iot_agent_deb_arm64	$${+0.01}$$	$${47.92}$$ < $${48.85}$$	$${+0.22}$$	$${11.14}$$ < $${12.09}$$
✅	iot_agent_deb_armhf	$${+0}$$	$${47.49}$$ < $${48.42}$$	$${+0}$$	$${11.21}$$ < $${12.16}$$
✅	iot_agent_rpm_amd64	$${+0.01}$$	$${50.45}$$ < $${51.38}$$	$${+0.24}$$	$${12.86}$$ < $${13.81}$$
✅	iot_agent_rpm_arm64	$${+0.01}$$	$${47.92}$$ < $${48.85}$$	$${+0.22}$$	$${11.16}$$ < $${12.11}$$
✅	iot_agent_suse_amd64	$${+0.01}$$	$${50.45}$$ < $${51.38}$$	$${+0.24}$$	$${12.86}$$ < $${13.81}$$

vboulineau

Code itself LGTM, but I think there's perhaps more work needed on the naming and curation.

pkg/collector/corechecks/containers/containerd/adapter.go

vboulineau · 2025-05-23T17:52:31Z

pkg/collector/corechecks/containers/generic/processor.go

@@ -166,6 +166,18 @@ func (p *Processor) processContainer(sender sender.Sender, tags []string, contai
 		p.sendMetric(sender.Rate, "container.memory.partial_stall", containerStats.Memory.PartialStallTime, tags)
 		p.sendMetric(sender.MonotonicCount, "container.memory.page_faults", containerStats.Memory.Pgfault, tags)
 		p.sendMetric(sender.MonotonicCount, "container.memory.major_page_faults", containerStats.Memory.Pgmajfault, tags)
+		p.sendMetric(sender.Gauge, "container.memory.active_anon", containerStats.Memory.ActiveAnon, tags)


Highlighting that we don't usually just push the names from underlying layer, there's a work to do to understand if these names can make sense in other context or conflict with other OSes.
Typically the container.memory.working_set_refault_anon is, I think, not coherent with the definition of working_set that already exists

I've renamed working_set_refault_* to refault_anon/refault_file. I think the working_set prefix mainly come from the fact that refault detection is managed by the workingset detection.

For the lruvec metrics (active/inactive anon/file + unevictable), I think it's linux specific. At least, I don't think there are equivalent in Windows.

Looking at windows memory metrics, the nearest equivalent would be windows_memory_modified_page_list_bytes with file_dirty. Though the doc says that it's page not actively used by system cache so I don't think they would be strictly equivalent.

Outside of that, I haven't found clear equivalent with windows metrics.

This PR adds additional metrics reported by cgroup: - shmem: Amount of memory used by tmpfs or shm_open - lruvec stats - active/inactive anon/file + unevictable - cache filesystem mapped/dirty/writeback - refault anon/file - page table size (cgroupv2 only) We currently only extract high level metrics from cgroup memory metrics like RSS and cache usage. This makes it difficult to understand to understand some OOM as shared memory and anonymous mmap is not covered. Plus, we don't have a lot of information regarding the state of the page cache. Those metrics will give a better insight on what's going on: how much active memory is used and whether the page cache is dirty or being flushed.

nubtron

Hi @bonnefoa, we keep track of metrics emitted by integrations in a file called metadata.csv, in this case it would be be https://github.com/DataDog/integrations-core/blob/master/container/metadata.csv . Since this PR adds new metrics that file should be updated as well, but it's located in another repo so a follow-up PR will be needed. Let me know if you have any question about metadata.csv!

nubtron

Approving this on behalf of Agent Integrations, but wanted to note that #container-integrations is the owner of the datadog-agent part this integration so they are better equipped to review this PR. There's another part in integrations-core - added a comment about that!

Follow up of DataDog/datadog-agent#37328 which adds additional memory metrics from cgroup.

bonnefoa requested a review from a team as a code owner May 23, 2025 14:37

github-actions bot added the long review PR is complex, plan time to review it label May 23, 2025

bonnefoa force-pushed the anthonin.bonnefoy/cgroup-metrics branch from 743ae18 to 9584700 Compare May 23, 2025 15:27

bonnefoa requested a review from a team as a code owner May 23, 2025 15:27

iadjivon approved these changes May 23, 2025

View reviewed changes

vboulineau reviewed May 23, 2025

View reviewed changes

bonnefoa force-pushed the anthonin.bonnefoy/cgroup-metrics branch from 9584700 to 14251c4 Compare May 27, 2025 13:42

github-actions bot added medium review PR review might take time and removed long review PR is complex, plan time to review it labels May 27, 2025

bonnefoa force-pushed the anthonin.bonnefoy/cgroup-metrics branch from 14251c4 to 35b4422 Compare May 27, 2025 14:26

bonnefoa added the qa/done QA done before merge and regressions are covered by tests label May 27, 2025

bonnefoa requested a review from vboulineau June 3, 2025 12:22

bonnefoa force-pushed the anthonin.bonnefoy/cgroup-metrics branch from 35b4422 to 93c5687 Compare June 19, 2025 08:18

nubtron reviewed Jun 19, 2025

View reviewed changes

nubtron approved these changes Jun 19, 2025

View reviewed changes

bonnefoa added a commit to DataDog/integrations-core that referenced this pull request Jun 23, 2025

Add new cgroup metrics metadata

9832658

Follow up of DataDog/datadog-agent#37328 which adds additional memory metrics from cgroup.

bonnefoa added a commit to DataDog/integrations-core that referenced this pull request Jun 23, 2025

Add new cgroup metrics metadata

6adaadd

Follow up of DataDog/datadog-agent#37328 which adds additional memory metrics from cgroup.

bonnefoa mentioned this pull request Jun 23, 2025

Add new cgroup metrics metadata DataDog/integrations-core#20564

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Expose more cgroup memory metrics #37328

Expose more cgroup memory metrics #37328

Uh oh!

Uh oh!

Uh oh!

Fine details of change detection per experiment

Explanation

Uh oh!

Uh oh!

Info

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Expose more cgroup memory metrics #37328

Are you sure you want to change the base?

Expose more cgroup memory metrics #37328

Uh oh!

Conversation

What does this PR do?

Motivation

Describe how you validated your changes

Possible Drawbacks / Trade-offs

Additional Notes

Uh oh!

Uh oh!

Uncompressed package size comparison

Decision

Uh oh!

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

Bounds Checks: ❌ Failed

Explanation

CI Pass/Fail Decision

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Static quality checks

Info

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!