8000 Add a custom LL metrics prometheus collector by kikkia · Pull Request #1150 · lavalink-devs/Lavalink · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add a custom LL metrics prometheus collector #1150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
May 30, 2025
Merged

Add a custom LL metrics prometheus collector #1150

merged 2 commits into from
May 30, 2025
+153 −0

Conversation

kikkia
Copy link
@kikkia kikkia commented May 21, 2025 •
edited by topi314
Loading

closes: #760

WIP to add a custom Prometheus collector for the custom LL metrics. I am going to be pretty busy all weekend but I thought I would throw this up sooner rather than later.

I haven't had time to test as I am going to bed and then will be busy for the weekend. If someone would build it and test it feel free, otherwise I will sometime early next week.

@kikkia kikkia force-pushed the add-custom-prometheus-metrics branch from cee2d0a to dea3e74 Compare May 22, 2025 01:10
@kikkia
Copy link
Author
kikkia commented May 26, 2025

Finally got around to confirming it worked.

# HELP lavalink_players_total Total number of players connected.
# TYPE lavalink_players_total gauge
lavalink_players_total 0.0
# HELP lavalink_playing_players_total Number of players currently playing audio.
# TYPE lavalink_playing_players_total gauge
lavalink_playing_players_total 0.0
# HELP lavalink_uptime_milliseconds Uptime of the node in milliseconds.
# TYPE lavalink_uptime_milliseconds gauge
lavalink_uptime_milliseconds 20367.0
# HELP lavalink_memory_free_bytes Memory statistics in bytes. (Free)
# TYPE lavalink_memory_free_bytes gauge
lavalink_memory_free_bytes 7.593824E7
# HELP lavalink_memory_used_bytes Memory statistics in bytes. (Used)
# TYPE lavalink_memory_used_bytes gauge
lavalink_memory_used_bytes 9.183392E7
# HELP lavalink_memory_allocated_bytes Memory statistics in bytes. (Allocated)
# TYPE lavalink_memory_allocated_bytes gauge
lavalink_memory_allocated_bytes 1.6777216E8
# HELP lavalink_memory_reservable_bytes Memory statistics in bytes. (Reservable)
# TYPE lavalink_memory_reservable_bytes gauge
lavalink_memory_reservable_bytes 1.6835936256E10
# HELP lavalink_cpu_cores CPU statistics. (Cores)
# TYPE lavalink_cpu_cores gauge
lavalink_cpu_cores 16.0
# HELP lavalink_cpu_system_load_percentage CPU statistics. (System Load)
# TYPE lavalink_cpu_system_load_percentage gauge
lavalink_cpu_system_load_percentage 0.0
# HELP lavalink_cpu_lavalink_load_percentage CPU statistics. (LL Load)
# TYPE lavalink_cpu_lavalink_load_percentage gauge
lavalink_cpu_lavalink_load_percentage 0.06622383252818036
# HELP logback_appender_total Logback log statements at various log levels
# TYPE logback_appender_total counter
logback_appender_total{level="debug",} 2.0
logback_appender_total{level="warn",} 1.0
logback_appender_total{level="trace",} 0.0
logback_appender_total{level="error",} 0.0
logback_appender_total{level="info",} 16.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 13.16
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.748259388754E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 153.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.4089624576E10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.3292672E8
# HELP lavalink_gc_pauses_seconds Garbage collection pauses by buckets
# TYPE lavalink_gc_pauses_seconds histogram
lavalink_gc_pauses_seconds_bucket{le="0.025",} 3.0
lavalink_gc_pauses_seconds_bucket{le="0.05",} 3.0
lavalink_gc_pauses_seconds_bucket{le="0.1",} 3.0
lavalink_gc_pauses_seconds_bucket{le="0.2",} 3.0
lavalink_gc_pauses_seconds_bucket{le="0.4",} 3.0
lavalink_gc_pauses_seconds_bucket{le="0.8",} 3.0
lavalink_gc_pauses_seconds_bucket{le="1.6",} 3.0
lavalink_gc_pauses_seconds_bucket{le="+Inf",} 3.0
lavalink_gc_pauses_seconds_count 3.0
lavalink_gc_pauses_seconds_sum 0.008
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
jvm_buffer_pool_used_bytes{pool="direct",} 106457.0
jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
# TYPE jvm_buffer_pool_capacity_bytes gauge
jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0
jvm_buffer_pool_capacity_bytes{pool="direct",} 106456.0
jvm_buffer_pool_capacity_bytes{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_buffers gauge
jvm_buffer_pool_used_buffers{pool="mapped",} 0.0
jvm_buffer_pool_used_buffers{pool="direct",} 20.0
jvm_buffer_pool_used_buffers{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 1.5083904E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 5.4029312E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 2.097152E8
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 4470656.0
jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 8610328.0
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 6994168.0
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 6.0015352E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 3209088.0
# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 15.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.054
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 44.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 24.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 44.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 50.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 38.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 1.0
jvm_threads_state{state="TIMED_WAITING",} 5.0
jvm_threads_state{state="UNKNOWN",} 0.0
# HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue.
# TYPE jvm_memory_objects_pending_finalization gauge
jvm_memory_objects_pending_finalization 0.0
# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 3.587532E7
jvm_memory_bytes_used{area="nonheap",} 8.8418912E7
# HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_committed gauge
jvm_memory_bytes_committed{area="heap",} 1.6777216E8
jvm_memory_bytes_committed{area="nonheap",} 9.1357184E7
# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",} 1.6835936256E10
jvm_memory_bytes_max{area="nonheap",} -1.0
# HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_init gauge
jvm_memory_bytes_init{area="heap",} 1.056964608E9
jvm_memory_bytes_init{area="nonheap",} 7667712.0
# HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_used gauge
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1500288.0
jvm_memory_pool_bytes_used{pool="Metaspace",} 6.0219712E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 1.5191552E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 7020704.0
jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 0.0
jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 2.8863488E7
jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 7011832.0
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 4486656.0
# HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_committed gauge
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 3276800.0
jvm_memory_pool_bytes_committed{pool="Metaspace",} 6.094848E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 1.5204352E7
jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 7405568.0
jvm_memory_pool_bytes_committed{pool="G1 Eden Space",} 8.388608E7
jvm_memory_pool_bytes_committed{pool="G1 Old Gen",} 7.5497472E7
jvm_memory_pool_bytes_committed{pool="G1 Survivor Space",} 8388608.0
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 4521984.0
# HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_max gauge
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 7606272.0
jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22023936E8
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="G1 Eden Space",} -1.0
jvm_memory_pool_bytes_max{pool="G1 Old Gen",} 1.6835936256E10
jvm_memory_pool_bytes_max{pool="G1 Survivor Space",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22028032E8
# HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_init gauge
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0
jvm_memory_pool_bytes_init{pool="G1 Eden Space",} 5.8720256E7
jvm_memory_pool_bytes_init{pool="G1 Old Gen",} 9.98244352E8
jvm_memory_pool_bytes_init{pool="G1 Survivor Space",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
# HELP jvm_memory_pool_collection_used_bytes Used bytes after last collection of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_used_bytes gauge
jvm_memory_pool_collection_used_bytes{pool="G1 Eden Space",} 0.0
jvm_memory_pool_collection_used_bytes{pool="G1 Old Gen",} 2.8863488E7
jvm_memory_pool_collection_used_bytes{pool="G1 Survivor Space",} 7011832.0
# HELP jvm_memory_pool_collection_committed_bytes Committed after last collection bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_committed_bytes gauge
jvm_memory_pool_collection_committed_bytes{pool="G1 Eden Space",} 8.388608E7
jvm_memory_pool_collection_committed_bytes{pool="G1 Old Gen",} 6.7108864E7
jvm_memory_pool_collection_committed_bytes{pool="G1 Survivor Space",} 8388608.0
# HELP jvm_memory_pool_collection_max_bytes Max bytes after last collection of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_max_bytes gauge
jvm_memory_pool_collection_max_bytes{pool="G1 Eden Space",} -1.0
jvm_memory_pool_collection_max_bytes{pool="G1 Old Gen",} 1.6835936256E10
jvm_memory_pool_collection_max_bytes{pool="G1 Survivor Space",} -1.0
# HELP jvm_memory_pool_collection_init_bytes Initial after last collection bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_init_bytes gauge
jvm_memory_pool_collection_init_bytes{pool="G1 Eden Space",} 5.8720256E7
jvm_memory_pool_collection_init_bytes{pool="G1 Old Gen",} 9.98244352E8
jvm_memory_pool_collection_init_bytes{pool="G1 Survivor Space",} 0.0
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 12093.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 12093.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 0.0
# HELP jvm_info VM version info
# TYPE jvm_info ga
8000
uge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Private Build",version="19.0.2+7-Ubuntu-0ubuntu322.04",} 1.0
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.748259394569E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.748259394569E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.748259394569E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.748259394569E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.748259394569E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.748259394569E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.748259394569E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.748259394569E9
# HELP lavalink_gc_pauses_seconds_created Garbage collection pauses by buckets
# TYPE lavalink_gc_pauses_seconds_created gauge
lavalink_gc_pauses_seconds_created 1.748259394508E9
# HELP logback_appender_created Logback log statements at various log levels
# TYPE logback_appender_created gauge
logback_appender_created{level="debug",} 1.748259394493E9
logback_appender_created{level="warn",} 1.748259394493E9
logback_appender_created{level="trace",} 1.748259394493E9
logback_appender_created{level="error",} 1.748259394493E9
logback_appender_created{level="info",} 1.748259394493E9

@kikkia kikkia changed the title WIP: Add a custom LL metrics prometheus collector Add a custom LL metrics prometheus collector May 26, 2025
@topi314
Copy link
Member
topi314 commented May 26, 2025

is there a specific reason you chose GaugeMetricFamily over Gauge?

@kikkia
Copy link
Author
kikkia commented May 26, 2025 •
edited
Loading

is there a specific reason you chose GaugeMetricFamily over Gauge?

https://www.javadoc.io/doc/io.prometheus/simpleclient/latest/io/prometheus/client/GaugeMetricFamily.html

Gauge metric family, for custom collectors and exporters.

https://stackoverflow.com/questions/55971741/prometheus-whats-the-difference-between-a-gauge-and-a-gaugemetricfamily

You can think of it that Gauge is an instrumentation-friendly object the takes care of tracking values, concurrency, labels. At the end of this it produces a GaugeMetricFamily data structure as a snapshot of its current state (it doesn't actually use a GaugeMetricFamily, but that's an implementation detail) when needed by a scrape. For custom collectors, you have to produce the GaugeMetricFamily yourself.

These are basically my understanding of it.

@topi314
Copy link
Member
topi314 commented May 26, 2025

as far as I am aware we don't need custom collectors
just define the metrics & call the update methods when things change
https://www.baeldung.com/java-prometheus-client

@kikkia
Copy link
Author
kikkia commented May 26, 2025

By using a custom collector we can separate this logic from application logic, as well as also having the actual metrics being only updated on demand.

For values like uptimeMs, memory stats, cpu load, etc, it makes sense to update them at scrape time. It also ensures that the metrics endpoint is serving the most up to date data.

@topi314
Copy link
Member
topi314 commented May 27, 2025

alright that is true, there is one other issue which should be addressed before this can be merged since load will be incorrect if you call the metrics too often then

#1132

Ideally we would cache the load & frame stats inbetween calls unless a certain time has passed

topi314
topi314 requested changes May 27, 2025
View reviewed changes
topi314
topi314 requested changes May 29, 2025
View reviewed changes
Copy link
Member
@topi314 topi314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should target dev
rebase your changes please

@kikkia kikkia force-pushed the add-custom-prometheus-metrics branch from 8808f85 to 8e906b2 Compare May 29, 2025 14:50
@kikkia kikkia changed the base branch from master to dev May 29, 2025 14:50
@topi314 topi314 merged commit 5c13c72 into lavalink-devs:dev May 30, 2025
1 check passed
@topi314 topi314 mentioned this pull request May 30, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@topi314 topi314 topi314 approved these changes

Assignees

@kikkia kikkia

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request] Add LavaLink statistics to built-in Prometheus provider
2 participants
@kikkia @topi314
0