rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Grafana: the connections panel shows in use and ready, not a pool that looks full (#251)

Browse Source 
Since #241 the daemon keeps all 20 connections open, so the panel's open line sat on the pool's
maximum and read as exhausted; in use has been at most 4 in a week. It draws in use, ready
(idle) and the pool now, and the prod check reads idle the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-15 21:25:50 +00:00
parent 6fde4b73cd
commit ca73b8136f
2 changed files with 6 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -196,8 +196,8 @@ a count says something happened, the lines and traces say why.
$Q prom 'max(max_over_time(ipx_deadair_busy_seconds{job="ipx"}[$range]))' 24h $Q prom 'max(max_over_time(ipx_deadair_busy_seconds{job="ipx"}[$range]))' 24h
``` ```
`up` 0 means Prometheus cannot reach the port: the container is down or `metrics_bind` is `up` 0 means Prometheus cannot reach the port: the container is down or `metrics_bind` is
unset. Connections in use at the pool's max (20 on Postgres) means something is waiting for unset. The daemon keeps all 20 connections open on Postgres (#241), so `idle` is the pool's 20 less
one: find the slow requests or scan of that time in check 3 and 5. A worker queue that only those in use; in use at 20, idle at 0, means something is waiting for one: find the slow requests or scan of that time in check 3 and 5. A worker queue that only
grows means downloads are not keeping up, or the worker is stuck on one job. A runtime queue grows means downloads are not keeping up, or the worker is stuck on one job. A runtime queue
above 0 for more than a scrape or two means every thread is busy: something is blocking a above 0 for more than a scrape or two means every thread is busy: something is blocking a
thread or burning one. Memory, open files or tasks only climbing across a day with no restart thread or burning one. Memory, open files or tasks only climbing across a day with no restart

11
grafana/dashboard.py
View File

@@ -259,13 +259,10 @@ row("ipx: process")
# Open connections stay open, idle, for ten minutes after a burst, so open at the max is not # Open connections stay open, idle, for ten minutes after a burst, so open at the max is not
# pressure; drawn as a line of its own it read as the pool maxing out (#199). # pressure; drawn as a line of its own it read as the pool maxing out (#199).
ts("Database connections", [prom('max(max_over_time(ipx_db_connections{job="ipx", state="in_use"}[$__interval]))', legend="in use"), ts("Database connections", [prom('max(max_over_time(ipx_db_connections{job="ipx", state="in_use"}[$__interval]))', legend="in use"),
prom('sum(ipx_db_connections{job="ipx"})', ref="B", legend="open"), prom('min(min_over_time(ipx_db_connections{job="ipx", state="idle"}[$__interval]))', ref="B", legend="ready"),
prom('max(ipx_db_connections_max{job="ipx"})', ref="C", legend="most the pool opens")], prom('max(ipx_db_connections_max{job="ipx"})', ref="C", legend="the pool")],
0, w=8, desc="In use reaching the most the pool opens means requests and scans are waiting for a connection. " 0, w=8, desc="The daemon keeps the whole pool open (#241), so in use and ready add up to the pool. In use reaching "
"Open connections stay open, idle, for ten minutes after a burst.")["fieldConfig"]["overrides"] = [ "the pool, ready at 0, means requests and scans are waiting for a connection (#251).")
{"matcher": {"id": "byName", "options": "open"},
"properties": [{"id": "custom.lineStyle", "value": {"fill": "dash", "dash": [4, 4]}},
{"id": "custom.fillOpacity", "value": 0}, {"id": "color", "value": {"mode": "fixed", "fixedColor": "#8e8e8e"}}]}]
ts("Memory", [prom('ipx_process_resident_bytes{job="ipx"}', legend="resident")], 8, w=8, unit="bytes") ts("Memory", [prom('ipx_process_resident_bytes{job="ipx"}', legend="resident")], 8, w=8, unit="bytes")
ts("Open files and sockets", [prom('ipx_process_open_fds{job="ipx"}', legend="open")], 16, w=8) ts("Open files and sockets", [prom('ipx_process_open_fds{job="ipx"}', legend="open")], 16, w=8)
y += 8 y += 8