Production log lines lost their trace ids after ba18b44
#411
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Deployed 2026-11-07 13:37 (ipx and ipx-directory), the fix for #408 left every JSON line without trace_id and span_id: none of the daemon's start lines, inside the start span, had them. WithTrace now reaches the span's OpenTelemetry context through the dispatch its layer was registered with (on_register_dispatch), and production's logging wrapped the JSON layer in an Option, whose Layer impl in tracing-subscriber 0.3.23 does not pass on_register_dispatch on. The log test built its own subscriber without the Option, so it passed. Found checking docker logs after the deploy. Loki lines lose their link to Tempo while it lasts.
Fixed in
f5a3847: both services box their stderr layer instead of an Option of each kind (a Box passes on_register_dispatch on), the formatter falls back to the default dispatcher when its layer was never registered, and the log test now builds its subscriber with the same function init uses, so a wrapper like the Option is under test. Deployed 13:43 (ipx) and 13:45 (directory): ipx's start lines name their trace again (4 of the 9 lines after start, those inside a span, as before), and the directory's request and check lines do. Lines went out without ids for about six minutes on ipx and eight on the directory.