Skip to content

otelmux: client-initiated request termination is recorded as a 500 with span status Error and no error.type #9375

Description

@danielloader

Split out from #9358 as requested in #9358 (comment). Same defect class as the otelhttp report, confirmed independently against otelmux.

Symptom

When a client disconnects mid-request, the handler observes the cancelled request context and typically returns an error, which surfaces as a 500. otelmux then records http.response.status_code=500 and sets the span status to Error, and sets no error.type at all, so a client disconnect is indistinguishable from a genuine server fault.

otelmux is the closest of the framework instrumentations to otelhttp: it uses the same internal/request.RespWriterWrapper (defaulting to 200, recording only what the handler wrote) and reads the status after tw.handler.ServeHTTP without inspecting r.Context().Err():

tw.handler.ServeHTTP(w, r.WithContext(ctx))
statusCode := rww.StatusCode()
span.SetStatus(tw.semconv.Status(statusCode))
span.SetAttributes(tw.semconv.ResponseTraceAttrs(semconv.ResponseTelemetry{
StatusCode: statusCode,
ReadBytes: bw.BytesRead(),
ReadError: bw.Error(),
WriteBytes: rww.BytesWritten(),
WriteError: rww.Error(),
})...)
metricAttributes := semconv.MetricAttributes{
Req: r,
StatusCode: statusCode,
Route: routeStr,
AdditionalAttributes: tw.metricAttributesFromRequest(r),
}
tw.semconv.RecordMetrics(ctx, semconv.ServerMetricData{
ServerName: tw.service,
ResponseSize: rww.BytesWritten(),
MetricAttributes: metricAttributes,
MetricData: semconv.MetricData{
RequestSize: bw.BytesRead(),
RequestDuration: time.Since(requestStartTime),
},
})

Note that otelmux already plumbs ReadError and WriteError into ResponseTraceAttrs from the body and response-writer wrappers, but the shared semconv implementation never turns those into error.type — so the observed failure cause is collected and then discarded. RecordMetrics likewise carries StatusCode only, with no error.type.

Reproduction

Confirmed on main at 5f30960 with a real HTTP/1.1 disconnect. The handler waits on r.Context().Done() and then writes 500:

router.Use(otelmux.Middleware("srv", otelmux.WithTracerProvider(tp)))
router.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
    <-r.Context().Done()
    w.WriteHeader(http.StatusInternalServerError)
    _, _ = w.Write([]byte("cancelled"))
})

The client issues the request with a cancellable context and cancels it after 100ms. Recorded span:

span name   = "GET /hello"
span status = Error ""
attr http.route                 = /hello
attr http.response.body.size    = 9
attr http.response.status_code  = 500

No error.type. Identical outcome to the otelhttp reproduction in #9358.

Suggested fix

Mirroring the triage in #9358 (comment):

  1. Prefer an observed response-write error, request-body read error, or request-context error as the failure cause — the first two are already available via rww.Error() and bw.Error().
  2. Set span status to Error and add a predictable, low-cardinality error.type for that cause.
  3. Preserve the observed/selected HTTP status code; for an error status with no more specific cause, use the status-code string as error.type.
  4. Add error.type to the metric attributes recorded by RecordMetrics, per the HTTP metric conventions.
  5. Add a real disconnect regression test so cancellation is detected before the outer ServeHTTP returns.

Because otelmux follows otelhttp's structure so closely, the fix here should be close to a direct port of whatever lands in #9358.

Note on shared code

otelmux/internal/semconv/server.go is generated from internal/shared/semconv/server.go.tmpl and is byte-identical to the copy vendored into otelhttp (verified by diff). The same is true for otelgin, otelecho and otelrestful. So whatever error.type classification lands in the shared template is picked up by all of them, but each middleware still needs its own wiring to detect and pass the failure cause, plus its own regression test.

Relevant requirements: HTTP span conventions, HTTP metric conventions, and the dropped-connection example.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions