docs(rfc): RFC 0014 - agent-level observability (OTLP relay)

[rhuss]

Summary

Proposes a supervisor-local telemetry relay that collects agent-emitted traces from inside network-isolated sandboxes, enriches them with sandbox context, and forwards them through the gateway to an external collector. Rooted in four persona workflows (agent developer, platform operator, workspace administrator, security reviewer).

• Telemetry relay: supervisor as OTLP sidecar (gRPC + HTTP on ports 4317/4318), with per-driver reachability analysis (Docker, Podman, K8s embedded/sidecar, VM)
• Span enrichment: sandbox ID, policy, user, workspace, driver as resource attributes
• Trace context propagation: inject-if-missing W3C traceparent on agent egress
• Log relay: centralized OCSF event collection via existing PushSandboxLogs mechanism
• Visibility domains: infrastructure vs agent traces, per-domain endpoint routing as future extension
• Multi-tenant observability: per-workspace OTLP routing via workspace settings tier (deferred to dedicated spec)
• Separation of concerns: agent developer, workspace admin, and global admin roles
• Short-lived agent support: supervisor flushes buffered spans on behalf of terminated agent processes

Includes a spike plan for relay validation (OTLP receiver, session protocol forwarding, performance measurement across driver topologies).

Related issues

• #1055 - Enterprise Observability
• #2508 - Sandbox supervisor OTel
• #1922 - Centralized sandbox log collection

Note on RFC number

RFC number 0014 is provisional. Per the RFC process, maintainers assign numbers from an issue. This PR uses the next available number; please reassign if needed.

Split from PR #2620 (RFC 0012) based on maintainer feedback. Infrastructure-level observability is covered in companion RFC 0013 (#2628).

Test plan

• Review RFC content for technical accuracy
• Verify persona workflows match real agent developer/operator needs
• Validate OTLP receiver reachability analysis against actual driver networking
• Complete the spike before advancing to review state

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rhuss]

Closing this PR. I overlooked the process documented in rfc/README.md: features start as GitHub issues, and RFCs are only created when maintainers request one from an existing issue. Apologies for skipping that step.

The agent-level observability content is being reformed as GitHub issues linked to the Enterprise Observability umbrella (#1055). The OTLP relay design, span enrichment, egress propagation, and multi-tenancy requirements will become a dedicated tracking issue. The shared telemetry relay infrastructure (transport channel, backpressure, scale mitigations) will be a separate issue linked to both the new agent-level issue and #2508 (supervisor OTel span emission).

The branch is kept for reference.