Controller OOMKilled in mixed-version raft read-only mode on 2.1.0 — ~5x memory regression vs 2.0.2

[bjornmage]

Summary

Upgrading one node of a 3-node OpenZiti HA controller cluster from 2.0.2 to the current main (2.1.0) controller build causes the upgraded follower to be OOMKilled repeatedly while the existing mesh holds it in mixed-version raft read-only mode. The 2.1.0 image climbs to the 1 GiB container limit in ~17 seconds — roughly a 5x memory regression versus the healthy 2.0.2 baseline.

Environment

Deployment	3-node HA controller cluster (raft), private edge routers
Baseline	All three nodes on openziti/ziti-controller:2.0.2 (deployed from a prerelease tag)
Upgraded image	main (2.1.0) controller build
Container memory limit	1 GiB
Healthy baseline memory	~150–190 MiB steady per controller

Upgrade scenario

Rolling, node-by-node upgrade (the standard HA pattern):

1. Node A is upgraded to the 2.1.0 image and starts.
2. Node A joins the existing mesh. Nodes B and C log the mixed-version read-only entry:
   peer <A> has version v2.1.0 be3dc38b1507 2026-08-04T03:24:44Z linux amd64, not matching local version, entering read-only mode
3. Within ~7–18 seconds Node A is OOMKilled (exit 137).
4. Nodes B and C log Node A exiting read-only mode as it dies.
5. The pod restarts and repeats the identical cycle.

Observed behavior

• Time-to-OOM: ~17 seconds from start in the longest cycle; ~7–18 s across cycles.
• Restarts: 4+ in the upgrade attempt window. All 5 boot cycles followed the identical pattern: join mesh → peers log read-only entry → OOMKilled → peers log read-only exit.
• Memory: the 2.1.0 follower grows from near-zero to the 1 GiB limit in ~17 s. Healthy 2.0.2 controllers sit at ~150–190 MiB. That is roughly a 5x memory regression, or an allocation storm, in the 2.1.0 serving path.

Datastore context

Before mesh join, the 2.1.0 run one-way-migrated the edge datastore v44 → v47. The migration completed successfully; the OOM occurs in the post-migration serving phase, not during migration. The datastore is now at v47 and cannot be downgraded — relevant for anyone considering a rollback.

Regression localization

The regression is not in the v2.1.0-pre1 → main delta. All 10 commits between them are unrelated (a sync comment, edge-ctrl error reporting, access checks, the ConnectionTracker reap refactor #4206, router-link reroute ordering #4079, route tag preservation #4124) and there is no go.mod/go.sum change. So the regression lives in the 2.0.2 → 2.1.0 delta and is shared by both pre1 and main.

Honest caveat

We could not reproduce this on a stable single-version cluster from this vantage, and the OOM'd pod's heap logs were not retained, so we cannot point to a specific allocation site. The mixed-version read-only raft serving path is the strong circumstantial suspect — the failure only ever occurred on the node being held in read-only mode — but this is not proven. Treat this as a request for the maintainers to investigate that path, not a definitive root-cause claim.

Not a duplicate

We found no existing issue for a controller OOM in 2.1.0. #4184 is the closest match but is router-only with a different mechanism (a LinkSendBuffer.drainDeadlines() goroutine leak in xgress on routers), unrelated to controller read-only serving.

What we're asking

1. Investigate the mixed-version raft read-only serving path on the controller for an allocation storm or unbounded growth introduced between 2.0.2 and 2.1.0.
2. Confirm whether a full-cluster (no-mixed-version) upgrade — taking down the raft cluster, upgrading all nodes, bringing them back together — avoids the condition, and whether that is the recommended migration path for a 2.0.x → 2.1.0 controller upgrade.

Happy to provide further diagnostics (redacted configs, logs, container resource metrics) on request.