Flipping the dataplane took about forty seconds. Dealing with what it broke took three weeks.
That ratio is the honest summary of an eBPF migration, and almost nothing written about eBPF networking tells you about it. The posts are about throughput graphs and iptables rule counts. The work is about conntrack, kernel versions, asymmetric return paths and a consumer group that will not stop rebalancing.
We run multi-cluster Calico Enterprise with the eBPF dataplane across AWS regions and on-premises data centres, on a Kubernetes platform behind clearing flows on the order of $4.5 trillion a day. We hold 99.99%. I am going to describe what went wrong, because the failure modes are the transferable part.
We moved for policy evaluation cost, not for benchmark numbers
The usual case for eBPF is latency and throughput. That is not why we moved.
As Service count and NetworkPolicy count grew, two things got uncomfortable. kube-proxy service programming time grew with the number of Services, so a large deployment event produced a window where some nodes had a stale view of endpoints. And conntrack table pressure under long-lived connection counts was close enough to limits that we were tuning rather than ignoring it.
Add two more wants: real source IP preservation for workloads that make authorisation decisions on client identity, and flow-level visibility we could actually query.
The latency improvement was welcome. It was never the justification. If someone pitches you an eBPF migration on p99 numbers alone, the migration will not survive its first incident review.
Flipping the dataplane resets every established connection
The switch itself is small. Patch the Installation to linuxDataplane: BPF, then stop kube-proxy from programming anything:
kubectl patch installation.operator.tigera.io default --type=merge \ -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF"}}}'kubectl patch ds -n kube-system kube-proxy --type=merge \ -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico":"true"}}}}}'
Two commands. What they do not tell you is that the service and conntrack state moves from kernel netfilter tables into BPF maps, and existing entries do not come with it.
Every established TCP flow through the node is now unknown to the dataplane.
For request/response HTTP traffic this is close to invisible. A connection resets, the client retries, nobody files a ticket. We validated in lower environments with exactly that kind of traffic and saw nothing.
Then we did it in an environment carrying Kafka.
The rebalance that would not settle
Kafka clients hold long-lived TCP connections with mutual TLS. When the dataplane flipped, those connections reset en masse. Producers started retrying inside their delivery timeout. Consumers dropped out of their groups, which triggered a rebalance, which took long enough under load that some members timed out during the rebalance, which triggered another rebalance.
It converged. It took far longer than the forty seconds the patch took, and for a stretch the lag graphs looked like a broker failure rather than a network change.
The lesson generalises past Kafka. Any dataplane change is a connection-reset event, and your validation environment probably has no long-lived connections in it. Load generators produce short flows. Databases, message brokers, gRPC channels with keepalives and mTLS sessions with session resumption do not.
Before you flip anything, inventory what holds connections open for hours.
DSR preserves source IPs by making your return path asymmetric
bpfExternalServiceMode: DSR is how you get real client IPs for traffic arriving through an external load balancer. The return packet goes straight from the pod’s node to the client instead of back through the balancer.
In a single-cloud topology with a network load balancer, this works well.
We are hybrid. Some ingress paths traverse stateful appliances on the way to on-premises sites. A stateful device that saw the forward flow and never sees the return flow holds a half-open entry; a device that sees a return flow it has no forward record for drops it. Asymmetric routing and stateful inspection are a long-standing incompatibility, and eBPF DSR walks straight into it.
The symptom was not a clean failure. Connections established and then stalled, intermittently, on one ingress path only. We spent longer than I would like looking at application logs.
What I would do differently: start in Tunnel mode, characterise every ingress path end to end, and move individual paths to DSR deliberately. We enabled DSR cluster-wide because the documentation presents it as the better option, and for most of our paths it was. “Most” is not a deployment strategy.
MTU is the failure that presents as an application bug
Our on-premises connectivity runs over encrypted 10 Gbps links, and the clusters sit inside a hub-and-spoke design spanning more than fifty VPCs. Overlay encapsulation and encryption each take their bite out of the path MTU.
When the dataplane changed, our MTU assumption no longer matched reality on one path.
The signature is specific and worth memorising. TLS handshakes complete — those packets are small. Metadata calls succeed. Then any transfer large enough to produce full-size segments hangs until a timeout. If path MTU discovery is being blackholed by something dropping ICMP fragmentation-needed, nothing in your logs will say “MTU.”
Set it explicitly rather than inferring it:
apiVersion: operator.tigera.io/v1kind: Installationmetadata: name: defaultspec: calicoNetwork: linuxDataplane: BPF mtu: 8941
Then verify on a node rather than trusting the manifest, because a mismatch between the configured value and the interface is exactly the bug:
ip link show | grep -E 'mtu' | headcalico-node -bpf counters dump | grep -i drop
Add an explicit large-payload test to your post-migration checklist. Not a health check. A multi-megabyte transfer across every path that leaves the cluster.
Kernel version is a platform contract, and we did not have one written down
Our AWS node images ran a 5.x kernel. Our on-premises RHEL nodes ran 4.18 with vendor backports. Calico supports both, which is the detail that made this hard rather than easy, because it meant nothing failed loudly.
What happened instead is that feature availability diverged by node pool. Some acceleration options were available on one side and not the other. Behaviour that we validated on one pool did not reproduce on another, and for a while we were chasing what looked like a nondeterministic bug but was a fleet with two different dataplanes wearing the same label.
eBPF makes your kernel a first-class platform dependency. Treat it that way. Pin a kernel floor as a documented contract, gate node image promotion on it, and refuse to run a mixed fleet through a dataplane migration.
This was the single largest time sink of the project, and it was entirely self-inflicted.
Failsafe ports are load-bearing and the defaults are not sufficient
If you apply host endpoint policy with a global default-deny, you are now writing firewall rules for the node itself, including the paths the node uses to stay in the cluster.
Failsafe ports exist for this. The defaults cover the obvious ones. They did not cover everything our on-premises nodes needed, and we made one node group unreachable in a way that required out-of-band access to fix.
Declare them explicitly, including the ones specific to your environment:
apiVersion: projectcalico.org/v3kind: FelixConfigurationmetadata: name: defaultspec: bpfEnabled: true bpfExternalServiceMode: Tunnel failsafeInboundHostPorts: - { protocol: tcp, port: 22 } - { protocol: tcp, port: 6443 } - { protocol: tcp, port: 2379 } failsafeOutboundHostPorts: - { protocol: udp, port: 53 } - { protocol: tcp, port: 6443 }
Apply host endpoint policy to one node you can afford to lose, first. Every time.
Federation turns every label selector into a global object
Multi-cluster Calico is not a routing feature. It is an identity feature. Federated endpoint identity means a policy in one cluster can select workloads in another, by label.
Which means the moment you enable it, your label namespace is a security boundary, and it was almost certainly not designed to be one.
apiVersion: projectcalico.org/v3kind: RemoteClusterConfigurationmetadata: name: peer-region-bspec: clusterAccessSecret: name: peer-region-b-kubeconfig namespace: calico-system syncOptions: overlayRoutingMode: Enabled
We had an allow rule written against a selector along the lines of app == "gateway", scoped by intent to a single cluster. After federation, that selector matched endpoints in the peer cluster too, and the rule permitted more than it was written to permit.
Nothing was exploited. We found it in flow logs during a review, which is the good version of finding it.
Two fixes, and I would apply both before enabling federation rather than after. Require a cluster or region label in every selector that could match federated endpoints. Then enforce that requirement in CI with admission policy, because the convention will not hold on its own.
Writing policy against a globally shared label space, with no tooling to tell you what a selector will match after federation, is the sharpest edge in multi-cluster Calico. Budget review time for it.
eBPF flow logs will outgrow your logging budget before they finish being useful
The visibility is genuinely good. The volume is genuinely a problem.
Default aggregation produces far more records than netfilter-based logging did, because you are now seeing things you previously could not. We had to tune aggregation and retention within the first week rather than treating it as a later optimisation. Our ingest estimate was off by roughly an order of magnitude — illustrative, since it depends entirely on your flow mix.
Decide up front which flows you need at full fidelity for compliance evidence and which you need only in aggregate, and set flowLogsFileAggregationKindForAllowed accordingly. Full fidelity on denied flows. Aggregate on allowed. Revisit when an investigation shows you what you actually wanted.
tcpdump is no longer telling you the truth
The debugging habit that costs the most time is reaching for tcpdump and trusting what comes back. In eBPF mode, traffic can be redirected before it reaches the point your capture is attached to, so a missing packet in a capture is not evidence of a missing packet.
Use the dataplane’s own view:
calico-node -bpf policy dump eth0 allcalico-node -bpf conntrack dumpcalico-node -bpf counters dump
The counters are where the answer usually is. A drop counter incrementing against a specific reason tells you more in ten seconds than an hour of packet captures from the wrong vantage point.
Migrate by node group, not by cluster
If I ran this again, I would not flip a cluster’s dataplane in place. I would build a new node group already in eBPF mode, cordon and drain the old group, and shift workloads across under pod disruption budgets.
That changes the character of the whole exercise. Connection resets become a rolling, paced event instead of a cluster-wide one. Kernel skew gets eliminated by construction, because the new group has one image. And the rollback is “cordon the new group and drain back,” which is a procedure rather than a second migration under pressure.
We flipped in place because the documentation describes it as supported, and it is. Supported and reversible are different properties, and on a platform where an outage is a market event, only the second one matters.
The other thing I would change is smaller and more cultural. We treated this as a networking project. It is a connection-lifecycle project, and the people who needed to be in the room early were the ones who own the brokers, the databases and the long-lived client pools — not the ones who own the CNI.
