UA-.NETStandard/docs/Kubernetes.md

Kubernetes High Availability Deployment

This is the Kubernetes deployment guide for OPC UA high availability. It covers the base OPCFoundation.NetStandard.Opc.Ua.Redundancy.Server features and the opt-in OPCFoundation.NetStandard.Opc.Ua.Redundancy.Kubernetes extension. The extension provides Kubernetes Lease leader election, EndpointSlice peer discovery, and HTTP readiness/liveness driven by OPC UA ServiceLevel.

See HighAvailability.md for the OPC 10000-4 §6.6 redundancy model. The worked server sample is samples/Redundancy/RedundantServer.

The API prefixes distinguish standardized OPC UA model wiring from deployment extensions:

• AddServerRedundancy(...), AddServerServiceLevel(...), and AddRequestServerStateChange(...) publish or maintain OPC 10000-4 §6.6 nodes and methods.
• UseDistributedAddressSpace(...), UseDistributedSessions(...), UseDistributedSubscriptionMirroring(...), UseKubernetesLeaderElection(...), UseKubernetesPeerDiscovery(...), and UseKubernetesReadiness(...) register beyond-spec extension services.

AddServerRedundancy(...) does not drive Server.ServiceLevel. For Kubernetes readiness, also register a ServiceLevel provider, commonly through AddServerServiceLevel(...) or the leader-aware provider used by the sample.

Contents

• Redundancy shapes on Kubernetes
• Running with Kubernetes API integration
  • Application setup with full K8s integration
• Running without Kubernetes API integration
  • Simple setup option
• RBAC
• Services
• StatefulSet or Deployment
• Readiness and ServiceLevel
• Kubernetes Lease leader election
• EndpointSlice peer discovery
• Secrets, certificates, and trust
• Time synchronization
• GDS and NTRS registration
• Security checklist

Redundancy shapes on Kubernetes

Use non-transparent redundancy when each pod has its own OPC UA endpoint. Clients then use Server.ServerRedundancy, FindServers, and ServiceLevel to fail over.

Use transparent redundancy when clients connect to one virtual address, typically a Service or external load balancer. To hide failover, the pods must share the endpoint URL, application URI, certificate, session state, and subscription state.

A multi-pod deployment needs a shared store that all replicas can reach for distributed address-space, session, and subscription state. Configure a networked backend with authentication, encryption, and capacity limits, such as the in-package Raft store or a CRDT gossip fabric over the pod network. Protect the store connection with authenticated encryption. Provision the shared record-protection key through a Kubernetes Secret or a secret store backed by the Container Storage Interface (CSI). Set limits and expiration for session, nonce, and subscription keys so a faulty replica cannot exhaust the backend.

Choose one of two deployment modes:

• With Kubernetes API integration: use Kubernetes-native Lease election, EndpointSlice discovery, and readiness.
• Without Kubernetes API integration: the in-package Raft/CRDT layers manage membership and leadership.

The following sections describe both options.

Running with Kubernetes API integration

Use the Opc.Ua.Redundancy.Kubernetes extension (UseKubernetesLeaderElection, UseKubernetesPeerDiscovery, UseKubernetesReadiness) when you want Kubernetes-native Lease leader election, EndpointSlice peer discovery, and ServiceLevel-driven HTTP readiness. This option calls the Kubernetes API, so it needs a ServiceAccount and the RBAC shown below.

Application setup with "full" K8s integration

services.AddOpcUa()
    .AddServer(server =>
    {
        // endpoint, certificate, and application configuration
    })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseDistributedAddressSpace(options =>
    {
        options.UseLeaderElection = true;
        options.LeaseDuration = TimeSpan.FromSeconds(30);
        options.RenewInterval = TimeSpan.FromSeconds(10);
        options.NodeId = Environment.GetEnvironmentVariable("HOSTNAME") ?? Environment.MachineName;
        // This wires the leader-aware IServiceLevelProvider that UseKubernetesReadiness() consumes.
    })
    .UseDistributedSessions(options =>
    {
        options.EnableFastReconnect = true;
    })
    .UseDistributedSubscriptionMirroring()
    .AddServerRedundancy(options =>
    {
        options.Mode = RedundancySupport.Transparent;
        options.CurrentServerId = Environment.GetEnvironmentVariable("HOSTNAME") ?? Environment.MachineName;
    })
    .UseKubernetesLeaderElection(options =>
    {
        options.LeaseName = "opcua-ha-leader";
        options.Kubernetes.NodeId = Environment.GetEnvironmentVariable("HOSTNAME") ?? Environment.MachineName;
    })
    .UseKubernetesPeerDiscovery(options =>
    {
        options.ServiceName = "opcua-ha-headless";
        options.LocalAddress = Environment.GetEnvironmentVariable("POD_IP");
    })
    .UseKubernetesReadiness(options =>
    {
        options.Port = KubernetesReadinessOptions.DefaultPort; // 4852
        options.ReadinessPath = "/readyz";
        options.LivenessPath = "/livez";
        options.ReadyMinimumServiceLevel = ServiceLevels.HealthyMinimum;
    });

The example is safe because UseDistributedAddressSpace(... options.UseLeaderElection = true ...) registers a leader-aware IServiceLevelProvider.

If you combine UseKubernetesLeaderElection() and UseKubernetesReadiness() without UseDistributedAddressSpace(), register your own IServiceLevelProvider, for example through AddServerServiceLevel(...), before UseKubernetesReadiness(). This keeps followers from remaining ready.

UseKubernetesLeaderElection uses the in-cluster service account token, namespace, and CA mounted at /var/run/secrets/kubernetes.io/serviceaccount. Outside Kubernetes it falls back to SharedStoreLeaseElection by default (UseSharedStoreFallback = true) so local development can still exercise the distributed path. (UseKubernetes() registers the shared in-cluster Kubernetes API client that these extensions consume; the leader-election, peer-discovery, and readiness helpers add it automatically, but you can call it directly when adding your own Kubernetes-backed component.)

Running without Kubernetes API integration

The Opc.Ua.Redundancy.Kubernetes extension is optional. Because the in-package consensus layers self-manage membership and leadership, you can run a fully working HA replica set on Kubernetes with no Kubernetes API access, RBAC, or ServiceAccount at all.

Simple setup option

• Strong consistency (via Raft) — UseRedundancyConsistency(RedundancyConsistencyMode.Strong) elects a single leader through the Raft protocol itself and provides a linearizable shared store, so no Kubernetes Lease is needed. Deploy the members as a StatefulSet with a headless Service; each pod derives its Raft node id from the StatefulSet ordinal and reaches its peers by stable pod DNS (pod-0.svc, pod-1.svc, …). A RaftCs.Storage.File WAL on a PersistentVolumeClaim lets a restarted pod rejoin without a full snapshot.
• Eventual consistency (via CRDT and Raft) — UseReplicatedAddressSpace/UseReplicatedSessions converge leaderlessly over gossip, so there is no election to coordinate. Point each pod's gossip peer list at the other pods' DNS names (again a headless Service over a StatefulSet). Exactly-once primitives such as the single-use nonce and the leader lease still ride the in-package Raft keyspace, so "eventual" here is the hybrid CRDT-plus-Raft store rather than pure gossip.

In both cases readiness/liveness can be a plain HTTP probe your app exposes from Server.ServiceLevel without UseKubernetesReadiness.

RBAC

Grant only namespace-scoped access to Leases for election and EndpointSlices for peer discovery.

apiVersion: v1
kind: Namespace
metadata:
  name: opcua
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: opcua-ha
  namespace: opcua
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: opcua-ha
  namespace: opcua
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update", "patch", "delete"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: opcua-ha
  namespace: opcua
subjects:
  - kind: ServiceAccount
    name: opcua-ha
    namespace: opcua
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: opcua-ha

Services

Use a headless Service for stable pod DNS and EndpointSlice peer discovery. Use a separate regular Service for a transparent virtual address or for client access to ready pods.

apiVersion: v1
kind: Service
metadata:
  name: opcua-ha-headless
  namespace: opcua
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: opcua-ha
  ports:
    - name: opcua-tcp
      port: 4840
      targetPort: opcua-tcp
---
apiVersion: v1
kind: Service
metadata:
  name: opcua-ha
  namespace: opcua
spec:
  type: LoadBalancer
  selector:
    app: opcua-ha
  ports:
    - name: opcua-tcp
      port: 4840
      targetPort: opcua-tcp

For clusters without external load balancers, use ClusterIP and place an ingress, gateway, node-level load balancer, or OT/DMZ routing appliance in front. For transparent redundancy, align the OPC UA ApplicationUri, endpoint URL, certificate subject alternative names, and trust configuration with the externally visible virtual address.

StatefulSet or Deployment

A StatefulSet gives stable ordinal names for per-replica endpoints and audit identity. A Deployment is suitable when clients and certificates only use the virtual Service address and no peer URI depends on pod names.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: opcua-ha
  namespace: opcua
spec:
  serviceName: opcua-ha-headless
  replicas: 3
  selector:
    matchLabels:
      app: opcua-ha
  template:
    metadata:
      labels:
        app: opcua-ha
    spec:
      serviceAccountName: opcua-ha
      terminationGracePeriodSeconds: 30
      containers:
        - name: server
          image: ghcr.io/example/opcua-ha:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: HA_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: HA_FAST_RECONNECT
              value: "true"
            - name: HA_RECORD_KEY
              valueFrom:
                secretKeyRef:
                  name: opcua-ha-kek
                  key: record-key
          ports:
            - name: opcua-tcp
              containerPort: 4840
            - name: health
              containerPort: 4852
          readinessProbe:
            httpGet:
              path: /readyz
              port: health
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /livez
              port: health
            periodSeconds: 10
            failureThreshold: 3

A Deployment variant is nearly identical: replace kind: StatefulSet with kind: Deployment, remove serviceName, and keep the same labels, probes, service account, and ports.

Readiness and ServiceLevel

UseKubernetesReadiness starts a small HTTP listener on port 4852 (KubernetesReadinessOptions.DefaultPort) when the server has started. The default deliberately avoids the ports web applications commonly use, such as 8080 or 9090: the probe shares the pod (and, with host networking, the node) with the application and its sidecars, so such a default collides with them. 4852 sits next to the OPC UA ports 4840/4843 and is unassigned in the IANA service name and port number registry; set Port if it is taken in your environment. /livez returns success while the process can answer the probe. /readyz returns HTTP 200 only when IServiceLevelProvider.GetServiceLevel() is at least ReadyMinimumServiceLevel, which defaults to the Healthy sub-range (200). This lets a client-facing Service route only to the current leader or otherwise healthy replicas.

For Warm active/passive deployments, LeaderServiceLevelProvider normally makes the leader Healthy and standbys Degraded; readiness therefore routes traffic to the leader. For Hot load-balanced deployments, use the Healthy 200-255 sub-range to reflect load and keep all healthy pods ready.

Kubernetes Lease leader election

UseKubernetesLeaderElection replaces the registered ILeaderElection with KubernetesLeaseLeaderElection when running in cluster. The Lease name defaults to opcua-server-leader; tune LeaseDuration and RenewInterval above worst-case scheduling, API-server, and network jitter. Outside Kubernetes, the extension can fall back to the base SharedStoreLeaseElection for local testing.

Leader election controls the writer role in UseDistributedAddressSpace and can drive ServiceLevel through LeaderServiceLevelProvider. A graceful shutdown should release the lease and give Kubernetes enough terminationGracePeriodSeconds to remove readiness before the pod exits.

For a strongly consistent alternative, UseRedundancyConsistency registers a native Raft ILeaderElection whose leadership is decided by the Raft consensus protocol itself (one leader per term, no split-brain) instead of a Kubernetes Lease or a lease-CAS. Wire the cluster with UseKubernetesRaftConsensus (in Opc.Ua.Redundancy.Kubernetes): run the replicas as a StatefulSet with stable network identities and an odd member count (3 or 5) for a fault-tolerant quorum. It maps each pod's StatefulSet ordinal to a Raft node id, resolves peers through the headless Service DNS ({statefulset}-{i}.{headless}) over a NanoMsgBusTransport, and (by default) places a crash-safe RaftCs.Storage.File WAL on a per-pod PersistentVolume so a restarted pod rejoins from its log rather than a full snapshot. Compose it with UseRedundancyConsistency so the shared store and election use the cluster:

services.AddOpcUa()
    .AddServer(server => { })
    .UseKubernetesRaftConsensus(o =>
    {
        o.HeadlessServiceName = "opcua-ha-headless";
        o.ReplicaCount = 3;                 // static membership; odd for quorum
        o.StoragePath = "/var/lib/opcua/raft"; // PersistentVolume mount
    })
    .UseRedundancyConsistency(RedundancyConsistencyMode.Strong)
    .UseDistributedAddressSpace()
    .UseDistributedSessions(s => s.EnableFastReconnect = true);

See Consistency modes for the strong (Raft) vs. eventual (CRDT) trade-offs.

EndpointSlice peer discovery

UseKubernetesPeerDiscovery polls EndpointSlices for the configured headless Service and builds peer URIs from address, port name, URI scheme, and port options. Set LocalAddress to the pod IP or DNS name so the local pod is excluded. The discovered ServerUris are bridged into the generic IPeerDiscovery seam, so they update the client-facing RedundantServerSet (published through FindServers) dynamically at runtime — no manual feeding into ServerRedundancyOptions.RedundantPeers is required, and static configuration remains the fallback until discovery finds peers (see High Availability → Dynamic peer discovery). EndpointSlice discovery yields OPC UA endpoints, not CRDT gossip endpoints; for an active/active gossip fabric that also scales dynamically, pair it with UseDnsPeerDiscovery on the gossip headless service.

Secrets, certificates, and trust

Provision the shared record-protection key as a Secret or, preferably, through the Secrets Store CSI driver backed by a KMS.

apiVersion: v1
kind: Secret
metadata:
  name: opcua-ha-kek
  namespace: opcua
type: Opaque
data:
  record-key: <base64-of-32-or-more-random-bytes>

For transparent redundancy or session fast reconnect, mount the same ApplicationInstanceCertificate and private key into every replica because the client signs an ActivateSession challenge against one logical application. For non-transparent redundancy, each server may have its own ApplicationUri and certificate and is managed independently by the certificate manager. Never bake private keys, trust lists, or shared store credentials into images.

The shared transparent-mode private key has replica-set-wide blast radius: compromise of one pod or mounted secret compromises the virtual OPC UA server identity for every replica behind the Service. Store the key in a Kubernetes Secret only when that matches the cluster's threat model; otherwise use a CSI-backed secret store or HSM/KMS integration with strict RBAC, audit, and node access controls.

Rotate the shared transparent-mode certificate/key with an overlap window. Issue a replacement certificate for the same virtual endpoint identity, update client/GDS trust lists to trust the replacement before rollout, update the mounted secret or CSI version, restart or reload pods until every ready replica presents the new certificate, then revoke/remove the old certificate and delete old secret versions. For suspected key compromise, mark readiness unhealthy or drain the Service before any replica with the old key can continue serving clients.

Time synchronization

OPC 10000-4 requires redundant server sets to be time synchronized. Kubernetes nodes should run NTP or PTP, monitor drift, and keep Lease durations comfortably above expected clock and scheduling jitter. Certificate validity, SecureChannel token validation, audit records, historian timestamps, SourceTimestamp/ServerTimestamp comparisons, and EventId strategies all depend on sane clocks.

GDS and NTRS registration

For non-transparent redundancy, register each server with GDS and include the NTRS ServerCapability so clients and Network Topology and Routing Services can discover the redundant set. AddServerRedundancy(...) advertises NTRS by default for Cold/Warm/Hot/HotAndMirrored modes. For transparent redundancy, register the virtual application/endpoint that clients use; replicas can still have operational identities for diagnostics, but discovery should return the stable virtual address.

Security checklist

• Restrict RBAC to the namespace and resources above; add NetworkPolicy so only the OPC UA pods that need it can reach the Kubernetes API and shared store.
• Protect shared-store traffic with mutual TLS or an equivalent authenticated channel and fail closed if the store is unavailable for state that is required by the selected redundancy mode.
• Use IRecordProtector with a key ring for shared records; rotate keys without writing plaintext records to the backend.
• Keep UseDistributedSessions(... EnableFastReconnect = true) opt-in and validate that single-use nonce storage is strongly consistent across replicas.
• Apply TTLs, eviction, and keyspace quotas for session, nonce, subscription, and retransmission keys.
• Run readiness from ServiceLevel rather than a raw TCP probe when clients should only reach the active or healthy server.