UA-.NETStandard/docs/Kubernetes.md

# Kubernetes High Availability Deployment

This is the Kubernetes deployment guide for OPC UA high availability. It
covers the base `OPCFoundation.NetStandard.Opc.Ua.Redundancy.Server`
features and the opt-in
`OPCFoundation.NetStandard.Opc.Ua.Redundancy.Kubernetes` extension. The
extension provides Kubernetes Lease leader election, EndpointSlice peer
discovery, and HTTP readiness/liveness driven by OPC UA `ServiceLevel`.

See [HighAvailability.md](HighAvailability.md) for the OPC 10000-4 §6.6 redundancy model. The worked server sample is `samples/Redundancy/RedundantServer`.

The API prefixes distinguish standardized OPC UA model wiring from
deployment extensions:

- `AddServerRedundancy(...)`, `AddServerServiceLevel(...)`, and
  `AddRequestServerStateChange(...)` publish or maintain OPC 10000-4 §6.6
  nodes and methods.
- `UseDistributedAddressSpace(...)`, `UseDistributedSessions(...)`,
  `UseDistributedSubscriptionMirroring(...)`,
  `UseKubernetesLeaderElection(...)`, `UseKubernetesPeerDiscovery(...)`, and
  `UseKubernetesReadiness(...)` register beyond-spec extension services.

`AddServerRedundancy(...)` does not drive `Server.ServiceLevel`. For
Kubernetes readiness, also register a `ServiceLevel` provider, commonly
through `AddServerServiceLevel(...)` or the leader-aware provider used by the
sample.

## Contents

- [Redundancy shapes on Kubernetes](#redundancy-shapes-on-kubernetes)
- [Running with Kubernetes API integration](#running-with-kubernetes-api-integration)
  - [Application setup with full K8s integration](#application-setup-with-full-k8s-integration)
- [Running without Kubernetes API integration](#running-without-kubernetes-api-integration)
  - [Simple setup option](#simple-setup-option)
- [RBAC](#rbac)
- [Services](#services)
- [StatefulSet or Deployment](#statefulset-or-deployment)
- [Readiness and ServiceLevel](#readiness-and-servicelevel)
- [Kubernetes Lease leader election](#kubernetes-lease-leader-election)
- [EndpointSlice peer discovery](#endpointslice-peer-discovery)
- [Secrets, certificates, and trust](#secrets-certificates-and-trust)
- [Time synchronization](#time-synchronization)
- [GDS and NTRS registration](#gds-and-ntrs-registration)
- [Security checklist](#security-checklist)

## Redundancy shapes on Kubernetes

Use **non-transparent redundancy** when each pod has its own OPC UA
endpoint. Clients then use `Server.ServerRedundancy`, `FindServers`, and
`ServiceLevel` to fail over.

Use **transparent redundancy** when clients connect to one virtual address,
typically a `Service` or external load balancer. To hide failover, the pods
must share the endpoint URL, application URI, certificate, session state,
and subscription state.

A multi-pod deployment needs a shared store that all replicas can reach for
distributed address-space, session, and subscription state. Configure a
networked backend with authentication, encryption, and capacity limits, such
as the in-package Raft store or a CRDT gossip fabric over the pod network.
Protect the store connection with authenticated encryption. Provision the
shared record-protection key through a Kubernetes Secret or a secret store
backed by the Container Storage Interface (CSI). Set limits and expiration
for session, nonce, and subscription keys so
a faulty replica cannot exhaust the backend.

Choose one of two deployment modes:

- **With Kubernetes API integration:** use Kubernetes-native Lease election,
  EndpointSlice discovery, and readiness.
- **Without Kubernetes API integration:** the in-package Raft/CRDT layers
  manage membership and leadership.

The following sections describe both options.

## Running with Kubernetes API integration

Use the `Opc.Ua.Redundancy.Kubernetes` extension (`UseKubernetesLeaderElection`, `UseKubernetesPeerDiscovery`, `UseKubernetesReadiness`) when you want Kubernetes-native Lease leader election, EndpointSlice peer discovery, and `ServiceLevel`-driven HTTP readiness. This option calls the Kubernetes API, so it needs a `ServiceAccount` and the [RBAC](#rbac) shown below.

### Application setup with "full" K8s integration

```csharp
services.AddOpcUa()
    .AddServer(server =>
    {
        // endpoint, certificate, and application configuration
    })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseDistributedAddressSpace(options =>
    {
        options.UseLeaderElection = true;
        options.LeaseDuration = TimeSpan.FromSeconds(30);
        options.RenewInterval = TimeSpan.FromSeconds(10);
        options.NodeId = Environment.GetEnvironmentVariable("HOSTNAME") ?? Environment.MachineName;
        // This wires the leader-aware IServiceLevelProvider that UseKubernetesReadiness() consumes.
    })
    .UseDistributedSessions(options =>
    {
        options.EnableFastReconnect = true;
    })
    .UseDistributedSubscriptionMirroring()
    .AddServerRedundancy(options =>
    {
        options.Mode = RedundancySupport.Transparent;
        options.CurrentServerId = Environment.GetEnvironmentVariable("HOSTNAME") ?? Environment.MachineName;
    })
    .UseKubernetesLeaderElection(options =>
    {
        options.LeaseName = "opcua-ha-leader";
        options.Kubernetes.NodeId = Environment.GetEnvironmentVariable("HOSTNAME") ?? Environment.MachineName;
    })
    .UseKubernetesPeerDiscovery(options =>
    {
        options.ServiceName = "opcua-ha-headless";
        options.LocalAddress = Environment.GetEnvironmentVariable("POD_IP");
    })
    .UseKubernetesReadiness(options =>
    {
        options.Port = KubernetesReadinessOptions.DefaultPort; // 4852
        options.ReadinessPath = "/readyz";
        options.LivenessPath = "/livez";
        options.ReadyMinimumServiceLevel = ServiceLevels.HealthyMinimum;
    });
```

The example is safe because
`UseDistributedAddressSpace(... options.UseLeaderElection = true ...)`
registers a leader-aware `IServiceLevelProvider`.

If you combine `UseKubernetesLeaderElection()` and
`UseKubernetesReadiness()` without `UseDistributedAddressSpace()`, register
your own `IServiceLevelProvider`, for example through
`AddServerServiceLevel(...)`, before `UseKubernetesReadiness()`. This keeps
followers from remaining ready.

`UseKubernetesLeaderElection` uses the in-cluster service account token, namespace, and CA mounted at `/var/run/secrets/kubernetes.io/serviceaccount`. Outside Kubernetes it falls back to `SharedStoreLeaseElection` by default (`UseSharedStoreFallback = true`) so local development can still exercise the distributed path. (`UseKubernetes()` registers the shared in-cluster Kubernetes API client that these extensions consume; the leader-election, peer-discovery, and readiness helpers add it automatically, but you can call it directly when adding your own Kubernetes-backed component.)

## Running without Kubernetes API integration

The `Opc.Ua.Redundancy.Kubernetes` extension is **optional**. Because the in-package consensus layers self-manage membership and leadership, you can run a fully working HA replica set on Kubernetes with **no Kubernetes API access, RBAC, or ServiceAccount** at all.

### Simple setup option

- **Strong** consistency (via Raft) — `UseRedundancyConsistency(RedundancyConsistencyMode.Strong)` elects a single leader through the Raft protocol itself and provides a linearizable shared store, so no Kubernetes Lease is needed. Deploy the members as a `StatefulSet` with a headless `Service`; each pod derives its Raft node id from the StatefulSet ordinal and reaches its peers by stable pod DNS (`pod-0.svc`, `pod-1.svc`, …). A `RaftCs.Storage.File` WAL on a `PersistentVolumeClaim` lets a restarted pod rejoin without a full snapshot.
- **Eventual** consistency (via CRDT and Raft) — `UseReplicatedAddressSpace`/`UseReplicatedSessions` converge leaderlessly over gossip, so there is no election to coordinate. Point each pod's gossip peer list at the other pods' DNS names (again a headless `Service` over a `StatefulSet`). Exactly-once primitives such as the single-use nonce and the leader lease still ride the in-package Raft keyspace, so "eventual" here is the hybrid CRDT-plus-Raft store rather than pure gossip.

In both cases readiness/liveness can be a plain HTTP probe your app exposes from `Server.ServiceLevel` without `UseKubernetesReadiness`.

## RBAC

Grant only namespace-scoped access to Leases for election and EndpointSlices for peer discovery.

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: opcua
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: opcua-ha
  namespace: opcua
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: opcua-ha
  namespace: opcua
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update", "patch", "delete"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: opcua-ha
  namespace: opcua
subjects:
  - kind: ServiceAccount
    name: opcua-ha
    namespace: opcua
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: opcua-ha
```

## Services

Use a headless Service for stable pod DNS and EndpointSlice peer discovery. Use a separate regular Service for a transparent virtual address or for client access to ready pods.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: opcua-ha-headless
  namespace: opcua
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: opcua-ha
  ports:
    - name: opcua-tcp
      port: 4840
      targetPort: opcua-tcp
---
apiVersion: v1
kind: Service
metadata:
  name: opcua-ha
  namespace: opcua
spec:
  type: LoadBalancer
  selector:
    app: opcua-ha
  ports:
    - name: opcua-tcp
      port: 4840
      targetPort: opcua-tcp
```

For clusters without external load balancers, use `ClusterIP` and place an ingress, gateway, node-level load balancer, or OT/DMZ routing appliance in front. For transparent redundancy, align the OPC UA `ApplicationUri`, endpoint URL, certificate subject alternative names, and trust configuration with the externally visible virtual address.

## StatefulSet or Deployment

A StatefulSet gives stable ordinal names for per-replica endpoints and audit identity. A Deployment is suitable when clients and certificates only use the virtual Service address and no peer URI depends on pod names.

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: opcua-ha
  namespace: opcua
spec:
  serviceName: opcua-ha-headless
  replicas: 3
  selector:
    matchLabels:
      app: opcua-ha
  template:
    metadata:
      labels:
        app: opcua-ha
    spec:
      serviceAccountName: opcua-ha
      terminationGracePeriodSeconds: 30
      containers:
        - name: server
          image: ghcr.io/example/opcua-ha:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: HA_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: HA_FAST_RECONNECT
              value: "true"
            - name: HA_RECORD_KEY
              valueFrom:
                secretKeyRef:
                  name: opcua-ha-kek
                  key: record-key
          ports:
            - name: opcua-tcp
              containerPort: 4840
            - name: health
              containerPort: 4852
          readinessProbe:
            httpGet:
              path: /readyz
              port: health
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /livez
              port: health
            periodSeconds: 10
            failureThreshold: 3
```

A Deployment variant is nearly identical: replace `kind: StatefulSet` with `kind: Deployment`, remove `serviceName`, and keep the same labels, probes, service account, and ports.

## Readiness and ServiceLevel

`UseKubernetesReadiness` starts a small HTTP listener on port 4852 (`KubernetesReadinessOptions.DefaultPort`) when the server has started. The default deliberately avoids the ports web applications commonly use, such as 8080 or 9090: the probe shares the pod (and, with host networking, the node) with the application and its sidecars, so such a default collides with them. 4852 sits next to the OPC UA ports 4840/4843 and is unassigned in the IANA service name and port number registry; set `Port` if it is taken in your environment. `/livez` returns success while the process can answer the probe. `/readyz` returns HTTP 200 only when `IServiceLevelProvider.GetServiceLevel()` is at least `ReadyMinimumServiceLevel`, which defaults to the Healthy sub-range (`200`). This lets a client-facing Service route only to the current leader or otherwise healthy replicas.

For Warm active/passive deployments, `LeaderServiceLevelProvider` normally makes the leader Healthy and standbys Degraded; readiness therefore routes traffic to the leader. For Hot load-balanced deployments, use the Healthy 200-255 sub-range to reflect load and keep all healthy pods ready.

## Kubernetes Lease leader election

`UseKubernetesLeaderElection` replaces the registered `ILeaderElection` with `KubernetesLeaseLeaderElection` when running in cluster. The Lease name defaults to `opcua-server-leader`; tune `LeaseDuration` and `RenewInterval` above worst-case scheduling, API-server, and network jitter. Outside Kubernetes, the extension can fall back to the base `SharedStoreLeaseElection` for local testing.

Leader election controls the writer role in `UseDistributedAddressSpace` and can drive `ServiceLevel` through `LeaderServiceLevelProvider`. A graceful shutdown should release the lease and give Kubernetes enough `terminationGracePeriodSeconds` to remove readiness before the pod exits.

For a strongly consistent alternative, `UseRedundancyConsistency` registers a native Raft `ILeaderElection` whose leadership is decided by the Raft consensus protocol itself (one leader per term, no split-brain) instead of a Kubernetes Lease or a lease-CAS. Wire the cluster with `UseKubernetesRaftConsensus` (in `Opc.Ua.Redundancy.Kubernetes`): run the replicas as a StatefulSet with stable network identities and an odd member count (3 or 5) for a fault-tolerant quorum. It maps each pod's StatefulSet ordinal to a Raft node id, resolves peers through the headless Service DNS (`{statefulset}-{i}.{headless}`) over a `NanoMsgBusTransport`, and (by default) places a crash-safe `RaftCs.Storage.File` WAL on a per-pod PersistentVolume so a restarted pod rejoins from its log rather than a full snapshot. Compose it with `UseRedundancyConsistency` so the shared store and election use the cluster:

```csharp
services.AddOpcUa()
    .AddServer(server => { })
    .UseKubernetesRaftConsensus(o =>
    {
        o.HeadlessServiceName = "opcua-ha-headless";
        o.ReplicaCount = 3;                 // static membership; odd for quorum
        o.StoragePath = "/var/lib/opcua/raft"; // PersistentVolume mount
    })
    .UseRedundancyConsistency(RedundancyConsistencyMode.Strong)
    .UseDistributedAddressSpace()
    .UseDistributedSessions(s => s.EnableFastReconnect = true);
```

See [Consistency modes](HighAvailability.md) for the strong (Raft) vs. eventual (CRDT) trade-offs.

## EndpointSlice peer discovery

`UseKubernetesPeerDiscovery` polls EndpointSlices for the configured headless Service and builds peer URIs from address, port name, URI scheme, and port options. Set `LocalAddress` to the pod IP or DNS name so the local pod is excluded. The discovered ServerUris are bridged into the generic `IPeerDiscovery` seam, so they update the client-facing `RedundantServerSet` (published through `FindServers`) **dynamically at runtime** — no manual feeding into `ServerRedundancyOptions.RedundantPeers` is required, and static configuration remains the fallback until discovery finds peers (see [High Availability → Dynamic peer discovery](HighAvailability.md#dynamic-peer-discovery-beyond-66-opt-in)). EndpointSlice discovery yields OPC UA endpoints, not CRDT gossip endpoints; for an active/active gossip fabric that also scales dynamically, pair it with `UseDnsPeerDiscovery` on the gossip headless service.

## Secrets, certificates, and trust

Provision the shared record-protection key as a Secret or, preferably, through the Secrets Store CSI driver backed by a KMS.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: opcua-ha-kek
  namespace: opcua
type: Opaque
data:
  record-key: <base64-of-32-or-more-random-bytes>
```

For transparent redundancy or session fast reconnect, mount the same `ApplicationInstanceCertificate` and private key into every replica because the client signs an `ActivateSession` challenge against one logical application. For non-transparent redundancy, each server may have its own ApplicationUri and certificate and is managed independently by the certificate manager. Never bake private keys, trust lists, or shared store credentials into images.

The shared transparent-mode private key has replica-set-wide blast radius: compromise of one pod or mounted secret compromises the virtual OPC UA server identity for every replica behind the Service. Store the key in a Kubernetes Secret only when that matches the cluster's threat model; otherwise use a CSI-backed secret store or HSM/KMS integration with strict RBAC, audit, and node access controls.

Rotate the shared transparent-mode certificate/key with an overlap window. Issue a replacement certificate for the same virtual endpoint identity, update client/GDS trust lists to trust the replacement before rollout, update the mounted secret or CSI version, restart or reload pods until every ready replica presents the new certificate, then revoke/remove the old certificate and delete old secret versions. For suspected key compromise, mark readiness unhealthy or drain the Service before any replica with the old key can continue serving clients.

## Time synchronization

OPC 10000-4 requires redundant server sets to be time synchronized. Kubernetes nodes should run NTP or PTP, monitor drift, and keep Lease durations comfortably above expected clock and scheduling jitter. Certificate validity, SecureChannel token validation, audit records, historian timestamps, SourceTimestamp/ServerTimestamp comparisons, and EventId strategies all depend on sane clocks.

## GDS and NTRS registration

For non-transparent redundancy, register each server with GDS and include the `NTRS` ServerCapability so clients and Network Topology and Routing Services can discover the redundant set. `AddServerRedundancy(...)` advertises `NTRS` by default for Cold/Warm/Hot/HotAndMirrored modes. For transparent redundancy, register the virtual application/endpoint that clients use; replicas can still have operational identities for diagnostics, but discovery should return the stable virtual address.

## Security checklist

- Restrict RBAC to the namespace and resources above; add NetworkPolicy so only the OPC UA pods that need it can reach the Kubernetes API and shared store.
- Protect shared-store traffic with mutual TLS or an equivalent authenticated channel and fail closed if the store is unavailable for state that is required by the selected redundancy mode.
- Use `IRecordProtector` with a key ring for shared records; rotate keys without writing plaintext records to the backend.
- Keep `UseDistributedSessions(... EnableFastReconnect = true)` opt-in and validate that single-use nonce storage is strongly consistent across replicas.
- Apply TTLs, eviction, and keyspace quotas for session, nonce, subscription, and retransmission keys.
- Run readiness from `ServiceLevel` rather than a raw TCP probe when clients should only reach the active or healthy server.