Cloud Native Architecture – Public Sector

Architectures: GitOps-Native Multi-Cluster Kubernetes Platform for EU Data Sovereignty

Thu, 11 Jun 2026 00:00:00 +0000

Relevant Projects

Kubernetes

Using since: 2019
Current version: 1.33.x

The runtime substrate for all workloads. Cluster API manages the lifecycle of every cluster across Hetzner bare metal, AWS, and Azure declaratively from Git. No cloud console required.

Argo CD

Using since: 2021
Current version: v3.x

The reconciliation engine for the entire fleet. A fix committed once to the shared KubeAid chart library propagates to every cluster on the next ArgoCD sync no manual per-cluster patching.

Prometheus

Using since: 2020
Current version: v3.x

Generated per-cluster from a single Jsonnet vars file using kube-prometheus. Custom alerting rules for TLS expiry forecasting, backup SLIs, database replication lag, and Kubernetes ListWatch failures compose with upstream rule libraries without YAML merge conflicts.

Cilium

Using since: 2022
Current version: v1.17.x

Network policies are embedded alongside every workload manifest in KubeAid. FQDN egress rules for every external API dependency ship with the chart. Operators do not write network policies — KubeAid writes them once and applies them to every cluster.

cert-manager

Using since: 2021
Current version: v1.17.x

Defaults to DNS-01 ACME challenge validation with cloud-specific IAM scoping: IRSA on AWS, Workload Identity on Azure, API tokens on Hetzner. A CertificateNotReady alerting rule and external TLS expiry probe ship as KubeAid defaults added after a silent TLS expiry failure in production.

Velero

Using since: 2021
Current version: v1.15.x

Backup schedules, IAM policies, and object storage (S3, GCS, Azure Blob) are provisioned at cluster bootstrap. Velero also exports the Sealed Secrets private key to external object storage at bootstrap — making DR proven at provision time, not during a drill.

Cluster API

Using since: 2022
Current version: v1.10.x

Manages cluster lifecycle declaratively across Hetzner bare metal, AWS, and Azure. MachineHealthChecks detect and replace unhealthy nodes automatically. Cluster topology is version-controlled in Git.

Helm

Using since: 2019
Current version: v3.x

Over 100 curated Helm charts form the KubeAid chart library. Each chart ships with pre-wired Cilium network policies, Prometheus alerting rules, and ArgoCD ignoreDifferences configurations. Customers override only genuine differences.

TLDR; Synopsis

This reference architecture describes how Obmondo operates production Kubernetes for dozens of customer clusters across four cloud providers and bare metal with a team of under 10 engineers using a two-repo GitOps pattern built entirely on CNCF projects.

The core insight: every production failure fixed once in the shared platform repo (KubeAid) propagates to every cluster on the next ArgoCD sync. No cluster is ever patched manually. No cluster becomes a snowflake.

This architecture targets:

Zero snowflake clusters — a fix committed once applies everywhere within hours via ArgoCD
Full EU data sovereignty — identical stack on Hetzner bare metal in Germany, on-premises datacentres, and EU-region cloud VMs, with no vendor control plane, no proprietary APIs, and full auditability
Disaster recovery by default — backup schedules, IAM, and Sealed Secrets key export are automated at cluster bootstrap; if a cluster can be created, it can be recovered
Sub-10-engineer fleet operations — platform-level abstractions that scale cluster count without scaling headcount

Organization

Obmondo (EnableIT ApS) is a Danish Managed Kubernetes provider. It builds and operates production Kubernetes platforms for customers in financial services, healthcare, and public sector organizations across Denmark and the EU. All infrastructure must satisfy GDPR, NIS2, and in many cases ISO 27001 — by architecture, not by policy.

KubeAid, the open-source platform that powers Obmondo’s managed service, is available at https://github.com/Obmondo/KubeAid under the Apache 2.0 license.

Teams

Platform Engineering maintains KubeAid the shared chart library, Jsonnet monitoring templates, and ArgoCD application definitions that apply uniformly to every cluster. A fix here reaches every customer automatically.

SRE / Customer Operations handles day-2 operations: incident response, capacity planning, and customer-specific overrides in per-customer config repos. With KubeAid abstracting the platform layer, this team focuses exclusively on the ~10% that is genuinely different per customer.

Architecture

Goals

Eliminate snowflake clusters. Every production failure becomes a KubeAid default. The silent TLS expiry became DNS-01 + external probe + alert. The duplicate Prometheus timestamp alert became a scrape-level relabeling rule. The RBAC gap became a ClusterRole fix. Each fix landed once; every cluster received it automatically.

EU data sovereignty without vendor lock-in. No proprietary control plane. No per-node SaaS fees. No APIs that cannot be audited. The entire stack runs on CNCF projects identically on Hetzner bare metal in Germany, on-premises in Danish datacentres, and in EU-region cloud VMs. Customers can move it, fork it, and audit every component.

Disaster recovery proven at provision time. Backup schedules, IAM policies, object storage, and Sealed Secrets private key export are automated at bootstrap. The DR gap was discovered simultaneously on every cluster during a planned drill and made impossible to miss on any future cluster.

Observability without per-cluster overhead. Monitoring configuration is generated from a single Jsonnet vars file. One engineer maintains alerting for the entire fleet. Custom rules compose with upstream libraries without merge conflicts.

Architecture Overview

The architecture follows a strict two-repo pattern: Repo 1 — KubeAid (shared platform defaults) Over 100 curated Helm charts with pre-wired integrations. ArgoCD ApplicationSets deploy these charts to every cluster. Monitoring is generated per-cluster from a single Jsonnet vars file using kube-prometheus. Cilium network policies ship alongside workload manifests. DR configuration is provisioned at bootstrap. Every production failure becomes a default here.

Repo 2 — Customer config (genuine differences only) Each customer config repo holds only what is genuinely different: cloud provider, node sizes, alerting thresholds, compliance scope. If a value matches the KubeAid default, it does not exist in the config repo. ArgoCD reconciles both repos continuously across every cluster.

Per-cluster stack (all CNCF projects):

Layer	Project	Role
Lifecycle	Cluster API	Declarative cluster provisioning across all providers
Networking	Cilium	CNI + FQDN egress policies + Hubble metrics
TLS	cert-manager	DNS-01 ACME with cloud IAM scoping
GitOps	ArgoCD	Continuous reconciliation from both repos
Observability	Prometheus + Alertmanager	Generated from Jsonnet per cluster
Ingress	Traefik	Wildcard TLS, HTTP→HTTPS redirect
Backup / DR	Velero	Scheduled backups + Sealed Secrets key export
Secrets	Sealed Secrets (Bitnami)	Encrypted secrets in Git, key exported at bootstrap

Infrastructure Layer

Cluster API manages cluster lifecycle across all providers. The same declarative topology definition works on Hetzner bare metal (HCloud provider), AWS, and Azure. MachineHealthChecks detect unhealthy nodes and trigger automatic replacement. Node removal by CAPI is tracked and DaemonSet ghost pods are handled via alerting.

On Hetzner bare metal, clusters run on dedicated physical servers with ZFS and Ceph storage. Network isolation is enforced by Cilium at the pod level. No hypervisor, no shared infrastructure with other tenants.

Networking Layer

Cilium is the CNI for every cluster. Every KubeAid Helm chart ships with a CiliumNetworkPolicy alongside the workload manifest. FQDN egress rules for every external API dependency are pre-written and applied by ArgoCD. Operators do not write network policies they are a platform default.

FQDN policy enforcement is critical for compliance: customers need to attest that workloads only communicate with approved external endpoints. Cilium’s Hubble metrics feed into Prometheus for network observability.

TLS and Certificate Management

cert-manager defaults to DNS-01 ACME challenge validation. The switch from HTTP-01 was forced by a production failure: Traefik’s global HTTP→HTTPS redirect for HSTS compliance made HTTP-01 ACME validation permanently impossible every challenge request was redirected before reaching the solver pod. Certificates failed to renew silently for up to 90 days.

DNS-01 with cloud-specific IAM scoping (IRSA on AWS, Workload Identity on Azure, API tokens on Hetzner) is now the only supported challenge type. An external TLS expiry probe and a CertificateNotReady Alertmanager rule ship as KubeAid defaults.

Observability Layer

Prometheus and Alertmanager are generated per-cluster from a single Jsonnet vars file using kube-prometheus. Custom alerting rules compose with upstream rule libraries using Jsonnet merging semantics no YAML merge conflicts.

Custom rules added after production failures:

CertificateNotReady — 30-day TLS expiry warning
VeleroBackupMissed — backup age SLI
PrometheusKubernetesListWatchFailures — RBAC-induced scrape gaps
KubeDaemonSetMisScheduled — CAPI node removal detection
Database replication lag alerts per database type

One engineer maintains monitoring configuration for the entire fleet.

Disaster Recovery

Velero backup schedules, IAM policies, and object storage configuration are provisioned at cluster bootstrap. DR is not a follow-up task it is a bootstrap invariant.

The Sealed Secrets private key gap was discovered during a planned DR drill: a key that exists only inside the cluster cannot survive cluster loss, and every sealed secret in Git becomes permanently unrecoverable ciphertext. The private key is now automatically exported to external object storage via Velero at bootstrap. This protection is mandatory it is impossible to create a new KubeAid cluster without it.

Velero backup completion metrics are scraped by Prometheus. An alert fires if a backup has not succeeded within its scheduled window.

Can you expand on why you are using those projects/services?

ArgoCD over Flux: ApplicationSets allow a single ArgoCD instance to manage an arbitrary number of clusters from a centralized config. ArgoCD’s multi-source application support maps cleanly to the two-repo pattern — one source for KubeAid defaults, one for customer overrides. The ArgoCD UI provides fleet-wide visibility into sync status and drift.

Cilium over Calico/Flannel: FQDN-based egress policies are a hard compliance requirement. Cilium’s CiliumNetworkPolicy with FQDN selectors is the only CNI that supports this at the policy layer without a proxy. Hubble metrics integrate directly with Prometheus. eBPF-based enforcement has lower overhead than iptables at scale.

cert-manager with DNS-01 exclusively: HTTP-01 is broken by Traefik’s global HTTP→HTTPS redirect, which cannot be disabled without breaking HSTS compliance. DNS-01 with cloud IAM scoping works identically on bare metal and cloud, and does not require inbound HTTP access.

Cluster API over cloud-specific tooling: Provider-agnostic cluster lifecycle in Git. The same declarative model covers Hetzner bare metal, AWS, and Azure. Rolling control plane replacements, MachineHealthChecks, and autoscaling are provider-agnostic concerns handled by CAPI — not bespoke automation per cloud.

Helm for chart packaging: The KubeAid chart library wraps upstream charts and embeds the platform layer (network policies, monitoring rules, ArgoCD ignoreDifferences) alongside the workload. Customers use standard Helm values overrides. No custom CRDs or operators required for the integration layer.

What works particularly well

The two-repo pattern scales linearly. A platform fix committed to KubeAid propagates to every cluster within hours. A team of under 10 engineers operates dozens of production clusters across four cloud providers with no per-cluster patching.

Production failures become permanent fixes. Every incident is an opportunity to close the gap for every cluster simultaneously. The fleet never re-experiences the same failure in the same way.

Jsonnet for monitoring composition. kube-prometheus Jsonnet templates allow custom alerting rules to compose with upstream libraries without merge conflicts. One engineer maintains alerting for the entire fleet.

Cilium FQDN policies as a compliance primitive. Embedding FQDN egress rules alongside every workload manifest means network policy is not a separate compliance exercise it ships with the workload definition and is applied everywhere.

Velero key export as a bootstrap invariant. Making Sealed Secrets private key export mandatory at cluster creation means DR is never an afterthought. The constraint was added after finding the same gap on every cluster simultaneously during a drill.

Zero vendor control plane. The same CNCF stack runs on Hetzner bare metal in Germany, on-premises in Danish datacentres, and in EU-region cloud VMs. Customers can move it, fork it, and audit every component. GDPR, NIS2, and ISO 27001 requirements are satisfied by architecture.

What needs improvement

ArgoCD ignoreDifferences maintenance. Runtime drift Azure webhook injections, controller-managed fields, CRD caBundle rotation requires ongoing ignoreDifferences tuning in every affected chart. Each cloud provider introduces its own drift patterns. This operational overhead ideally belongs upstream in the charts themselves.

kube-prometheus regeneration across clusters. When a fix lands in a shared Jsonnet library, manifests must be regenerated for every affected cluster. This is currently a per-cluster manual step. A CI pipeline that detects library changes, regenerates all affected clusters, and opens PRs automatically would eliminate the gap.

Cluster API bare metal host pool management. Rolling control plane replacements require a spare host in the pool. When all bare metal hosts are occupied, a new control plane node cannot be provisioned and the rollout stalls. Better capacity planning automation is needed.

Sealed Secrets rotation. Sealed Secrets use asymmetric encryption keyed to a specific cluster key. Key rotation requires re-sealing every secret in the config repo. Tooling to automate re-sealing across the fleet is not yet in place.

What sort of “glue” had to be developed?

KubeAid chart library. The 100+ Helm chart wrappers that embed Cilium policies, Prometheus rules, and ArgoCD ignoreDifferences alongside upstream charts. These are the integration layer pre-built wiring that no operator writes from scratch for each cluster.

kube-prometheus Jsonnet library extensions. Custom libsonnet files that extend the upstream kube-prometheus library with Obmondo-specific alerting rules. These compose cleanly with upstream rules via Jsonnet merging semantics.

ArgoCD ApplicationSet templates. Parameterized ApplicationSet definitions that deploy the full KubeAid chart suite to any cluster whose config repo follows the two-repo pattern. Adding a new cluster is a config-repo operation, not a platform operation.

Sealed Secrets key export automation. A bootstrap step that exports the Sealed Secrets private key to external object storage via Velero immediately after cluster creation. This step is mandatory it cannot be skipped.

Prometheus alert composition templates. A set of reusable Jsonnet patterns for constructing SLI-based alerts (backup age, certificate validity window, database replication lag) that are consistent across the fleet without per-cluster duplication.

How did the Architecture Evolve

The architecture began as manually-managed clusters with per-cluster Helm deployments. The first pain point was drift: a fix applied to cluster A was not applied to clusters B–Z. The second was silent failures that were only discovered at impact.

Each failure drove a platform default:

Failure	Root Cause	KubeAid Default Added
Silent TLS expiry (90 days)	HTTP-01 blocked by Traefik redirect	DNS-01 + CertificateNotReady alert + external probe
Duplicate Prometheus timestamps	Two kubelet endpoints emitting same metric	Scrape-level relabeling rule in kube-prometheus
Service discovery silent failure	Missing RBAC on prometheus-k8s ServiceAccount	ClusterRole fix in prometheus-k8s chart
DR gap across all clusters	Sealed Secrets key not exported	Mandatory Velero key export at bootstrap
snowflake clusters	Per-cluster manual patching	Two-repo GitOps with KubeAid defaults

Each fix landed in KubeAid once. Every cluster received it. No future cluster will hit any of these failures in the same way.

What’s next for your architecture?

Automated kube-prometheus regeneration. A CI pipeline that detects when a shared Jsonnet library changes, regenerates manifests for all affected clusters, and opens PRs automatically eliminating the manual per-cluster regeneration step.

Deeper OpenTelemetry integration. Distributed tracing across customer workloads, integrated with the existing Prometheus metrics pipeline for a unified observability experience.

Automated compliance evidence generation. Generating GDPR, NIS2, and ISO 27001 evidence artifacts directly from Prometheus metrics and Kubernetes audit logs compliance as a cluster output, not a manual exercise.

Broader FQDN egress coverage. Extending Cilium FQDN policy defaults to every KubeAid chart in the library, closing the remaining gap between workloads that have pre-wired policies and those that do not.

Sealed Secrets rotation tooling. Automation to re-seal every secret in a config repo after a cluster key rotation making key rotation a safe, routine operation rather than a risky manual process.

Key Takeaways / Lessons

Every manual fix is a fix that has not been applied everywhere. The two-repo GitOps pattern is not primarily about automation it is about making the gap visible. If a fix requires touching a per-cluster config repo, it is a signal that the fix belongs in KubeAid instead.

Silent failures are the expensive ones. TLS expiry, backup failures, and RBAC gaps were all silent no component logged an error until impact. The alerting investment (CertificateNotReady, VeleroBackupMissed, PrometheusKubernetesListWatchFailures) pays back in avoided incidents, not reduced noise.

Compliance by architecture, not by policy. GDPR, NIS2, and ISO 27001 requirements are satisfied by the architecture itself: no vendor control plane, full auditability, data residency by infrastructure choice, DR proven at bootstrap. Policy documents that reference the architecture are a consequence, not the mechanism.

The CNCF ecosystem composability is the product. ArgoCD reconciles Helm charts. Helm packages Cilium, Prometheus, cert-manager, Velero. Prometheus scrapes Cilium Hubble metrics, cert-manager certificate status, and Velero backup completion metrics. Each CNCF project does one thing well. KubeAid is the integration layer and it is open source because the problems it solves are not unique to Obmondo.

DR gaps are always discovered on every cluster simultaneously. If a DR gap exists, it exists everywhere because every cluster was provisioned from the same template. The corollary is equally powerful: close the gap once, and it closes everywhere.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: A Cloud Native Scientific Computing Platform for CERN NextGen AI

Mon, 09 Mar 2026 00:00:00 +0000

Relevant Projects

Argo

ArgoCD is used to manage deployments of all services across multiple clusters and environments. Argo Workflows is used to manage multiple day-2 cluster operations.

Bootc

Bootc provides transactional, in-place operating system images and updates using OCI/Docker container images. Bootc is used to build the minimal base images for our cluster nodes.

ContainerSSH

ContainerSSH offers a SSH frontend to containers/pods running on Kubernetes clusters. Used to expose SSH as a way to access existing sessions in the cluster, with multiple authentication mechanisms offered (Kerberos, OIDC/OAuth2).

Longhorn

Longhorn offers cloud native distributed block storage for Kubernetes. Used to offer in-cluster shared storage to users, with individual and team getting dedicated volumes with read-write-multi access and automated backups.

Kubernetes

Kubernetes provides the required workload scheduling and orchestration for the diverse workloads running in our scientific platform.

Kubeflow

Kubeflow offers tools to manage the complete MLOps lifecycle. Used for profile management and quotas for users and teams, instantiation of notebook servers, pipelines for common and reusable tasks, hyper-parameter tuning with Katib and managing inference endpoints.

Kserve

Kserve standardizes hosting of inference endpoints. Used to encapsulate multiple runtime flavors, such as ONNX, Triton and several others, and offering a declarative way to define inference servers.

Kyverno

Kyverno offers policy as code with support for YAML and CEL based policies. Used as a key component for policy enforcement and mutating workloads according to those policies, adding required settings to expose storage systems, set resources based on GPUs, etc.

Kueue

Kueue is a kubernetes-native system offering advanced scheduling capabilities and quota management. Used to provide job queues and quotas, gang scheduling, fair sharing, among other capabilities.

Prometheus

Prometheus gathers the metrics and insights from all components in the cluster. Used for system and service metrics as well as providing individual workload performance insights on cpu, memory, power and other areas.

stargz-snapshotter

stargz-snapshotter provides lazy pulling of container images. Used to handle efficient job start and execution with image sizes of over 100GB in some cases.

Some additional projects outside the CNCF are essential to this deployment.

MLFlow, a project under the Linux Foundation used for model management and versioning
Nvidia GPU operator to setup and manage drivers and configurations for Nvidia GPUs
AMD GPU operator to setup and manage drivers and configurations for AMD GPUs

TLDR; Synopsis

This reference architecture describes a deployment supporting multiple teams in CERN’s new flagship “Next Generation Triggers” project, looking at innovative computing technologies for data acquisition and processing for the High-Luminosity Large Hadron Collider and beyond.

The cluster and platform target:

Multiple scientific use cases, covering traditional numerical computing as well as machine learning workloads across the different CERN experiments. Scientific computing and in particular high performance computing (HPC) have relied on cloud native tooling for parts of their workloads for several years, but relied on tools like SLURM for advanced scheduling capabilities. This is our first production deployment to offer the full stack based only on cloud native infrastructure
Access to both accelerators (in particular GPUs) as well as specialized nodes such as high CPU core count and high CPU clock frequency. In an earlier stage of experimentation also FPGAs are being integrated targeting fast inference in the CERN experiment online filters
In a shared pool of resources allow interactive access (including SSH, VSCode, Notebooks and just kubectl), traditional batch, MPI and training workloads and inference
Integration with the existing infrastructure at CERN for CI / CD systems (on-premises GitLab and GitHub), identity, and efficient access to multiple storage systems for both user and physics data

A pure cloud native based infrastructure can now successfully serve scientific computing workloads, with advanced scheduling features such as co-scheduling, fair sharing, among others.

Use Cases & Requirements

A set of requirements and use cases was initially defined when designing the architecture. The figure below shows how a shared pool of resources, mostly on-premises but integrating public cloud and supercomputing centers, should be accessed from different services.

Below we highlight specific requirements in terms of hardware and user facing functionality.

Hardware

Support for an heterogeneous set of resources: multiple CPU types, GPUs from multiple vendors, FPGAs and specialized accelerators, all in a shared pool of resources
Integration with multiple network interconnects targeting low latency, including at least Infiniband and RDMA over Converged Ethernet (RoCEv2)
A hybrid deployment integrating external resources, both from public cloud providers and supercomputing centers

User Facing

Curated environments based on container images and maintained by the platform team for the most common user setups, covering ML workloads but also traditional scientific computing. Particularly important has been ensuring these environments are compatible with the existing ways of working, with session environments setup with backwards compatibility for existing physicist tools and scripts
Easily customizable environments, either via dedicated environments maintained by user teams or the ability to install additional packages at runtime. This means users have sudo capabilities inside their sessions
Interactive access to sessions with the ability to choose the amount of GPUs at creation, and a corresponding CPU and memory allocation depending on the type of GPU selected. Once created, access available to the session via notebooks, local vscode instances and most importantly SSH for compatibility with the existing ways of working
Batch access to resources, with support for advanced scheduling capabilities such as queues, quotas, co-scheduling, fair sharing. In addition to the high priority user submissions of training or MPI jobs, the system should be able to backfill unused resources with lower priority workloads to ensure high usage efficiency
Support for the complete machine learning lifecycle, including data preparation, training, hyper-parameter tuning and model inference. In particular, support, efficient integration and automation using common training and tuning frameworks
Model management and versioning, integrated with the rest of the platform with collection and storage of training metadata and logging

Architecture

The diagram below shows how the different projects and tools match the requirements.

Areas of particular interest where effort was required include compute, scheduling, networking, storage and observability.

Compute

Proper isolation and reproducibility is essential for reliable performance and results, removing the effect of noisy neighbors and the latency between CPU and GPU. GPU nodes follow a NUMA-aware dual-socket layout, designed to preserve locality between CPU, memory, and accelerator resources. Each node has two CPU sockets, exposed as two NUMA nodes.

Depending on the node type, GPUs are distributed evenly across these NUMA domains: either 8 GPUs per node, with 4 GPUs attached to each NUMA node, or 4 GPUs per node, with 2 GPUs attached to each NUMA node.

Some relevant configurations to ensure the desired reproducibility and isolation.

CPU and memory resource allocations (requests and limits) scale with the number of GPUs requested by a session: pods receive resources in proportion to the selected GPU count while remaining aligned with the corresponding NUMA locality. This minimizes cross-socket communication, reduces latency between CPU and GPU, and improves the consistency of performance-sensitive workloads

Control CPU Management Policies on the Node, as documented here with the following settings on the kubelet.

cpu-manager-policy=static
cpu-manager-policy-options=full-pcpus-only=true
memory-manager-policy=Static
topology-manager-policy=restricted

Reserved systems resources for kubelet and other add-ons.

system-reserved=cpu=2,memory=1000Mi
reserved-memory=0:memory=1000Mi

Efficient access and distribution of container images, to accelerate the start of sessions based on both curated and custom environments each being multiple 10s of GBs in size. We provide this with a custom daemonset pre-pulling all curated images in advance when published, as well as the ability to do image streaming with the stargz-snapshotter.

Capability to burst out to external resources, in particular public cloud providers and HPC resources.

Scheduling

Bin packing in the scheduling profile instead of the default workload spread across nodes, with strategy MostAllocated ensuring better availability for workloads requiring full nodes.

Advanced scheduling features for queues supporting different resource types and QoS, workload co-scheduling, quotas and fair sharing to optimize overall resource utilization.
Kueue is the main component being used to achieve the advanced scheduling functionality we need.

Networking

Low latency networking such as Infiniband and RDMA over Converged Ethernet (RoCEv2) supporting both traditional CPU and GPU MPI workloads. Currently done by enabling hostNetwork and exposing the corresponding PCI devices for these specific use cases. Driver and lifecycle management of IB/RoCEv2 networking resources is controlled using the Nvidia network operator.

Storage

Users get different storage tiers which fit different usages.

Node local

Very high IOPS but limited space, typically on the low TBs available to all workloads on that node. Not useful for multi-node jobs requiring a shared filesystem. Used also for GPU Direct Storage (GDS) with local NVMEs.

Cluster local

Shared filesystem across all nodes in the cluster, deployed using Longhorn. Limiting the number of network hops as much as possible ensures reasonable IOPS and scales out well in space available with the number of nodes in the cluster (typically 10s of TBs per node). Connection through a single switch for higher performance, as much as possible. Volumes stored in this filesystem are backed up to S3 storage relying on the internal Longhorn backup functionality, with incremental points daily for a week and monthly.

Central

Shared filesystem outside the cluster with much higher storage space available. Managed using CEPH with different IOPS available, up to 2000 guaranteed with bursting to higher values.

Observability

The stack provides visibility into hardware performance, resource efficiency, and environmental impact.

Telemetry Collection

Leveraging a multi-layered collection strategy integrated with the kube-prometheus-stack.

Accelerators: NVIDIA dcgm-exporter and AMD device-metrics-exporter provide deep-field GPU telemetry (utilization, memory, power, temperature, and frequency).

Power & Sustainability: IPMI and Kepler capture hardware-level power metrics. Kepler utilizes RAPL to attribute energy consumption to individual workloads.

System Metrics: Standardized node and container metrics are ingested via Prometheus for a unified view of the cluster.

Visualization and Analysis

Data is exposed via Grafana through three specialized dashboard tiers.

Cluster Overview: Tracks aggregate utilization (CPU, GPU, RAM, Network, Thermals) and node-level health. It highlights idle resources and historical trends to guide capacity planning.

User/Workload Analytics: Provides namespace-filtered views for individual developers to monitor their specific deployments. This view balances resource efficiency (allocated vs. actual usage) with performance profiling (GPU/CPU/RAM saturation) and power consumption, allowing users to independently debug bottlenecks and optimize job performance.

Sustainability Tracking: A dedicated dashboard for CO2-equivalent emissions, offering transparency into the carbon footprint at both the cluster and individual workload levels.

Alerting and Optimization

Alertmanager is configured to trigger notifications for idle resources. By monitoring the delta between allocated requests and actual utilization, the system identifies “zombie” workloads or over-provisioned namespaces, allowing for potential automated or manual resource reclamation to reduce costs and energy waste.

What works particularly well

Workload isolation which is a key aspect when considering needs for reliable benchmarking results. Recent versions of Kubernetes have all the required capabilities to ensure NUMA affinity between CPUs and GPUs, resource pinning to individual workloads and reservation for system services and add-ons.

GPU setup, configuration and monitoring with well supported and up to date operators for both Nvidia and AMD GPUs and automation for metric collection on utilization, power, memory, etc. This includes the initial node configuration required with loading drivers and exposing them to the workloads, as well as day-2 operations such as driver upgrades with integration with the default methods for cordoning and draining nodes.

Kyverno for validation and mutation of cluster resources, allowing a policy based mutation of the resource capabilities based on labels available to users. This ranges from attaching volumes for access to external storage, setting environment variables such as home directories or authentication, automation of resources for cpu and memory and many others. Validation policies also include ensuring users do not attempt invalid NUMA allocations of CPUs and GPUs. Kyverno was chosen after the initial choice of the OPA Gatekeeper had limitations when modifying fields outside the matching location.

What needs improvement

GPU failure detection and integration with the scheduler, either by cordoning nodes or blocking access to faulty GPUs. Depending on the type of fault, the device plugins (for both Nvidia and AMD) may stop exposing faulty devices, but this is not reliable in all cases. Options such as Nvidia Sentinel are being evaluated.

GPU partitioning currently at node level, limiting the ability to have in the same node devices being exposed fully and others being partitioned using MIG. This is currently not supported by the GPU operators, but should be available in the future with the DRA drivers.

Scheduling workloads across multiple clusters, while possible, does not allow seamless access to logs or launching interactive sessions as done for single clusters - `kubectl log` and `kubectl exec` type of request. This is ongoing work in Kueue but currently limits the workloads submitted outside the main cluster to batch-like workloads.

Limited support for checkpoint and restore in several types of workloads, in particular the non machine-learning workloads. This limits the ability to push overall usage of the cluster further up by suspending / preempting idle sessions without losing any work. Efforts such as criu and the checkpoint-restore working group promise to greatly advance the capabilities of the cloud native ecosystem in this area in an workload agnostic way.

Low latency networking with InfiniBand or RoCEv2 in our setup is currently not namespaced and exposed to the users through hostNetwork. In case user workloads are not trusted, other options that provide better network-level isolation should be explored, including SR-IOV and via efforts such as dranet.

What sort of “glue” have you had to develop?

A key goal of our architecture was ensure the complete functionality is available via cloud native APIs, easing the integration with all other tools in the ecosystem. The glue pieces below target ease of use.

Access via SSH was one of the main requests from our users, allowing backwards compatibility with years of custom scripts, continuous integration and several other “ways of working” that require this type of access. We invested internally in developing the required capabilities in the containerssh project, with management of multiple sessions, multiple authenticated methods (OAuth2, Kerberos, X509), among others.

Large number of mutating policies, allowing us to give a better experience to users that do not want to use kubectl or write yaml. Relying on metadata labels in the different resources hides the complexity of setting up volume mounts, environment variables, etc. Our current policies include setting tolerations to assign workloads to specific node flavors, additional environment configurations for MPI workloads, injecting user metadata to access storage systems and interact with internal services, mounting multiple storage systems at CERN or enabling RDMA and GPU Direct Storage.

What’s next for your architecture?

Interactive session management via notebooks, relying on the Kubeflow Notebooks UI. As of today, users require a minimal yaml and usage of the kubectl client to create, list and delete their interactive sessions, even if access is then available via ssh, notebooks, vscode, etc. An upcoming improvement is to offer a UI based interface to manage sessions, likely relying on the Kubeflow Notebook UI but applying to any type of workload.

DRA and automated partitioning in the cluster, as currently we still rely on the Nvidia and AMD operators to manage GPU resources for this particular setup and need to manually set the desired MIG configuration for each node/pool of nodes. This will allow us to have heterogeneous configurations in the same node (with both partitioned and non-partitioned devices) as well as, in the future when the DRA drivers get this functionality, automatic partitioning of devices based on the current workloads.

Bursting to HPC resources, as existing supercomputers and upcoming AI factories have a large number of available GPUs. The main requirement is to integrate with SLURM as an API to manage these remote resources, but in a way that is seamless to users of the service. Projects such as interLink promise to hide the SLURM backends behind the Kubernetes APIs in our platform.

Key Takeaways / Lessons

Adapt to existing ways of working: the success of the platform depends on acceptance by users, who often will not have the time to change their ways of working. Anticipate where effort is needed to meet users where they’re at, building the required glue on top of your cloud native infrastructure.

Iterative and quick development: when exposing a new platform to users with so many stack changes from previous deployments, the ability to iterate very quickly taking into account user feedback is essential. This likely means planning for an intense period after first exposing the services, with the risk of loosing users from the start otherwise.

Upstream first: this is only way to ensure long term sustainability of a platform, exposing requirements and working together with the rest of the community. Local, temporary patches, when required, should be done in parallel with the upstream contributions.

Cloud native is ready for scientific computing and AI/ML: if there were doubts, this experience cleared them up. Cloud native enables the next generation of scientific computing and AI/ML platforms, with all the advanced requirements from high performance computing together with the integration with all modern tools that talk cloud native.

Discussion

End user members may participate in the discussion thread for this architecture.