Cloud Native Architecture – gitops

Architectures: GitOps-Native Multi-Cluster Kubernetes Platform for EU Data Sovereignty

Thu, 11 Jun 2026 00:00:00 +0000

Relevant Projects

Kubernetes

Using since: 2019
Current version: 1.33.x

The runtime substrate for all workloads. Cluster API manages the lifecycle of every cluster across Hetzner bare metal, AWS, and Azure declaratively from Git. No cloud console required.

Argo CD

Using since: 2021
Current version: v3.x

The reconciliation engine for the entire fleet. A fix committed once to the shared KubeAid chart library propagates to every cluster on the next ArgoCD sync no manual per-cluster patching.

Prometheus

Using since: 2020
Current version: v3.x

Generated per-cluster from a single Jsonnet vars file using kube-prometheus. Custom alerting rules for TLS expiry forecasting, backup SLIs, database replication lag, and Kubernetes ListWatch failures compose with upstream rule libraries without YAML merge conflicts.

Cilium

Using since: 2022
Current version: v1.17.x

Network policies are embedded alongside every workload manifest in KubeAid. FQDN egress rules for every external API dependency ship with the chart. Operators do not write network policies — KubeAid writes them once and applies them to every cluster.

cert-manager

Using since: 2021
Current version: v1.17.x

Defaults to DNS-01 ACME challenge validation with cloud-specific IAM scoping: IRSA on AWS, Workload Identity on Azure, API tokens on Hetzner. A CertificateNotReady alerting rule and external TLS expiry probe ship as KubeAid defaults added after a silent TLS expiry failure in production.

Velero

Using since: 2021
Current version: v1.15.x

Backup schedules, IAM policies, and object storage (S3, GCS, Azure Blob) are provisioned at cluster bootstrap. Velero also exports the Sealed Secrets private key to external object storage at bootstrap — making DR proven at provision time, not during a drill.

Cluster API

Using since: 2022
Current version: v1.10.x

Manages cluster lifecycle declaratively across Hetzner bare metal, AWS, and Azure. MachineHealthChecks detect and replace unhealthy nodes automatically. Cluster topology is version-controlled in Git.

Helm

Using since: 2019
Current version: v3.x

Over 100 curated Helm charts form the KubeAid chart library. Each chart ships with pre-wired Cilium network policies, Prometheus alerting rules, and ArgoCD ignoreDifferences configurations. Customers override only genuine differences.

TLDR; Synopsis

This reference architecture describes how Obmondo operates production Kubernetes for dozens of customer clusters across four cloud providers and bare metal with a team of under 10 engineers using a two-repo GitOps pattern built entirely on CNCF projects.

The core insight: every production failure fixed once in the shared platform repo (KubeAid) propagates to every cluster on the next ArgoCD sync. No cluster is ever patched manually. No cluster becomes a snowflake.

This architecture targets:

Zero snowflake clusters — a fix committed once applies everywhere within hours via ArgoCD
Full EU data sovereignty — identical stack on Hetzner bare metal in Germany, on-premises datacentres, and EU-region cloud VMs, with no vendor control plane, no proprietary APIs, and full auditability
Disaster recovery by default — backup schedules, IAM, and Sealed Secrets key export are automated at cluster bootstrap; if a cluster can be created, it can be recovered
Sub-10-engineer fleet operations — platform-level abstractions that scale cluster count without scaling headcount

Organization

Obmondo (EnableIT ApS) is a Danish Managed Kubernetes provider. It builds and operates production Kubernetes platforms for customers in financial services, healthcare, and public sector organizations across Denmark and the EU. All infrastructure must satisfy GDPR, NIS2, and in many cases ISO 27001 — by architecture, not by policy.

KubeAid, the open-source platform that powers Obmondo’s managed service, is available at https://github.com/Obmondo/KubeAid under the Apache 2.0 license.

Teams

Platform Engineering maintains KubeAid the shared chart library, Jsonnet monitoring templates, and ArgoCD application definitions that apply uniformly to every cluster. A fix here reaches every customer automatically.

SRE / Customer Operations handles day-2 operations: incident response, capacity planning, and customer-specific overrides in per-customer config repos. With KubeAid abstracting the platform layer, this team focuses exclusively on the ~10% that is genuinely different per customer.

Architecture

Goals

Eliminate snowflake clusters. Every production failure becomes a KubeAid default. The silent TLS expiry became DNS-01 + external probe + alert. The duplicate Prometheus timestamp alert became a scrape-level relabeling rule. The RBAC gap became a ClusterRole fix. Each fix landed once; every cluster received it automatically.

EU data sovereignty without vendor lock-in. No proprietary control plane. No per-node SaaS fees. No APIs that cannot be audited. The entire stack runs on CNCF projects identically on Hetzner bare metal in Germany, on-premises in Danish datacentres, and in EU-region cloud VMs. Customers can move it, fork it, and audit every component.

Disaster recovery proven at provision time. Backup schedules, IAM policies, object storage, and Sealed Secrets private key export are automated at bootstrap. The DR gap was discovered simultaneously on every cluster during a planned drill and made impossible to miss on any future cluster.

Observability without per-cluster overhead. Monitoring configuration is generated from a single Jsonnet vars file. One engineer maintains alerting for the entire fleet. Custom rules compose with upstream libraries without merge conflicts.

Architecture Overview

The architecture follows a strict two-repo pattern: Repo 1 — KubeAid (shared platform defaults) Over 100 curated Helm charts with pre-wired integrations. ArgoCD ApplicationSets deploy these charts to every cluster. Monitoring is generated per-cluster from a single Jsonnet vars file using kube-prometheus. Cilium network policies ship alongside workload manifests. DR configuration is provisioned at bootstrap. Every production failure becomes a default here.

Repo 2 — Customer config (genuine differences only) Each customer config repo holds only what is genuinely different: cloud provider, node sizes, alerting thresholds, compliance scope. If a value matches the KubeAid default, it does not exist in the config repo. ArgoCD reconciles both repos continuously across every cluster.

Per-cluster stack (all CNCF projects):

Layer	Project	Role
Lifecycle	Cluster API	Declarative cluster provisioning across all providers
Networking	Cilium	CNI + FQDN egress policies + Hubble metrics
TLS	cert-manager	DNS-01 ACME with cloud IAM scoping
GitOps	ArgoCD	Continuous reconciliation from both repos
Observability	Prometheus + Alertmanager	Generated from Jsonnet per cluster
Ingress	Traefik	Wildcard TLS, HTTP→HTTPS redirect
Backup / DR	Velero	Scheduled backups + Sealed Secrets key export
Secrets	Sealed Secrets (Bitnami)	Encrypted secrets in Git, key exported at bootstrap

Infrastructure Layer

Cluster API manages cluster lifecycle across all providers. The same declarative topology definition works on Hetzner bare metal (HCloud provider), AWS, and Azure. MachineHealthChecks detect unhealthy nodes and trigger automatic replacement. Node removal by CAPI is tracked and DaemonSet ghost pods are handled via alerting.

On Hetzner bare metal, clusters run on dedicated physical servers with ZFS and Ceph storage. Network isolation is enforced by Cilium at the pod level. No hypervisor, no shared infrastructure with other tenants.

Networking Layer

Cilium is the CNI for every cluster. Every KubeAid Helm chart ships with a CiliumNetworkPolicy alongside the workload manifest. FQDN egress rules for every external API dependency are pre-written and applied by ArgoCD. Operators do not write network policies they are a platform default.

FQDN policy enforcement is critical for compliance: customers need to attest that workloads only communicate with approved external endpoints. Cilium’s Hubble metrics feed into Prometheus for network observability.

TLS and Certificate Management

cert-manager defaults to DNS-01 ACME challenge validation. The switch from HTTP-01 was forced by a production failure: Traefik’s global HTTP→HTTPS redirect for HSTS compliance made HTTP-01 ACME validation permanently impossible every challenge request was redirected before reaching the solver pod. Certificates failed to renew silently for up to 90 days.

DNS-01 with cloud-specific IAM scoping (IRSA on AWS, Workload Identity on Azure, API tokens on Hetzner) is now the only supported challenge type. An external TLS expiry probe and a CertificateNotReady Alertmanager rule ship as KubeAid defaults.

Observability Layer

Prometheus and Alertmanager are generated per-cluster from a single Jsonnet vars file using kube-prometheus. Custom alerting rules compose with upstream rule libraries using Jsonnet merging semantics no YAML merge conflicts.

Custom rules added after production failures:

CertificateNotReady — 30-day TLS expiry warning
VeleroBackupMissed — backup age SLI
PrometheusKubernetesListWatchFailures — RBAC-induced scrape gaps
KubeDaemonSetMisScheduled — CAPI node removal detection
Database replication lag alerts per database type

One engineer maintains monitoring configuration for the entire fleet.

Disaster Recovery

Velero backup schedules, IAM policies, and object storage configuration are provisioned at cluster bootstrap. DR is not a follow-up task it is a bootstrap invariant.

The Sealed Secrets private key gap was discovered during a planned DR drill: a key that exists only inside the cluster cannot survive cluster loss, and every sealed secret in Git becomes permanently unrecoverable ciphertext. The private key is now automatically exported to external object storage via Velero at bootstrap. This protection is mandatory it is impossible to create a new KubeAid cluster without it.

Velero backup completion metrics are scraped by Prometheus. An alert fires if a backup has not succeeded within its scheduled window.

Can you expand on why you are using those projects/services?

ArgoCD over Flux: ApplicationSets allow a single ArgoCD instance to manage an arbitrary number of clusters from a centralized config. ArgoCD’s multi-source application support maps cleanly to the two-repo pattern — one source for KubeAid defaults, one for customer overrides. The ArgoCD UI provides fleet-wide visibility into sync status and drift.

Cilium over Calico/Flannel: FQDN-based egress policies are a hard compliance requirement. Cilium’s CiliumNetworkPolicy with FQDN selectors is the only CNI that supports this at the policy layer without a proxy. Hubble metrics integrate directly with Prometheus. eBPF-based enforcement has lower overhead than iptables at scale.

cert-manager with DNS-01 exclusively: HTTP-01 is broken by Traefik’s global HTTP→HTTPS redirect, which cannot be disabled without breaking HSTS compliance. DNS-01 with cloud IAM scoping works identically on bare metal and cloud, and does not require inbound HTTP access.

Cluster API over cloud-specific tooling: Provider-agnostic cluster lifecycle in Git. The same declarative model covers Hetzner bare metal, AWS, and Azure. Rolling control plane replacements, MachineHealthChecks, and autoscaling are provider-agnostic concerns handled by CAPI — not bespoke automation per cloud.

Helm for chart packaging: The KubeAid chart library wraps upstream charts and embeds the platform layer (network policies, monitoring rules, ArgoCD ignoreDifferences) alongside the workload. Customers use standard Helm values overrides. No custom CRDs or operators required for the integration layer.

What works particularly well

The two-repo pattern scales linearly. A platform fix committed to KubeAid propagates to every cluster within hours. A team of under 10 engineers operates dozens of production clusters across four cloud providers with no per-cluster patching.

Production failures become permanent fixes. Every incident is an opportunity to close the gap for every cluster simultaneously. The fleet never re-experiences the same failure in the same way.

Jsonnet for monitoring composition. kube-prometheus Jsonnet templates allow custom alerting rules to compose with upstream libraries without merge conflicts. One engineer maintains alerting for the entire fleet.

Cilium FQDN policies as a compliance primitive. Embedding FQDN egress rules alongside every workload manifest means network policy is not a separate compliance exercise it ships with the workload definition and is applied everywhere.

Velero key export as a bootstrap invariant. Making Sealed Secrets private key export mandatory at cluster creation means DR is never an afterthought. The constraint was added after finding the same gap on every cluster simultaneously during a drill.

Zero vendor control plane. The same CNCF stack runs on Hetzner bare metal in Germany, on-premises in Danish datacentres, and in EU-region cloud VMs. Customers can move it, fork it, and audit every component. GDPR, NIS2, and ISO 27001 requirements are satisfied by architecture.

What needs improvement

ArgoCD ignoreDifferences maintenance. Runtime drift Azure webhook injections, controller-managed fields, CRD caBundle rotation requires ongoing ignoreDifferences tuning in every affected chart. Each cloud provider introduces its own drift patterns. This operational overhead ideally belongs upstream in the charts themselves.

kube-prometheus regeneration across clusters. When a fix lands in a shared Jsonnet library, manifests must be regenerated for every affected cluster. This is currently a per-cluster manual step. A CI pipeline that detects library changes, regenerates all affected clusters, and opens PRs automatically would eliminate the gap.

Cluster API bare metal host pool management. Rolling control plane replacements require a spare host in the pool. When all bare metal hosts are occupied, a new control plane node cannot be provisioned and the rollout stalls. Better capacity planning automation is needed.

Sealed Secrets rotation. Sealed Secrets use asymmetric encryption keyed to a specific cluster key. Key rotation requires re-sealing every secret in the config repo. Tooling to automate re-sealing across the fleet is not yet in place.

What sort of “glue” had to be developed?

KubeAid chart library. The 100+ Helm chart wrappers that embed Cilium policies, Prometheus rules, and ArgoCD ignoreDifferences alongside upstream charts. These are the integration layer pre-built wiring that no operator writes from scratch for each cluster.

kube-prometheus Jsonnet library extensions. Custom libsonnet files that extend the upstream kube-prometheus library with Obmondo-specific alerting rules. These compose cleanly with upstream rules via Jsonnet merging semantics.

ArgoCD ApplicationSet templates. Parameterized ApplicationSet definitions that deploy the full KubeAid chart suite to any cluster whose config repo follows the two-repo pattern. Adding a new cluster is a config-repo operation, not a platform operation.

Sealed Secrets key export automation. A bootstrap step that exports the Sealed Secrets private key to external object storage via Velero immediately after cluster creation. This step is mandatory it cannot be skipped.

Prometheus alert composition templates. A set of reusable Jsonnet patterns for constructing SLI-based alerts (backup age, certificate validity window, database replication lag) that are consistent across the fleet without per-cluster duplication.

How did the Architecture Evolve

The architecture began as manually-managed clusters with per-cluster Helm deployments. The first pain point was drift: a fix applied to cluster A was not applied to clusters B–Z. The second was silent failures that were only discovered at impact.

Each failure drove a platform default:

Failure	Root Cause	KubeAid Default Added
Silent TLS expiry (90 days)	HTTP-01 blocked by Traefik redirect	DNS-01 + CertificateNotReady alert + external probe
Duplicate Prometheus timestamps	Two kubelet endpoints emitting same metric	Scrape-level relabeling rule in kube-prometheus
Service discovery silent failure	Missing RBAC on prometheus-k8s ServiceAccount	ClusterRole fix in prometheus-k8s chart
DR gap across all clusters	Sealed Secrets key not exported	Mandatory Velero key export at bootstrap
snowflake clusters	Per-cluster manual patching	Two-repo GitOps with KubeAid defaults

Each fix landed in KubeAid once. Every cluster received it. No future cluster will hit any of these failures in the same way.

What’s next for your architecture?

Automated kube-prometheus regeneration. A CI pipeline that detects when a shared Jsonnet library changes, regenerates manifests for all affected clusters, and opens PRs automatically eliminating the manual per-cluster regeneration step.

Deeper OpenTelemetry integration. Distributed tracing across customer workloads, integrated with the existing Prometheus metrics pipeline for a unified observability experience.

Automated compliance evidence generation. Generating GDPR, NIS2, and ISO 27001 evidence artifacts directly from Prometheus metrics and Kubernetes audit logs compliance as a cluster output, not a manual exercise.

Broader FQDN egress coverage. Extending Cilium FQDN policy defaults to every KubeAid chart in the library, closing the remaining gap between workloads that have pre-wired policies and those that do not.

Sealed Secrets rotation tooling. Automation to re-seal every secret in a config repo after a cluster key rotation making key rotation a safe, routine operation rather than a risky manual process.

Key Takeaways / Lessons

Every manual fix is a fix that has not been applied everywhere. The two-repo GitOps pattern is not primarily about automation it is about making the gap visible. If a fix requires touching a per-cluster config repo, it is a signal that the fix belongs in KubeAid instead.

Silent failures are the expensive ones. TLS expiry, backup failures, and RBAC gaps were all silent no component logged an error until impact. The alerting investment (CertificateNotReady, VeleroBackupMissed, PrometheusKubernetesListWatchFailures) pays back in avoided incidents, not reduced noise.

Compliance by architecture, not by policy. GDPR, NIS2, and ISO 27001 requirements are satisfied by the architecture itself: no vendor control plane, full auditability, data residency by infrastructure choice, DR proven at bootstrap. Policy documents that reference the architecture are a consequence, not the mechanism.

The CNCF ecosystem composability is the product. ArgoCD reconciles Helm charts. Helm packages Cilium, Prometheus, cert-manager, Velero. Prometheus scrapes Cilium Hubble metrics, cert-manager certificate status, and Velero backup completion metrics. Each CNCF project does one thing well. KubeAid is the integration layer and it is open source because the problems it solves are not unique to Obmondo.

DR gaps are always discovered on every cluster simultaneously. If a DR gap exists, it exists everywhere because every cluster was provisioned from the same template. The corollary is equally powerful: close the gap once, and it closes everywhere.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: End-to-End Cloud Native Telco Platform Automation at Swisscom

Mon, 16 Mar 2026 00:00:00 +0000

Relevant Projects

CNCF Projects

Kubernetes

Using since: 2021

The Kubernetes API is the database backend and control plane of the entire automation platform. It acts as the runtime for all CNFs, operators, and platform services. Custom Resource Definitions extend the API to cover telco-specific concerns like IPAM, DNS, and network function configuration.

Flux

Using since: 2022

Flux is the GitOps engine for continuous reconciliation. It monitors Git repositories and synchronizes all desired state — CNF deployment manifests, Custom Resources, DNS endpoints, certificate requests, IP claims, and test definitions — into Kubernetes clusters.

cert-manager

Using since: 2023

Automated certificate lifecycle management integrated with Swisscom’s internal PKI. Certificate requests are expressed as Kubernetes CRs, reconciled by Flux, and managed by cert-manager. Private keys never leave the cluster.

Headlamp

Using since: 2025

Kubernetes dashboard for the management cluster, providing cluster visibility, RBAC-based access control, a CRD documentation browser, and extensible plugin system. Swisscom is listed as an official Headlamp adopter.

SDC (Schema Driven Configuration)

Using since: 2024

Used as the Config Sync Operator to push assembled configurations to CNFs. SDC enables vendor-agnostic, declarative configuration management using YANG schemas and NETCONF/gNMI protocols. Swisscom adopted SDC as its strategic configuration management solution and actively contributes features including config blame, drift detection, validation, testing compatibility with CNFs, and NETCONF Actions support. Swisscom is listed as an official SDC adopter.

CoreDNS

Using since: 2021

In-cluster DNS service discovery for Kubernetes services. Also used with conditional forwarding to route queries for private 5G zones (e.g., 3gppnetwork.org) to the authoritative PowerDNS servers.

ExternalDNS

Using since: 2023

Kubernetes-native automation of DNS records in PowerDNS using Custom Resources and annotations.

MetalLB

Using since: 2022

Load balancer for bare-metal Kubernetes clusters. MetalLB IP address pools are managed via KRM, with IP addresses dynamically allocated from NetBox via the NetBox Operator.

Kubebuilder / controller-runtime

Using since: 2022

Scaffolding framework and libraries for building custom Kubernetes operators. Used to build all domain-specific operators for CNF configuration abstraction, IPAM integration, config synchronization, and DNS automation.

Other Projects

PowerDNS

Using since: 2023

Authoritative DNS server supporting automation of advanced resource records (NAPTR, SRV) required for 5G/SIP via ExternalDNS.

NetBox Operator

Using since: 2024

Kubernetes operator for IPAM integration, open-sourced by Swisscom. Brings IPAM into the Kubernetes API with a claim model inspired by PersistentVolumeClaims — dynamically allocating IP prefixes and addresses from NetBox, managing their lifecycle through Kubernetes garbage collection, and supporting sticky IPs for disaster recovery.

NetBox

Using since: 2023

IP Address Management (IPAM) and network infrastructure modeling. Used as the IPAM backend for dynamic IP allocation across all CNFs and platform services.

TL;DR or Synopsis

Swisscom has built a cloud native telco platform for the end-to-end automation of its 5G standalone core network and cross-domain resource orchestration. The architecture replaces traditional imperative network management (Jenkins pipelines, Ansible playbooks) with a fully declarative, Kubernetes-native automation model driven by GitOps and the Kubernetes Resource Model (KRM).

A mobile core development environment contains approximately 2,000 pods with over 5,000 interdependent configuration parameters across Cloud-Native Network Functions (CNFs) such as UPF, SMF, AMF, UDM, UDR, BSF, NRF, NSSF, and AUSF. Engineers express high-level intents as Kubernetes Custom Resources; custom operators dynamically assemble full configurations at runtime — fetching IP addresses from IPAM, secrets from Vault, certificates from PKI, and infrastructure details from the cluster.

While the 5G core is the primary domain, the orchestration framework extends across multiple network domains and infrastructure services, applying the same intent-based automation patterns consistently.

Organisation

Swisscom is the leading Telecommunications/ISP and ICT company and offers mobile, Internet and TV products, as well as comprehensive IT and digital services to private and business customers. Swisscom’s expertise in cloud native technologies is well-established, as evidenced by its status as a former Gold member and Management Board member of the Cloud Foundry Foundation, along with its certification for Cloud Foundry. Additionally, Swisscom demonstrates a strong commitment to the Open-Source community, having been a CNCF Silver Member for several years and serving as a Kubernetes Certified Service Provider (KCSP) partner. Our skilled employees have delivered numerous talks and presentations at prestigious events such as KubeCon, Cloud Native Zürich, Swiss Cloud Native Day, KCD Suisse Romande, ContainerDays.

The company has embarked on a strategic transformation from a traditional telecom operator (“Telco”) to a technology company (“TechCo”) with 5G as a central driver. Swisscom operates an extensive 5G Non-Standalone (NSA) network covering 99% of the Swiss population. The cloud native platform described here powers the 5G Standalone (SA) core.

Teams

Multiple teams collaborate on this platform:

Cloud Native Resource Orchestration — creates a robust framework for orchestrating cloud-native resources such as CNFs, IPAM, Networks, DNS, Kubernetes Clusters, and more. Designs and operates GitOps pipelines, builds Kubernetes operators, and develops the management cluster UI/UX.
Mobile Cloud Native Engineering — designs, implements, and operates the cloud native 5G core platform, including GitOps pipelines, Kubernetes operators, and network function lifecycle management.
DNS Engineering — builds and operates the highly reliable cloud native DNS service underpinning the 5G core and other infrastructures.
Network Engineering — provides IPAM and Network-as-a-Service.
Platform & Developer Experience — manages Kubernetes clusters and builds developer tooling.

Architecture overview & Goals

Goals

Full GitOps for the 5G Core — Extend GitOps beyond CNF deployment to include network function configuration, certificate management, DNS record provisioning, IP address management, and testing — achieving continuous reconciliation across all layers.
Declarative, Intent-Based Configuration — Replace static, low-level configuration manifests with abstract, intent-driven Custom Resources. Engineers specify what they want using a high level intent (e.g., “this CNF needs an IP address from a subnet in network zone A”) rather than how to achieve it, with Kubernetes operators dynamically assembling configurations at runtime.
Automated CD — High level of automation for telco deployment rollouts. This includes rethinking Change Processes as well as building solid CI/CD/CT pipelines to ensure a highly reliable network.
In-Band with Kubernetes — Bring all automation in-band with the Kubernetes API, eliminating out-of-band tools like Jenkins pipelines and Ansible playbooks. This ensures that the Kubernetes orchestrator has full visibility and control over all resources, enabling self-healing and reconciliation.
Cloud Native DNS Service — Operate a highly reliable, geo-redundant, on-premises DNS service for the 5G core using open-source technologies (CoreDNS, PowerDNS, ExternalDNS), fully automated via GitOps and Kubernetes Custom Resources.
Contribute to the Ecosystem and Shape the Industry Discussion — Open-source key components built or contributed to during this journey (NetBox Operator, SDC, demo code) to enable other organizations to adopt similar patterns. Contribute to Meetups and Conferences in order to achieve broader success of Cloud Native adoption in the Telco Community.

Architecture overview

The platform is organized in three layers:

The Intent Layer stores high-level desired state in Git — engineers define what they want using concise Custom Resources (e.g., a DNN configuration with hostname, region, and SNSSAI).

The Automation Layer runs on Kubernetes and continuously reconciles. Flux pulls intents from Git. Custom operators dynamically assemble full configurations by fetching IP addresses from NetBox (via the NetBox Operator) and creating connectivity in a Network-as-a-Service platform. It then pushes the KRM formatted Runtime Configuration to a Git repository for the Runtime layer to consume.

The Runtime Layer hosts the 5G core CNFs and supporting services. Flux pulls intents from Git. SDC pushes the assembled configuration to CNFs via NETCONF/gNMI. MetalLB, cert-manager, External Secrets Operator, ExternalDNS are used to configure the Workload cluster.

Key Design Principles

GitOps + KRM: Git stores high-level intents; Kubernetes manages dynamic, low-level configuration assembly. This is a shared source of truth across Git and Kubernetes.
Continuous Reconciliation: Every aspect of the system is continuously reconciled against the desired state — following the four OpenGitOps principles.
Abstraction over Complexity: Engineers work with simple, high-level intents; operators handle the complex assembly.

Can you expand on why you are using those projects/services?

From GitOps to KRM

Standard GitOps tools (Flux, Argo CD) combined with Helm/Kustomize have limitations for complex telco use cases: they cannot use live Kubernetes resources as inputs during rendering, cannot invoke custom business logic during template processing, and cannot dynamically assemble configurations from multiple sources. By extending GitOps with the Kubernetes Resource Model (KRM) — Custom Resource Definitions, Custom Resources, and custom Operators — the platform achieves dynamic configuration assembly at runtime.

Configuration Abstraction & Dynamic Assembly

A typical 5G core configuration includes IP addresses, VLAN IDs, DNS records, NF variables, secret references, and certificate references. Previously, engineers had to know all values upfront and embed them statically. The new intent-based model inverts this. The following is an example of a DNN configuration in pseudo-yaml-code.

On the intent layer, only a very stripped-down KRM manifest exists:

apiVersion: telco.swisscom.com/v1
kind: Dnn
spec:
 hostname: "gprs.swisscom.com"
 ipv4: true
 region: ch-east
 type: MobileInternet
 snssai: [1, 10]

This KRM manifest is stored in git and synced to the Management cluster using Flux. From this, the Operators in the automation layer create intermediate resources, in this case an IP Address via NetBox:

apiVersion: netbox.dev/v1
kind: IpAddressClaim
spec:
 parentPrefixSelector:
 region: "ch-east" # from Dnn.spec.region
 family: "IPv4" # from Dnn.spec.ipv4
status:
 ipAddress: 1.2.3.4

The Automation Layer creates the following low level resources for the Runtime Layer:

apiVersion: config.sdcio.dev/v1alpha1
kind: Config
metadata:
 labels:
 config.sdcio.dev/targetName: ch-east  # from Dnn.spec.region
 config.sdcio.dev/targetNamespace: default
spec:
 priority: 10
 config:
 - path: /
 value:
 dnn:
 - name: "gprs.swisscom.com" # from Dnn.spec.hostname
 ip: 1.2.3.4 # from IpAddressClaim.status.ipAddress
 snssai: [1, 10] # from Dnn.spec.snssai
 type: MobileInternet  # from Dnn.spec.type
---
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
spec:
 endpoints:
 - dnsName: "gprs.swisscom.com" # from Dnn.spec.hostname
 recordType: A
 targets:
 - 1.2.3.4 # from IpAddressClaim.status.ipAddress

SDC will now sync the configuration to the 5G CNF and ExternalDNS will create the DNS records in the authoritative PowerDNS backend.

What has worked well?

Intent-based configuration dramatically reduced the complexity engineers face. Instead of managing thousands of interdependent parameters, they work with concise Custom Resources.
Full GitOps reconciliation across all layers (deployment, configuration, DNS, certificates, IPAM), configuration drift is detected and reverted to ensure consistency.
Custom Kubernetes Operators (built with Kubebuilder/controller-runtime) proved to be the right pattern for telco domain-specific concerns, providing full reconciliation support and native KRM integration.
The claim model for IPAM (NetBox Operator) elegantly solved dynamic IP allocation by following established Kubernetes patterns (PVC analogy).
Bringing all automation in-band with Kubernetes gave the orchestrator full visibility and control, enabling self-healing and eliminating the brittleness of out-of-band tools.
DNS resilience engineering — dedicated hackathons, chaos testing, and disaster recovery playbooks significantly improved DNS service reliability.
Cross-team collaboration on a shared platform and KRM patterns accelerated adoption across multiple network domains.

What has not worked well?

NETCONF as a configuration protocol introduces complexity — it requires SDC as an intermediary and prevents fully Kubernetes-native configuration. Ideally, CNF vendors would support native K8s APIs using CRs/CRDs or Secrets/ConfigMaps.
Tooling gap for KRM-based configuration assembly — no mature, community-standard Kubernetes-native tool exists for dynamic configuration hydration. Swisscom had to build custom operators to fill this gap.
GitOps+KRM auditability trade-off — with dynamically assembled configurations, not all state is visible in Git history. The team continues to explore automated intermediary Git layers.
Cumbersome vendor configuration manifests — large, monolithic configuration files from CNF vendors (with ~5,000 interdependent parameters) required significant effort to decompose into intent-based abstractions.
Telco’s imperative model and Kubernetes’ declarative approach do not align well - SDC follows the declarative paradigm, where users define the desired state and SDC determines the actions to achieve it. In contrast, NETCONF/gNMI use an imperative model that requires explicit ordered steps (“do A, then B, then C”). Translating declarative goals into imperative sequences is complex when user‑defined ordering matters, such as for firewall rules where evaluation order affects behaviour. Example: Issue “Support user sorted lists”

What sort of “glue” have you had to develop?

Custom Kubernetes Operators — domain-specific operators for CNF configuration abstraction, config synchronization, IPAM integration, and DNS automation, all scaffolded with Kubebuilder and controller-runtime.
Configuration hydration logic — the CNF Config Operator dynamically assembles full configurations from multiple sources (NetBox, Vault, cluster environment) based on high-level intents.
SDC contributions — significant development work on SDC including early testing of CNF compatibility, configuration validation, monitoring, config blame, drift detection, and NETCONF Actions support.
ExternalDNS NAPTR support — contributed PR #4212 to enable NAPTR record support for SIP phone calls.
Resilient DNS architecture — created a reference architecture and open sourced on GitHub.
Headlamp plugins — Custom Resource plugin.

How did the Architecture Evolve

Journey

The architecture evolved significantly from traditional imperative automation to the current declarative, KRM-based model:

Phase 1 — Ansible + Jenkins: Initial automation used Ansible playbooks triggered by Jenkins pipelines. Configuration was fire-and-forget with no continuous reconciliation.
Phase 2 — GitOps for deployment: Introduced Flux for CNF deployment, but configuration remained out-of-band via Ansible/NETCONF.
Phase 3 — Full GitOps + KRM: Extended GitOps to cover configuration, DNS, IPAM, certificates, and testing. Built custom operators and adopted SDC for config synchronization. Achieved continuous reconciliation across all layers.

Key lessons

De facto GitOps for operators is not true GitOps — creating Helm releases in Git while configuring NFs via NETCONF out-of-band breaks the GitOps model. Bringing configuration into the Kubernetes API was essential.
Bringing everything in-band with Kubernetes enables self-healing, reconciliation, and eliminates the brittleness of out-of-band tools.
Git as the only source of truth is insufficient — the shared source of truth model (Git for intents, Kubernetes for dynamic state) was a deliberate and necessary evolution.
Abstraction is critical — engineers cannot effectively manage 5,000+ parameters directly. Intent-based CRs with dynamic assembly significantly reduced cognitive load and errors.
Custom Kubernetes Operators are the right pattern for domain-specific concerns that existing tools cannot address.
Contribute upstream — local patches create long-term maintenance burden. Swisscom prioritizes upstream contributions (ExternalDNS, SDC, NetBox Operator) for sustainability.

What’s next for your architecture?

Mature SDC Integration — Continue expanding SDC for full lifecycle management with continuous reconciliation via gNMI and NETCONF, including completion of NETCONF Actions support.
Eliminate NETCONF Dependency — Work with CNF vendors to move toward fully Kubernetes-native configuration APIs, reducing reliance on legacy telco protocols.
Advanced Dynamic Configuration Assembly — Develop more sophisticated Kubernetes operators for multi-source configuration hydration, enabling even more complex intent-based workflows across multiple network domains.
Multi-Cluster & Edge Expansion — Scale the architecture to additional edge locations and Kubernetes clusters while maintaining consistent GitOps-driven automation.
Community Tooling for KRM — Contribute toward a mature, Kubernetes-native tool for dynamic configuration assembly that the wider cloud native community can adopt, addressing the current gap in tooling identified during the project.
Resilience & Reliability — Ongoing improvements to cross-cluster redundancy, disaster recovery playbooks, chaos testing framework, and enhanced monitoring/alerting.
Observability & AIOps — Integrate AI-driven operations capabilities leveraging the rich telemetry data from the platform’s monitoring stack.
Cross-Domain Expansion — Extend the orchestration framework to additional network domains and infrastructure services beyond the 5G core, applying the same intent-based automation patterns consistently.

Community Contributions

Contribution	Details
NetBox Operator	Open-sourced under https://github.com/netbox-community/netbox-operator
SDC Contributions	Active contributor to the SDC project (on its path to CNCF incubation)
KRM Demo Code	https://github.com/swisscom/containerdays-2024-krm
Conference Talks	KubeCon EU, ContainerDays, Open Source Summit EU
CNCF/LFN Whitepaper	Co-authored Accelerating Cloud Native in Telco

Discussion

End user members may participate in the discussion thread for this architecture.